Data Sources

Data Sources management lets you upload enterprise knowledge bases, industry materials, and other data to provide accurate context support for AI analysis and content generation.

Overview

Data Sources management provides:

• 📤 File Upload - Support for CSV and Markdown files
• 🌐 Remote API Data Source - Configure a remote HTTP endpoint, automatically fetch and parse it into tabular data
• 🗂️ Data Organization - Flexible naming, description, and tags
• 📊 Data Statistics - Row count, column count, data quality score
• 🔄 Manual Refresh - Remote API data sources can sync the latest data anytime

Add Data Source (File & Remote API)

On the Dashboard → Data Sources page, click the Add Data Source button in the top-right corner. The dialog offers two source types:

• Upload File: Upload a local CSV / Markdown file (≤ 10MB per file).
• Remote API: Configure a remote HTTP endpoint. The platform requests the endpoint per your config, parses the returned JSON, and converts it into a unified "rows / columns / fields" table.

Option 1: Upload File

1. Switch to the Upload File tab.
2. Drag and drop or click to select a .csv / .md file.
3. Fill in the Data Source Name (required), description, and tags.
4. Click Create Data Source. The system parses the file in the background and generates field analysis.

Option 2: Remote API Data Source

1. Switch to the Remote API tab.
2. Fill in the endpoint configuration (see "Interface Contract" below).
3. Click Test Connection. The platform immediately requests the endpoint and shows a data preview (first few rows) to validate the configuration.
4. Once confirmed, click Create Data Source. The platform performs the first sync and persists the data.
5. Afterwards, click the Refresh button on the data source card to re-fetch the latest data anytime.

For security, remote endpoints only support http / https protocols, and access to localhost and private/internal addresses is blocked. Credentials (Token / API Key / password) are never echoed back to the frontend after saving.

Interface Contract

The platform acts as the caller, requesting your endpoint per your configuration and converting the response into a table. Your endpoint must satisfy the following contract.

Interface Signature

<METHOD> <URL>?<queryParams>
Headers: <custom headers> + <auth header>
Body:    <body> (POST / PUT only, JSON string)

Request (Platform → Your Endpoint)

Response (Your Endpoint → Platform)

• The response Content-Type must be JSON.
• The data body must be an array of objects, where each object represents one flat key-value row.
• If the array is wrapped inside a response object, use dataPath to point to its location.

Recommended unified envelope:

{
  "code": 0,
  "message": "ok",
  "data": [
    { "date": "2026-05-01", "visits": 1280, "source": "google" },
    { "date": "2026-05-02", "visits": 1543, "source": "google" }
  ]
}

This corresponds to dataPath = "data". If the endpoint returns an array directly (root is an array), leave dataPath empty.

Upon receiving the array of records, the platform will:

• Use the union of record keys as column names (ordered by the first record);
• Extract each record's values by column to form data rows (nested objects/arrays are serialized to JSON strings);
• Automatically infer field types (text / number / date / boolean) and compute a data quality score.

Configuration Example

A GET endpoint with Bearer auth that reads data from data.list:

Uploading Data

Supported Formats

Document Types

• PDF documents
• Word documents (.docx, .doc)
• Excel spreadsheets (.xlsx, .xls)
• PowerPoint (.pptx, .ppt)
• Text files (.txt)
• Markdown files (.md)
• CSV data files

Limits

• Single file size: ≤ 50MB
• Batch upload: ≤ 100 files
• Total storage: Varies by plan (Basic 5GB, Professional 50GB, Enterprise unlimited)

Upload Methods

• Single File Upload - Upload individual files with metadata
• Batch Upload - Upload multiple files at once with shared settings
• URL Import - Import files from URLs with optional auto-sync

Data Management

Data List

View all uploaded data sources with name, type, size, update time, and indexing status.

Data Categories

Organize data sources into categories:

• Product documentation
• Market materials
• Internal resources
• Customer cases
• Training materials

Tag Management

Use tags for flexible cross-category organization and quick filtering.

Data Details

Document Information

• File details (name, type, size, page count)
• Category and tags
• Indexing status
• Content summary
• Usage statistics (citations, search hits)

Content Preview

Preview documents directly in the browser with pagination support.

Version History

Track document versions with comparison and rollback capabilities.

Data Search

Semantic Search

AI-powered search that understands meaning and context, not just keywords. Results include relevance scores and highlighted matching content.

Advanced Search

Filter by keywords, category, tags, file type, upload date, file size, and uploader.

Data Security

Access Control

Configure visibility and access permissions:

• Team visible
• Admin only
• Specific members
• Public (API accessible)

Audit Log

Track all data operations including uploads, downloads, citations, and metadata changes.

Best Practices

Data Organization

• Use descriptive naming: Category_Name_Version_Date.ext
• Organize by business line, document type, or time period
• Use tags to supplement categories

Content Quality

• Verify accuracy before uploading
• Remove sensitive information
• Keep documents updated
• Regular maintenance and cleanup

FAQ

Q: What file formats are supported? A: PDF, Word, Excel, PowerPoint, TXT, Markdown, and CSV.

Q: Is there a file size limit? A: Single files up to 50MB. Total storage depends on your plan.

Q: Is uploaded data secure? A: Yes, we use encrypted storage, access controls, and audit logs for comprehensive security.