# Data

Data components connect the flow to external data sources — HTTP APIs, files, databases, object storage, and inbound webhooks.

---

## API Request

Makes an HTTP request to an external URL and passes the response downstream.

**Use in a flow**

1. Add an **API Request** node to the canvas.
2. Enter one or more **URLs** and select the HTTP **Method**.
3. Optionally configure headers, query parameters, and body.
4. Connect the **Data** or **DataFrame** output to the next step.

```
API Request (Data) ──► Processing / Agent / Output
```

You can also paste a **cURL** command to auto-fill request fields (enable **Use cURL**).

**Parameters — Request**

| Parameter | Default | Hidden | Description |
| --- | --- | --- | --- |
| URLs | — | No | One or more URLs to call, separated by commas. |
| Method | GET | No | HTTP method: GET, POST, PATCH, PUT, or DELETE. |
| Use cURL | false | No | When enabled, populate fields from the **cURL** input instead of manual entry. |
| cURL | — | Yes | Paste a cURL command to fill headers, body, and method. Visible when **Use cURL** is enabled. |
| Query Parameters | — | Yes | Query string parameters appended to the URL. |
| Body | — | Yes | Request body as key-value pairs (for POST, PATCH, PUT). |
| Headers | User-Agent default | Yes | HTTP headers to send with the request. |

**Parameters — Options**

| Parameter | Default | Hidden | Description |
| --- | --- | --- | --- |
| Timeout | 30 | Yes | Request timeout in seconds. |
| Follow Redirects | true | Yes | Whether to follow HTTP redirects. |
| Save to File | false | Yes | Save the API response to a temporary file. |
| Include HTTPx Metadata | false | Yes | Include headers, status code, and redirect history in the output. |
| Enable Retry | false | Yes | Retry failed requests. |
| Max Retries | 3 | Yes | Maximum retry attempts. |
| Backoff Factor | 2.0 | Yes | Multiplier for delay between retries. |
| Min Backoff (seconds) | 1.0 | Yes | Minimum wait between retries. |
| Max Backoff (seconds) | 10.0 | Yes | Maximum wait between retries. |
| Retry Status Codes | 404,408,429,500,502,503,504 | Yes | HTTP status codes that trigger a retry. |
| Workspace ID | — | Yes | Workspace ID for secret resolution in URLs and headers. |

**Outputs**

| Output | Description |
| --- | --- |
| Data | The API response as data objects. |
| DataFrame | The API response as a table. |

**Limitations**

- **External network** — The URL must be reachable from the flow runtime.
- **Secrets** — Use workspace secret resolution where supported; do not hard-code credentials in flows.

---

## File

Loads a file (uploaded or from a URL) and makes its content available to the flow.

**Use in a flow**

1. Add a **File** node to the canvas.
2. Upload files under **Files**, or connect a **Server File Path** from another component.
3. Connect **Data**, **DataFrame**, or **Message** output to downstream steps.

```
File (Data) ──► CharacterTextSplitter / Agent / Embeddings
```

**Parameters**

| Parameter | Default | Hidden | Description |
| --- | --- | --- | --- |
| Files | — | No | Upload one or more files, or provide file paths. |
| Server File Path | — | Yes | Connect a Data or Message object with a server file path instead of uploading. |
| Separator | \n\n | Yes | Separator between multiple file contents when output as Message. |
| Processing Concurrency | 1 | Yes | Number of files to process in parallel when multiple files are loaded. |
| Silent Errors | false | Yes | If enabled, file errors are logged but do not stop the flow. |
| Delete Server File After Processing | true | Yes | Remove server-side files after processing. |
| Ignore Unsupported Extensions | true | Yes | Skip files with unsupported extensions. |
| Ignore Unspecified Files | false | Yes | Skip Data objects without a file path property. |

**Outputs**

| Output | Description |
| --- | --- |
| Data | Parsed file content as data objects. |
| DataFrame | File content as a table. |
| Message | File content as a single message string. |

**Limitations**

- **Text files** — Primarily supports text-based file types (txt, csv, json, md, and similar).
- **Archives** — Zip and tar bundles are supported; contents are extracted and processed.

---

## MySQL

Executes a query against a MySQL database and returns the results.

**Use in a flow**

1. Add a **MySQL** node to the canvas.
2. Enter the **Database URL** for your MySQL instance.
3. Enter the **MySQL Query** to run.
4. Connect **Query Results** to downstream components.

```
MySQL (Query Results) ──► Agent / Output / Processing
```

`mysql://` URLs are automatically converted to use the PyMySQL driver.

**Parameters**

| Parameter | Default | Hidden | Description |
| --- | --- | --- | --- |
| Database URL | — | No | MySQL connection URL (e.g., `mysql://user:pass@host:3306/dbname`). |
| MySQL Query | — | No | The SQL query to execute. |
| Schema | — | Yes | Optional schema or database name (can also be set in the URL). |
| Include Columns | true | Yes | Include column names in the result. |
| Add Error Details | false | Yes | Include detailed error information in the result when a query fails. |
| Pool Size | 5 | Yes | Number of connections kept in the pool. |
| Max Overflow | 10 | Yes | Extra connections allowed beyond the pool size. |

**Outputs**

| Output | Description |
| --- | --- |
| Query Results | Query results as a DataFrame. |

**Limitations**

- **MySQL only** — Requires a valid MySQL connection URL.
- **Credentials in URL** — Connection strings may contain passwords; handle securely.
- **Query permissions** — Results depend on the database user’s access rights.

---

## PostgreSQL

Executes a query against a PostgreSQL database and returns the results.

**Use in a flow**

1. Add a **PostgreSQL** node to the canvas.
2. Enter the **Database URL** for your PostgreSQL instance.
3. Enter the **Postgresql Query** to run.
4. Connect **Query Results** to downstream components.

```
PostgreSQL (Query Results) ──► Agent / Output / Processing
```

`postgresql://` URLs are automatically converted to use the psycopg driver.

**Parameters**

| Parameter | Default | Hidden | Description |
| --- | --- | --- | --- |
| Database URL | — | No | PostgreSQL connection URL (e.g., `postgresql://user:pass@host:5432/dbname`). |
| Postgresql Query | — | No | The SQL query to execute. |
| Schema | — | Yes | Optional schema name (can also be set in the URL). |
| Include Columns | true | Yes | Include column names in the result. |
| Add Error Details | false | Yes | Include detailed error information in the result when a query fails. |
| Pool Size | 5 | Yes | Number of connections kept in the pool. |
| Max Overflow | 10 | Yes | Extra connections allowed beyond the pool size. |

**Outputs**

| Output | Description |
| --- | --- |
| Query Results | Query results as a DataFrame. |

**Limitations**

- **PostgreSQL only** — Requires a valid PostgreSQL connection URL.
- **Credentials in URL** — Connection strings may contain passwords; handle securely.
- **Query permissions** — Results depend on the database user’s access rights.

---

## Redshift

Executes a query against an Amazon Redshift data warehouse and returns the results.

**Use in a flow**

1. Add a **Redshift** node to the canvas.
2. Enter the **Database URL** for your Redshift cluster.
3. Enter the **Redshift Query** to run.
4. Connect **Query Results** to downstream components.

```
Redshift (Query Results) ──► Agent / Output / Processing
```

`redshift://` URLs are automatically converted to use the redshift_connector driver.

**Parameters**

| Parameter | Default | Hidden | Description |
| --- | --- | --- | --- |
| Database URL | — | No | Redshift connection URL (e.g., `redshift://user:pass@host:5439/dbname`). |
| Redshift Query | — | No | The SQL query to execute. |
| Schema | — | Yes | Optional schema name (can also be set in the URL). |
| Include Columns | true | Yes | Include column names in the result. |
| Add Error Details | false | Yes | Include detailed error information in the result when a query fails. |
| Pool Size | 5 | Yes | Number of connections kept in the pool. |
| Max Overflow | 10 | Yes | Extra connections allowed beyond the pool size. |

**Outputs**

| Output | Description |
| --- | --- |
| Query Results | Query results as a DataFrame. |

**Limitations**

- **Redshift only** — Requires a valid Redshift connection URL and network access to the cluster.
- **Credentials in URL** — Connection strings may contain passwords; handle securely.
- **Query permissions** — Results depend on the database user’s access rights.

---

## S3 Operations

Reads from or writes to an Amazon S3 bucket by listing buckets and objects and generating pre-signed URLs for secure access.

**Use in a flow**

1. Add an **S3 Operations** node to the canvas.
2. Select **AWS Region**, **S3 Bucket**, and one or more **Object Keys**.
3. Choose **Operations** (GET, PUT, POST, DELETE) for the pre-signed URLs you need.
4. Connect **Result** output to downstream steps.

```
S3 Operations (Result) ──► API Request / Agent / Output
```

Pre-signed URLs allow time-limited access to S3 objects without embedding long-lived credentials in the flow.

**Parameters**

| Parameter | Default | Hidden | Description |
| --- | --- | --- | --- |
| AWS Region | us-east-1 | No | AWS region where the bucket is located. |
| S3 Bucket | — | No | Select a bucket from your AWS account. Refresh to reload the list. |
| Object Keys | — | No | Select one or more objects in the bucket. |
| Operations | GET | No | One or more operations for the generated URLs: GET, PUT, POST, or DELETE. |
| Schema JSON (Optional) | — | Yes | JSON override for region, bucket, object keys, and operations. When set, manual inputs are ignored. |

**Outputs**

| Output | Description |
| --- | --- |
| Result | Pre-signed URLs and metadata (bucket, objects, operations, region) as a data object. |

**Limitations**

- **AWS credentials** — IAM permissions must allow listing and pre-signing for the selected bucket.
- **Pre-signed URLs** — Access is via generated URLs, not direct in-flow object streaming.
- **URL expiry** — Pre-signed URLs use a fixed expiration configured by the runtime.

---

## SQL Query

Executes a SQL query against a configured database connection and returns the results.

**Use in a flow**

1. Add a **SQL Query** node to the canvas.
2. Enter the **Database URL** for your database.
3. Enter the **SQL Query** to run.
4. Connect **Query Results** to downstream components.

```
SQL Query (Query Results) ──► Agent / Output / Processing
```

Works with any database supported by SQLAlchemy when a valid connection URL is provided.

**Parameters**

| Parameter | Default | Hidden | Description |
| --- | --- | --- | --- |
| Database URL | — | No | SQLAlchemy-compatible connection URL. |
| SQL Query | — | No | The SQL query to execute. |
| Include Columns | true | Yes | Include column names in the result. |
| Add Error | false | Yes | Append error details to the result when a query fails. |

**Outputs**

| Output | Description |
| --- | --- |
| Query Results | Query results as a DataFrame. |

**Limitations**

- **Connection URL** — Must be a valid SQLAlchemy database URL for your engine.
- **Credentials** — Connection strings may contain passwords; handle securely.
- **Query permissions** — Results depend on the database user’s access rights.

---

## URL

Fetches content from a URL and returns the response. Can crawl linked pages recursively from a root URL.

**Use in a flow**

1. Add a **URL** node to the canvas.
2. Enter one or more **URLs** to fetch.
3. Set **Max Depth** to control how far linked pages are followed.
4. Connect **Data**, **Message**, or **DataFrame** output downstream.

```
URL (Message) ──► CharacterTextSplitter / Agent / Embeddings
```

At depth 1, only the initial page is loaded. Higher depths follow links on those pages.

**Parameters**

| Parameter | Default | Hidden | Description |
| --- | --- | --- | --- |
| URLs | — | No | One or more root URLs to crawl. |
| Max Depth | 1 | No | How many link levels to follow from each root URL (1 = initial page only). |
| Prevent Outside | true | Yes | Stay within the same domain as the root URL. |
| Use Async | true | Yes | Use asynchronous loading for faster crawls. |
| Output Format | Text | Yes | `Text` extracts readable text from HTML; `HTML` returns raw HTML. |
| Timeout | 30 | Yes | Request timeout in seconds. |
| Headers | User-Agent | Yes | HTTP headers to send with requests. |

**Outputs**

| Output | Description |
| --- | --- |
| Data | Fetched page content as data objects (text or HTML per **Output Format**). |
| Message | Combined text from all fetched pages as a single message. |
| DataFrame | Fetched content as a table. |

**Limitations**

- **Network access** — URLs must be reachable from the flow runtime.
- **Crawl scope** — **Max Depth** and **Prevent Outside** control how much of a site is loaded; large sites can produce large outputs.
- **Invalid URLs** — Malformed URLs cause the component to fail.

---

## Webhook

Receives an inbound HTTP request and uses the payload as input to the flow.

**Use in a flow**

1. Add a **Webhook** node to the canvas (typically at the start).
2. Copy the **Endpoint** URL from the component — external systems send requests to this URL to trigger the flow.
3. Connect **Data** output to downstream components.

```
External system ──POST──► Webhook ──► Processing ──► Output
```

When a live request arrives, the request body is passed downstream. Use **Payload** to test with sample JSON without an external caller.

**Parameters**

| Parameter | Hidden | Description |
| --- | --- | --- |
| Endpoint | No | The webhook URL for external systems to call. Copy this value into your integration. |
| Payload | Yes | Sample request body for testing (JSON). Used when no live HTTP request is received. |
| cURL | Yes | Example cURL command for the webhook endpoint. |

**Outputs**

| Output | Description |
| --- | --- |
| Data | The webhook payload as a data object. JSON bodies are parsed; non-JSON text is wrapped in a `payload` field. |

**Limitations**

- **Public endpoint** — The webhook URL must be reachable by the calling system.
- **Flow must be deployed** — Inbound webhooks require a published or running flow endpoint.
- **Payload format** — Downstream components receive whatever structure the caller sends.