Data components connect the flow to external data sources — HTTP APIs, files, databases, object storage, and inbound webhooks.
API Request
Makes an HTTP request to an external URL and passes the response downstream.
Use in a flow
- Add an API Request node to the canvas.
- Enter one or more URLs and select the HTTP Method.
- Optionally configure headers, query parameters, and body.
- Connect the Data or DataFrame output to the next step.
API Request (Data) ──► Processing / Agent / Output
You can also paste a cURL command to auto-fill request fields (enable Use cURL).
Parameters — Request
| Parameter | Default | Hidden | Description |
|---|---|---|---|
| URLs | — | No | One or more URLs to call, separated by commas. |
| Method | GET | No | HTTP method: GET, POST, PATCH, PUT, or DELETE. |
| Use cURL | false | No | When enabled, populate fields from the cURL input instead of manual entry. |
| cURL | — | Yes | Paste a cURL command to fill headers, body, and method. Visible when Use cURL is enabled. |
| Query Parameters | — | Yes | Query string parameters appended to the URL. |
| Body | — | Yes | Request body as key-value pairs (for POST, PATCH, PUT). |
| Headers | User-Agent default | Yes | HTTP headers to send with the request. |
Parameters — Options
| Parameter | Default | Hidden | Description |
|---|---|---|---|
| Timeout | 30 | Yes | Request timeout in seconds. |
| Follow Redirects | true | Yes | Whether to follow HTTP redirects. |
| Save to File | false | Yes | Save the API response to a temporary file. |
| Include HTTPx Metadata | false | Yes | Include headers, status code, and redirect history in the output. |
| Enable Retry | false | Yes | Retry failed requests. |
| Max Retries | 3 | Yes | Maximum retry attempts. |
| Backoff Factor | 2.0 | Yes | Multiplier for delay between retries. |
| Min Backoff (seconds) | 1.0 | Yes | Minimum wait between retries. |
| Max Backoff (seconds) | 10.0 | Yes | Maximum wait between retries. |
| Retry Status Codes | 404,408,429,500,502,503,504 | Yes | HTTP status codes that trigger a retry. |
| Workspace ID | — | Yes | Workspace ID for secret resolution in URLs and headers. |
Outputs
| Output | Description |
|---|---|
| Data | The API response as data objects. |
| DataFrame | The API response as a table. |
Limitations
- External network — The URL must be reachable from the flow runtime.
- Secrets — Use workspace secret resolution where supported; do not hard-code credentials in flows.
File
Loads a file (uploaded or from a URL) and makes its content available to the flow.
Use in a flow
- Add a File node to the canvas.
- Upload files under Files, or connect a Server File Path from another component.
- Connect Data, DataFrame, or Message output to downstream steps.
File (Data) ──► CharacterTextSplitter / Agent / Embeddings
Parameters
| Parameter | Default | Hidden | Description |
|---|---|---|---|
| Files | — | No | Upload one or more files, or provide file paths. |
| Server File Path | — | Yes | Connect a Data or Message object with a server file path instead of uploading. |
| Separator | \n\n | Yes | Separator between multiple file contents when output as Message. |
| Processing Concurrency | 1 | Yes | Number of files to process in parallel when multiple files are loaded. |
| Silent Errors | false | Yes | If enabled, file errors are logged but do not stop the flow. |
| Delete Server File After Processing | true | Yes | Remove server-side files after processing. |
| Ignore Unsupported Extensions | true | Yes | Skip files with unsupported extensions. |
| Ignore Unspecified Files | false | Yes | Skip Data objects without a file path property. |
Outputs
| Output | Description |
|---|---|
| Data | Parsed file content as data objects. |
| DataFrame | File content as a table. |
| Message | File content as a single message string. |
Limitations
- Text files — Primarily supports text-based file types (txt, csv, json, md, and similar).
- Archives — Zip and tar bundles are supported; contents are extracted and processed.
MySQL
Executes a query against a MySQL database and returns the results.
Use in a flow
- Add a MySQL node to the canvas.
- Enter the Database URL for your MySQL instance.
- Enter the MySQL Query to run.
- Connect Query Results to downstream components.
MySQL (Query Results) ──► Agent / Output / Processing
mysql:// URLs are automatically converted to use the PyMySQL driver.
Parameters
| Parameter | Default | Hidden | Description |
|---|---|---|---|
| Database URL | — | No | MySQL connection URL (e.g., mysql://user:pass@host:3306/dbname). |
| MySQL Query | — | No | The SQL query to execute. |
| Schema | — | Yes | Optional schema or database name (can also be set in the URL). |
| Include Columns | true | Yes | Include column names in the result. |
| Add Error Details | false | Yes | Include detailed error information in the result when a query fails. |
| Pool Size | 5 | Yes | Number of connections kept in the pool. |
| Max Overflow | 10 | Yes | Extra connections allowed beyond the pool size. |
Outputs
| Output | Description |
|---|---|
| Query Results | Query results as a DataFrame. |
Limitations
- MySQL only — Requires a valid MySQL connection URL.
- Credentials in URL — Connection strings may contain passwords; handle securely.
- Query permissions — Results depend on the database user’s access rights.
PostgreSQL
Executes a query against a PostgreSQL database and returns the results.
Use in a flow
- Add a PostgreSQL node to the canvas.
- Enter the Database URL for your PostgreSQL instance.
- Enter the Postgresql Query to run.
- Connect Query Results to downstream components.
PostgreSQL (Query Results) ──► Agent / Output / Processing
postgresql:// URLs are automatically converted to use the psycopg driver.
Parameters
| Parameter | Default | Hidden | Description |
|---|---|---|---|
| Database URL | — | No | PostgreSQL connection URL (e.g., postgresql://user:pass@host:5432/dbname). |
| Postgresql Query | — | No | The SQL query to execute. |
| Schema | — | Yes | Optional schema name (can also be set in the URL). |
| Include Columns | true | Yes | Include column names in the result. |
| Add Error Details | false | Yes | Include detailed error information in the result when a query fails. |
| Pool Size | 5 | Yes | Number of connections kept in the pool. |
| Max Overflow | 10 | Yes | Extra connections allowed beyond the pool size. |
Outputs
| Output | Description |
|---|---|
| Query Results | Query results as a DataFrame. |
Limitations
- PostgreSQL only — Requires a valid PostgreSQL connection URL.
- Credentials in URL — Connection strings may contain passwords; handle securely.
- Query permissions — Results depend on the database user’s access rights.
Redshift
Executes a query against an Amazon Redshift data warehouse and returns the results.
Use in a flow
- Add a Redshift node to the canvas.
- Enter the Database URL for your Redshift cluster.
- Enter the Redshift Query to run.
- Connect Query Results to downstream components.
Redshift (Query Results) ──► Agent / Output / Processing
redshift:// URLs are automatically converted to use the redshift_connector driver.
Parameters
| Parameter | Default | Hidden | Description |
|---|---|---|---|
| Database URL | — | No | Redshift connection URL (e.g., redshift://user:pass@host:5439/dbname). |
| Redshift Query | — | No | The SQL query to execute. |
| Schema | — | Yes | Optional schema name (can also be set in the URL). |
| Include Columns | true | Yes | Include column names in the result. |
| Add Error Details | false | Yes | Include detailed error information in the result when a query fails. |
| Pool Size | 5 | Yes | Number of connections kept in the pool. |
| Max Overflow | 10 | Yes | Extra connections allowed beyond the pool size. |
Outputs
| Output | Description |
|---|---|
| Query Results | Query results as a DataFrame. |
Limitations
- Redshift only — Requires a valid Redshift connection URL and network access to the cluster.
- Credentials in URL — Connection strings may contain passwords; handle securely.
- Query permissions — Results depend on the database user’s access rights.
S3 Operations
Reads from or writes to an Amazon S3 bucket by listing buckets and objects and generating pre-signed URLs for secure access.
Use in a flow
- Add an S3 Operations node to the canvas.
- Select AWS Region, S3 Bucket, and one or more Object Keys.
- Choose Operations (GET, PUT, POST, DELETE) for the pre-signed URLs you need.
- Connect Result output to downstream steps.
S3 Operations (Result) ──► API Request / Agent / Output
Pre-signed URLs allow time-limited access to S3 objects without embedding long-lived credentials in the flow.
Parameters
| Parameter | Default | Hidden | Description |
|---|---|---|---|
| AWS Region | us-east-1 | No | AWS region where the bucket is located. |
| S3 Bucket | — | No | Select a bucket from your AWS account. Refresh to reload the list. |
| Object Keys | — | No | Select one or more objects in the bucket. |
| Operations | GET | No | One or more operations for the generated URLs: GET, PUT, POST, or DELETE. |
| Schema JSON (Optional) | — | Yes | JSON override for region, bucket, object keys, and operations. When set, manual inputs are ignored. |
Outputs
| Output | Description |
|---|---|
| Result | Pre-signed URLs and metadata (bucket, objects, operations, region) as a data object. |
Limitations
- AWS credentials — IAM permissions must allow listing and pre-signing for the selected bucket.
- Pre-signed URLs — Access is via generated URLs, not direct in-flow object streaming.
- URL expiry — Pre-signed URLs use a fixed expiration configured by the runtime.
SQL Query
Executes a SQL query against a configured database connection and returns the results.
Use in a flow
- Add a SQL Query node to the canvas.
- Enter the Database URL for your database.
- Enter the SQL Query to run.
- Connect Query Results to downstream components.
SQL Query (Query Results) ──► Agent / Output / Processing
Works with any database supported by SQLAlchemy when a valid connection URL is provided.
Parameters
| Parameter | Default | Hidden | Description |
|---|---|---|---|
| Database URL | — | No | SQLAlchemy-compatible connection URL. |
| SQL Query | — | No | The SQL query to execute. |
| Include Columns | true | Yes | Include column names in the result. |
| Add Error | false | Yes | Append error details to the result when a query fails. |
Outputs
| Output | Description |
|---|---|
| Query Results | Query results as a DataFrame. |
Limitations
- Connection URL — Must be a valid SQLAlchemy database URL for your engine.
- Credentials — Connection strings may contain passwords; handle securely.
- Query permissions — Results depend on the database user’s access rights.
URL
Fetches content from a URL and returns the response. Can crawl linked pages recursively from a root URL.
Use in a flow
- Add a URL node to the canvas.
- Enter one or more URLs to fetch.
- Set Max Depth to control how far linked pages are followed.
- Connect Data, Message, or DataFrame output downstream.
URL (Message) ──► CharacterTextSplitter / Agent / Embeddings
At depth 1, only the initial page is loaded. Higher depths follow links on those pages.
Parameters
| Parameter | Default | Hidden | Description |
|---|---|---|---|
| URLs | — | No | One or more root URLs to crawl. |
| Max Depth | 1 | No | How many link levels to follow from each root URL (1 = initial page only). |
| Prevent Outside | true | Yes | Stay within the same domain as the root URL. |
| Use Async | true | Yes | Use asynchronous loading for faster crawls. |
| Output Format | Text | Yes | Text extracts readable text from HTML; HTML returns raw HTML. |
| Timeout | 30 | Yes | Request timeout in seconds. |
| Headers | User-Agent | Yes | HTTP headers to send with requests. |
Outputs
| Output | Description |
|---|---|
| Data | Fetched page content as data objects (text or HTML per Output Format). |
| Message | Combined text from all fetched pages as a single message. |
| DataFrame | Fetched content as a table. |
Limitations
- Network access — URLs must be reachable from the flow runtime.
- Crawl scope — Max Depth and Prevent Outside control how much of a site is loaded; large sites can produce large outputs.
- Invalid URLs — Malformed URLs cause the component to fail.
Webhook
Receives an inbound HTTP request and uses the payload as input to the flow.
Use in a flow
- Add a Webhook node to the canvas (typically at the start).
- Copy the Endpoint URL from the component — external systems send requests to this URL to trigger the flow.
- Connect Data output to downstream components.
External system ──POST──► Webhook ──► Processing ──► Output
When a live request arrives, the request body is passed downstream. Use Payload to test with sample JSON without an external caller.
Parameters
| Parameter | Hidden | Description |
|---|---|---|
| Endpoint | No | The webhook URL for external systems to call. Copy this value into your integration. |
| Payload | Yes | Sample request body for testing (JSON). Used when no live HTTP request is received. |
| cURL | Yes | Example cURL command for the webhook endpoint. |
Outputs
| Output | Description |
|---|---|
| Data | The webhook payload as a data object. JSON bodies are parsed; non-JSON text is wrapped in a payload field. |
Limitations
- Public endpoint — The webhook URL must be reachable by the calling system.
- Flow must be deployed — Inbound webhooks require a published or running flow endpoint.
- Payload format — Downstream components receive whatever structure the caller sends.