Data
Data components connect the flow to external data sources — HTTP APIs, files, databases, object storage, and inbound webhooks.
API Request
Makes an HTTP request to an external URL and passes the response downstream.
Use in a flow
Add an API Request node to the canvas.
Enter one or more URLs and select the HTTP Method.
Optionally configure headers, query parameters, and body.
Connect the Data or DataFrame output to the next step.
API Request (Data) ──► Processing / Agent / Output
You can also paste a cURL command to auto-fill request fields (enable Use cURL).
Parameters — Request
Parameter |
Default |
Hidden |
Description |
|---|---|---|---|
URLs |
— |
No |
One or more URLs to call, separated by commas. |
Method |
GET |
No |
HTTP method: GET, POST, PATCH, PUT, or DELETE. |
Use cURL |
false |
No |
When enabled, populate fields from the cURL input instead of manual entry. |
cURL |
— |
Yes |
Paste a cURL command to fill headers, body, and method. Visible when Use cURL is enabled. |
Query Parameters |
— |
Yes |
Query string parameters appended to the URL. |
Body |
— |
Yes |
Request body as key-value pairs (for POST, PATCH, PUT). |
Headers |
User-Agent default |
Yes |
HTTP headers to send with the request. |
Parameters — Options
Parameter |
Default |
Hidden |
Description |
|---|---|---|---|
Timeout |
30 |
Yes |
Request timeout in seconds. |
Follow Redirects |
true |
Yes |
Whether to follow HTTP redirects. |
Save to File |
false |
Yes |
Save the API response to a temporary file. |
Include HTTPx Metadata |
false |
Yes |
Include headers, status code, and redirect history in the output. |
Enable Retry |
false |
Yes |
Retry failed requests. |
Max Retries |
3 |
Yes |
Maximum retry attempts. |
Backoff Factor |
2.0 |
Yes |
Multiplier for delay between retries. |
Min Backoff (seconds) |
1.0 |
Yes |
Minimum wait between retries. |
Max Backoff (seconds) |
10.0 |
Yes |
Maximum wait between retries. |
Retry Status Codes |
404,408,429,500,502,503,504 |
Yes |
HTTP status codes that trigger a retry. |
Workspace ID |
— |
Yes |
Workspace ID for secret resolution in URLs and headers. |
Outputs
Output |
Description |
|---|---|
Data |
The API response as data objects. |
DataFrame |
The API response as a table. |
Limitations
External network — The URL must be reachable from the flow runtime.
Secrets — Use workspace secret resolution where supported; do not hard-code credentials in flows.
File
Loads a file (uploaded or from a URL) and makes its content available to the flow.
Use in a flow
Add a File node to the canvas.
Upload files under Files, or connect a Server File Path from another component.
Connect Data, DataFrame, or Message output to downstream steps.
File (Data) ──► CharacterTextSplitter / Agent / Embeddings
Parameters
Parameter |
Default |
Hidden |
Description |
|---|---|---|---|
Files |
— |
No |
Upload one or more files, or provide file paths. |
Server File Path |
— |
Yes |
Connect a Data or Message object with a server file path instead of uploading. |
Separator |
\n\n |
Yes |
Separator between multiple file contents when output as Message. |
Processing Concurrency |
1 |
Yes |
Number of files to process in parallel when multiple files are loaded. |
Silent Errors |
false |
Yes |
If enabled, file errors are logged but do not stop the flow. |
Delete Server File After Processing |
true |
Yes |
Remove server-side files after processing. |
Ignore Unsupported Extensions |
true |
Yes |
Skip files with unsupported extensions. |
Ignore Unspecified Files |
false |
Yes |
Skip Data objects without a file path property. |
Outputs
Output |
Description |
|---|---|
Data |
Parsed file content as data objects. |
DataFrame |
File content as a table. |
Message |
File content as a single message string. |
Limitations
Text files — Primarily supports text-based file types (txt, csv, json, md, and similar).
Archives — Zip and tar bundles are supported; contents are extracted and processed.
MySQL
Executes a query against a MySQL database and returns the results.
Use in a flow
Add a MySQL node to the canvas.
Enter the Database URL for your MySQL instance.
Enter the MySQL Query to run.
Connect Query Results to downstream components.
MySQL (Query Results) ──► Agent / Output / Processing
mysql:// URLs are automatically converted to use the PyMySQL driver.
Parameters
Parameter |
Default |
Hidden |
Description |
|---|---|---|---|
Database URL |
— |
No |
MySQL connection URL (e.g., |
MySQL Query |
— |
No |
The SQL query to execute. |
Schema |
— |
Yes |
Optional schema or database name (can also be set in the URL). |
Include Columns |
true |
Yes |
Include column names in the result. |
Add Error Details |
false |
Yes |
Include detailed error information in the result when a query fails. |
Pool Size |
5 |
Yes |
Number of connections kept in the pool. |
Max Overflow |
10 |
Yes |
Extra connections allowed beyond the pool size. |
Outputs
Output |
Description |
|---|---|
Query Results |
Query results as a DataFrame. |
Limitations
MySQL only — Requires a valid MySQL connection URL.
Credentials in URL — Connection strings may contain passwords; handle securely.
Query permissions — Results depend on the database user’s access rights.
PostgreSQL
Executes a query against a PostgreSQL database and returns the results.
Use in a flow
Add a PostgreSQL node to the canvas.
Enter the Database URL for your PostgreSQL instance.
Enter the Postgresql Query to run.
Connect Query Results to downstream components.
PostgreSQL (Query Results) ──► Agent / Output / Processing
postgresql:// URLs are automatically converted to use the psycopg driver.
Parameters
Parameter |
Default |
Hidden |
Description |
|---|---|---|---|
Database URL |
— |
No |
PostgreSQL connection URL (e.g., |
Postgresql Query |
— |
No |
The SQL query to execute. |
Schema |
— |
Yes |
Optional schema name (can also be set in the URL). |
Include Columns |
true |
Yes |
Include column names in the result. |
Add Error Details |
false |
Yes |
Include detailed error information in the result when a query fails. |
Pool Size |
5 |
Yes |
Number of connections kept in the pool. |
Max Overflow |
10 |
Yes |
Extra connections allowed beyond the pool size. |
Outputs
Output |
Description |
|---|---|
Query Results |
Query results as a DataFrame. |
Limitations
PostgreSQL only — Requires a valid PostgreSQL connection URL.
Credentials in URL — Connection strings may contain passwords; handle securely.
Query permissions — Results depend on the database user’s access rights.
Redshift
Executes a query against an Amazon Redshift data warehouse and returns the results.
Use in a flow
Add a Redshift node to the canvas.
Enter the Database URL for your Redshift cluster.
Enter the Redshift Query to run.
Connect Query Results to downstream components.
Redshift (Query Results) ──► Agent / Output / Processing
redshift:// URLs are automatically converted to use the redshift_connector driver.
Parameters
Parameter |
Default |
Hidden |
Description |
|---|---|---|---|
Database URL |
— |
No |
Redshift connection URL (e.g., |
Redshift Query |
— |
No |
The SQL query to execute. |
Schema |
— |
Yes |
Optional schema name (can also be set in the URL). |
Include Columns |
true |
Yes |
Include column names in the result. |
Add Error Details |
false |
Yes |
Include detailed error information in the result when a query fails. |
Pool Size |
5 |
Yes |
Number of connections kept in the pool. |
Max Overflow |
10 |
Yes |
Extra connections allowed beyond the pool size. |
Outputs
Output |
Description |
|---|---|
Query Results |
Query results as a DataFrame. |
Limitations
Redshift only — Requires a valid Redshift connection URL and network access to the cluster.
Credentials in URL — Connection strings may contain passwords; handle securely.
Query permissions — Results depend on the database user’s access rights.
S3 Operations
Reads from or writes to an Amazon S3 bucket by listing buckets and objects and generating pre-signed URLs for secure access.
Use in a flow
Add an S3 Operations node to the canvas.
Select AWS Region, S3 Bucket, and one or more Object Keys.
Choose Operations (GET, PUT, POST, DELETE) for the pre-signed URLs you need.
Connect Result output to downstream steps.
S3 Operations (Result) ──► API Request / Agent / Output
Pre-signed URLs allow time-limited access to S3 objects without embedding long-lived credentials in the flow.
Parameters
Parameter |
Default |
Hidden |
Description |
|---|---|---|---|
AWS Region |
us-east-1 |
No |
AWS region where the bucket is located. |
S3 Bucket |
— |
No |
Select a bucket from your AWS account. Refresh to reload the list. |
Object Keys |
— |
No |
Select one or more objects in the bucket. |
Operations |
GET |
No |
One or more operations for the generated URLs: GET, PUT, POST, or DELETE. |
Schema JSON (Optional) |
— |
Yes |
JSON override for region, bucket, object keys, and operations. When set, manual inputs are ignored. |
Outputs
Output |
Description |
|---|---|
Result |
Pre-signed URLs and metadata (bucket, objects, operations, region) as a data object. |
Limitations
AWS credentials — IAM permissions must allow listing and pre-signing for the selected bucket.
Pre-signed URLs — Access is via generated URLs, not direct in-flow object streaming.
URL expiry — Pre-signed URLs use a fixed expiration configured by the runtime.
SQL Query
Executes a SQL query against a configured database connection and returns the results.
Use in a flow
Add a SQL Query node to the canvas.
Enter the Database URL for your database.
Enter the SQL Query to run.
Connect Query Results to downstream components.
SQL Query (Query Results) ──► Agent / Output / Processing
Works with any database supported by SQLAlchemy when a valid connection URL is provided.
Parameters
Parameter |
Default |
Hidden |
Description |
|---|---|---|---|
Database URL |
— |
No |
SQLAlchemy-compatible connection URL. |
SQL Query |
— |
No |
The SQL query to execute. |
Include Columns |
true |
Yes |
Include column names in the result. |
Add Error |
false |
Yes |
Append error details to the result when a query fails. |
Outputs
Output |
Description |
|---|---|
Query Results |
Query results as a DataFrame. |
Limitations
Connection URL — Must be a valid SQLAlchemy database URL for your engine.
Credentials — Connection strings may contain passwords; handle securely.
Query permissions — Results depend on the database user’s access rights.
URL
Fetches content from a URL and returns the response. Can crawl linked pages recursively from a root URL.
Use in a flow
Add a URL node to the canvas.
Enter one or more URLs to fetch.
Set Max Depth to control how far linked pages are followed.
Connect Data, Message, or DataFrame output downstream.
URL (Message) ──► CharacterTextSplitter / Agent / Embeddings
At depth 1, only the initial page is loaded. Higher depths follow links on those pages.
Parameters
Parameter |
Default |
Hidden |
Description |
|---|---|---|---|
URLs |
— |
No |
One or more root URLs to crawl. |
Max Depth |
1 |
No |
How many link levels to follow from each root URL (1 = initial page only). |
Prevent Outside |
true |
Yes |
Stay within the same domain as the root URL. |
Use Async |
true |
Yes |
Use asynchronous loading for faster crawls. |
Output Format |
Text |
Yes |
|
Timeout |
30 |
Yes |
Request timeout in seconds. |
Headers |
User-Agent |
Yes |
HTTP headers to send with requests. |
Outputs
Output |
Description |
|---|---|
Data |
Fetched page content as data objects (text or HTML per Output Format). |
Message |
Combined text from all fetched pages as a single message. |
DataFrame |
Fetched content as a table. |
Limitations
Network access — URLs must be reachable from the flow runtime.
Crawl scope — Max Depth and Prevent Outside control how much of a site is loaded; large sites can produce large outputs.
Invalid URLs — Malformed URLs cause the component to fail.
Webhook
Receives an inbound HTTP request and uses the payload as input to the flow.
Use in a flow
Add a Webhook node to the canvas (typically at the start).
Copy the Endpoint URL from the component — external systems send requests to this URL to trigger the flow.
Connect Data output to downstream components.
External system ──POST──► Webhook ──► Processing ──► Output
When a live request arrives, the request body is passed downstream. Use Payload to test with sample JSON without an external caller.
Parameters
Parameter |
Hidden |
Description |
|---|---|---|
Endpoint |
No |
The webhook URL for external systems to call. Copy this value into your integration. |
Payload |
Yes |
Sample request body for testing (JSON). Used when no live HTTP request is received. |
cURL |
Yes |
Example cURL command for the webhook endpoint. |
Outputs
Output |
Description |
|---|---|
Data |
The webhook payload as a data object. JSON bodies are parsed; non-JSON text is wrapped in a |
Limitations
Public endpoint — The webhook URL must be reachable by the calling system.
Flow must be deployed — Inbound webhooks require a published or running flow endpoint.
Payload format — Downstream components receive whatever structure the caller sends.