Data ==== Data components connect the flow to external data sources — HTTP APIs, files, databases, object storage, and inbound webhooks. ---- .. _component-ref-api-request: API Request ----------- Makes an HTTP request to an external URL and passes the response downstream. **Use in a flow** 1. Add an **API Request** node to the canvas. 2. Enter one or more **URLs** and select the HTTP **Method**. 3. Optionally configure headers, query parameters, and body. 4. Connect the **Data** or **DataFrame** output to the next step. .. code-block:: text API Request (Data) ──► Processing / Agent / Output You can also paste a **cURL** command to auto-fill request fields (enable **Use cURL**). **Parameters — Request** .. list-table:: :widths: 30 10 10 50 :header-rows: 1 * - Parameter - Default - Hidden - Description * - URLs - — - No - One or more URLs to call, separated by commas. * - Method - GET - No - HTTP method: GET, POST, PATCH, PUT, or DELETE. * - Use cURL - false - No - When enabled, populate fields from the **cURL** input instead of manual entry. * - cURL - — - Yes - Paste a cURL command to fill headers, body, and method. Visible when **Use cURL** is enabled. * - Query Parameters - — - Yes - Query string parameters appended to the URL. * - Body - — - Yes - Request body as key-value pairs (for POST, PATCH, PUT). * - Headers - User-Agent default - Yes - HTTP headers to send with the request. **Parameters — Options** .. list-table:: :widths: 30 10 10 50 :header-rows: 1 * - Parameter - Default - Hidden - Description * - Timeout - 30 - Yes - Request timeout in seconds. * - Follow Redirects - true - Yes - Whether to follow HTTP redirects. * - Save to File - false - Yes - Save the API response to a temporary file. * - Include HTTPx Metadata - false - Yes - Include headers, status code, and redirect history in the output. * - Enable Retry - false - Yes - Retry failed requests. * - Max Retries - 3 - Yes - Maximum retry attempts. * - Backoff Factor - 2.0 - Yes - Multiplier for delay between retries. * - Min Backoff (seconds) - 1.0 - Yes - Minimum wait between retries. * - Max Backoff (seconds) - 10.0 - Yes - Maximum wait between retries. * - Retry Status Codes - 404,408,429,500,502,503,504 - Yes - HTTP status codes that trigger a retry. * - Workspace ID - — - Yes - Workspace ID for secret resolution in URLs and headers. **Outputs** .. list-table:: :widths: 25 75 :header-rows: 1 * - Output - Description * - Data - The API response as data objects. * - DataFrame - The API response as a table. **Limitations** - **External network** — The URL must be reachable from the flow runtime. - **Secrets** — Use workspace secret resolution where supported; do not hard-code credentials in flows. ---- .. _component-ref-file: File ---- Loads a file (uploaded or from a URL) and makes its content available to the flow. **Use in a flow** 1. Add a **File** node to the canvas. 2. Upload files under **Files**, or connect a **Server File Path** from another component. 3. Connect **Data**, **DataFrame**, or **Message** output to downstream steps. .. code-block:: text File (Data) ──► CharacterTextSplitter / Agent / Embeddings **Parameters** .. list-table:: :widths: 30 10 10 50 :header-rows: 1 * - Parameter - Default - Hidden - Description * - Files - — - No - Upload one or more files, or provide file paths. * - Server File Path - — - Yes - Connect a Data or Message object with a server file path instead of uploading. * - Separator - \\n\\n - Yes - Separator between multiple file contents when output as Message. * - Processing Concurrency - 1 - Yes - Number of files to process in parallel when multiple files are loaded. * - Silent Errors - false - Yes - If enabled, file errors are logged but do not stop the flow. * - Delete Server File After Processing - true - Yes - Remove server-side files after processing. * - Ignore Unsupported Extensions - true - Yes - Skip files with unsupported extensions. * - Ignore Unspecified Files - false - Yes - Skip Data objects without a file path property. **Outputs** .. list-table:: :widths: 25 75 :header-rows: 1 * - Output - Description * - Data - Parsed file content as data objects. * - DataFrame - File content as a table. * - Message - File content as a single message string. **Limitations** - **Text files** — Primarily supports text-based file types (txt, csv, json, md, and similar). - **Archives** — Zip and tar bundles are supported; contents are extracted and processed. ---- .. _component-ref-mysql: MySQL ----- Executes a query against a MySQL database and returns the results. **Use in a flow** 1. Add a **MySQL** node to the canvas. 2. Enter the **Database URL** for your MySQL instance. 3. Enter the **MySQL Query** to run. 4. Connect **Query Results** to downstream components. .. code-block:: text MySQL (Query Results) ──► Agent / Output / Processing ``mysql://`` URLs are automatically converted to use the PyMySQL driver. **Parameters** .. list-table:: :widths: 30 10 10 50 :header-rows: 1 * - Parameter - Default - Hidden - Description * - Database URL - — - No - MySQL connection URL (e.g., ``mysql://user:pass@host:3306/dbname``). * - MySQL Query - — - No - The SQL query to execute. * - Schema - — - Yes - Optional schema or database name (can also be set in the URL). * - Include Columns - true - Yes - Include column names in the result. * - Add Error Details - false - Yes - Include detailed error information in the result when a query fails. * - Pool Size - 5 - Yes - Number of connections kept in the pool. * - Max Overflow - 10 - Yes - Extra connections allowed beyond the pool size. **Outputs** .. list-table:: :widths: 25 75 :header-rows: 1 * - Output - Description * - Query Results - Query results as a DataFrame. **Limitations** - **MySQL only** — Requires a valid MySQL connection URL. - **Credentials in URL** — Connection strings may contain passwords; handle securely. - **Query permissions** — Results depend on the database user's access rights. ---- .. _component-ref-postgresql: PostgreSQL ---------- Executes a query against a PostgreSQL database and returns the results. **Use in a flow** 1. Add a **PostgreSQL** node to the canvas. 2. Enter the **Database URL** for your PostgreSQL instance. 3. Enter the **Postgresql Query** to run. 4. Connect **Query Results** to downstream components. .. code-block:: text PostgreSQL (Query Results) ──► Agent / Output / Processing ``postgresql://`` URLs are automatically converted to use the psycopg driver. **Parameters** .. list-table:: :widths: 30 10 10 50 :header-rows: 1 * - Parameter - Default - Hidden - Description * - Database URL - — - No - PostgreSQL connection URL (e.g., ``postgresql://user:pass@host:5432/dbname``). * - Postgresql Query - — - No - The SQL query to execute. * - Schema - — - Yes - Optional schema name (can also be set in the URL). * - Include Columns - true - Yes - Include column names in the result. * - Add Error Details - false - Yes - Include detailed error information in the result when a query fails. * - Pool Size - 5 - Yes - Number of connections kept in the pool. * - Max Overflow - 10 - Yes - Extra connections allowed beyond the pool size. **Outputs** .. list-table:: :widths: 25 75 :header-rows: 1 * - Output - Description * - Query Results - Query results as a DataFrame. **Limitations** - **PostgreSQL only** — Requires a valid PostgreSQL connection URL. - **Credentials in URL** — Connection strings may contain passwords; handle securely. - **Query permissions** — Results depend on the database user's access rights. ---- .. _component-ref-redshift: Redshift -------- Executes a query against an Amazon Redshift data warehouse and returns the results. **Use in a flow** 1. Add a **Redshift** node to the canvas. 2. Enter the **Database URL** for your Redshift cluster. 3. Enter the **Redshift Query** to run. 4. Connect **Query Results** to downstream components. .. code-block:: text Redshift (Query Results) ──► Agent / Output / Processing ``redshift://`` URLs are automatically converted to use the redshift_connector driver. **Parameters** .. list-table:: :widths: 30 10 10 50 :header-rows: 1 * - Parameter - Default - Hidden - Description * - Database URL - — - No - Redshift connection URL (e.g., ``redshift://user:pass@host:5439/dbname``). * - Redshift Query - — - No - The SQL query to execute. * - Schema - — - Yes - Optional schema name (can also be set in the URL). * - Include Columns - true - Yes - Include column names in the result. * - Add Error Details - false - Yes - Include detailed error information in the result when a query fails. * - Pool Size - 5 - Yes - Number of connections kept in the pool. * - Max Overflow - 10 - Yes - Extra connections allowed beyond the pool size. **Outputs** .. list-table:: :widths: 25 75 :header-rows: 1 * - Output - Description * - Query Results - Query results as a DataFrame. **Limitations** - **Redshift only** — Requires a valid Redshift connection URL and network access to the cluster. - **Credentials in URL** — Connection strings may contain passwords; handle securely. - **Query permissions** — Results depend on the database user's access rights. ---- .. _component-ref-s3-operations: S3 Operations ------------- Reads from or writes to an Amazon S3 bucket by listing buckets and objects and generating pre-signed URLs for secure access. **Use in a flow** 1. Add an **S3 Operations** node to the canvas. 2. Select **AWS Region**, **S3 Bucket**, and one or more **Object Keys**. 3. Choose **Operations** (GET, PUT, POST, DELETE) for the pre-signed URLs you need. 4. Connect **Result** output to downstream steps. .. code-block:: text S3 Operations (Result) ──► API Request / Agent / Output Pre-signed URLs allow time-limited access to S3 objects without embedding long-lived credentials in the flow. **Parameters** .. list-table:: :widths: 30 10 10 50 :header-rows: 1 * - Parameter - Default - Hidden - Description * - AWS Region - us-east-1 - No - AWS region where the bucket is located. * - S3 Bucket - — - No - Select a bucket from your AWS account. Refresh to reload the list. * - Object Keys - — - No - Select one or more objects in the bucket. * - Operations - GET - No - One or more operations for the generated URLs: GET, PUT, POST, or DELETE. * - Schema JSON (Optional) - — - Yes - JSON override for region, bucket, object keys, and operations. When set, manual inputs are ignored. **Outputs** .. list-table:: :widths: 25 75 :header-rows: 1 * - Output - Description * - Result - Pre-signed URLs and metadata (bucket, objects, operations, region) as a data object. **Limitations** - **AWS credentials** — IAM permissions must allow listing and pre-signing for the selected bucket. - **Pre-signed URLs** — Access is via generated URLs, not direct in-flow object streaming. - **URL expiry** — Pre-signed URLs use a fixed expiration configured by the runtime. ---- .. _component-ref-sql-query: SQL Query --------- Executes a SQL query against a configured database connection and returns the results. **Use in a flow** 1. Add a **SQL Query** node to the canvas. 2. Enter the **Database URL** for your database. 3. Enter the **SQL Query** to run. 4. Connect **Query Results** to downstream components. .. code-block:: text SQL Query (Query Results) ──► Agent / Output / Processing Works with any database supported by SQLAlchemy when a valid connection URL is provided. **Parameters** .. list-table:: :widths: 30 10 10 50 :header-rows: 1 * - Parameter - Default - Hidden - Description * - Database URL - — - No - SQLAlchemy-compatible connection URL. * - SQL Query - — - No - The SQL query to execute. * - Include Columns - true - Yes - Include column names in the result. * - Add Error - false - Yes - Append error details to the result when a query fails. **Outputs** .. list-table:: :widths: 25 75 :header-rows: 1 * - Output - Description * - Query Results - Query results as a DataFrame. **Limitations** - **Connection URL** — Must be a valid SQLAlchemy database URL for your engine. - **Credentials** — Connection strings may contain passwords; handle securely. - **Query permissions** — Results depend on the database user's access rights. ---- .. _component-ref-url: URL --- Fetches content from a URL and returns the response. Can crawl linked pages recursively from a root URL. **Use in a flow** 1. Add a **URL** node to the canvas. 2. Enter one or more **URLs** to fetch. 3. Set **Max Depth** to control how far linked pages are followed. 4. Connect **Data**, **Message**, or **DataFrame** output downstream. .. code-block:: text URL (Message) ──► CharacterTextSplitter / Agent / Embeddings At depth 1, only the initial page is loaded. Higher depths follow links on those pages. **Parameters** .. list-table:: :widths: 30 10 10 50 :header-rows: 1 * - Parameter - Default - Hidden - Description * - URLs - — - No - One or more root URLs to crawl. * - Max Depth - 1 - No - How many link levels to follow from each root URL (1 = initial page only). * - Prevent Outside - true - Yes - Stay within the same domain as the root URL. * - Use Async - true - Yes - Use asynchronous loading for faster crawls. * - Output Format - Text - Yes - ``Text`` extracts readable text from HTML; ``HTML`` returns raw HTML. * - Timeout - 30 - Yes - Request timeout in seconds. * - Headers - User-Agent - Yes - HTTP headers to send with requests. **Outputs** .. list-table:: :widths: 25 75 :header-rows: 1 * - Output - Description * - Data - Fetched page content as data objects (text or HTML per **Output Format**). * - Message - Combined text from all fetched pages as a single message. * - DataFrame - Fetched content as a table. **Limitations** - **Network access** — URLs must be reachable from the flow runtime. - **Crawl scope** — **Max Depth** and **Prevent Outside** control how much of a site is loaded; large sites can produce large outputs. - **Invalid URLs** — Malformed URLs cause the component to fail. ---- .. _component-ref-webhook: Webhook ------- Receives an inbound HTTP request and uses the payload as input to the flow. **Use in a flow** 1. Add a **Webhook** node to the canvas (typically at the start). 2. Copy the **Endpoint** URL from the component — external systems send requests to this URL to trigger the flow. 3. Connect **Data** output to downstream components. .. code-block:: text External system ──POST──► Webhook ──► Processing ──► Output When a live request arrives, the request body is passed downstream. Use **Payload** to test with sample JSON without an external caller. **Parameters** .. list-table:: :widths: 30 10 60 :header-rows: 1 * - Parameter - Hidden - Description * - Endpoint - No - The webhook URL for external systems to call. Copy this value into your integration. * - Payload - Yes - Sample request body for testing (JSON). Used when no live HTTP request is received. * - cURL - Yes - Example cURL command for the webhook endpoint. **Outputs** .. list-table:: :widths: 25 75 :header-rows: 1 * - Output - Description * - Data - The webhook payload as a data object. JSON bodies are parsed; non-JSON text is wrapped in a ``payload`` field. **Limitations** - **Public endpoint** — The webhook URL must be reachable by the calling system. - **Flow must be deployed** — Inbound webhooks require a published or running flow endpoint. - **Payload format** — Downstream components receive whatever structure the caller sends.