Processing
==========

Processing components transform, filter, parse, and route data as it moves through the flow.

----

.. _component-ref-combine-data:

Combine Data
------------

Merges outputs from multiple upstream components into a single data object.

**Use in a flow**

1. Add a **Combine Data** node to the canvas.
2. Connect at least two **Data Inputs** from upstream components.
3. Choose an **Operation Type**.
4. Connect **DataFrame** output downstream.

.. code-block:: text

   Component A ──► Data Inputs ──┐
   Component B ──► Data Inputs ──┼──► Combine Data (DataFrame) ──► ...
   Component C ──► Data Inputs ──┘

**Parameters**

.. list-table::
   :widths: 30 10 60
   :header-rows: 1

   * - Parameter
     - Default
     - Description
   * - Data Inputs
     - —
     - Two or more Data objects to combine. Required.
   * - Operation Type
     - Concatenate
     - How to merge the inputs: **Concatenate**, **Append**, **Merge**, or **Join**.

**Operation types**

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Operation
     - Behavior
   * - Concatenate
     - Merge keys from all inputs into one row. Duplicate string keys are joined with newlines.
   * - Append
     - Each input becomes its own row in the output table.
   * - Merge
     - Merge keys; duplicate string values become lists.
   * - Join
     - Merge keys; duplicate keys from later inputs are renamed with a suffix (``_doc2``, ``_doc3``, etc.).

**Outputs**

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Output
     - Description
   * - DataFrame
     - Combined result as a table. Returns empty if fewer than two inputs are connected.

**Limitations**

- **Minimum inputs** — Requires at least two Data inputs to produce a result.
- **Single output type** — Output is always a DataFrame, not a raw Data object.

----

.. _component-ref-data-to-dataframe:

Data → DataFrame
----------------

Converts raw data into a tabular DataFrame structure for further processing.

**Use in a flow**

1. Add a **Data → DataFrame** node to the canvas.
2. Connect one or more **Data** objects to **Data or Data List**.
3. Connect **DataFrame** output to downstream steps.

.. code-block:: text

   API Request (Data) ──► Data → DataFrame ──► DataFrame Operations

**Parameters**

.. list-table::
   :widths: 30 70
   :header-rows: 1

   * - Parameter
     - Description
   * - Data or Data List
     - One or more Data objects to convert. Accepts a single Data object or a list.

**Outputs**

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Output
     - Description
   * - DataFrame
     - A table with one row per Data object.

**Limitations**

- **Data input only** — Input must be Data objects; other types raise an error.
- **Column names** — Column names come from keys in each Data object's data fields.

----

.. _component-ref-data-operations:

Data Operations
---------------

Applies transformations to structured data — select keys, combine, filter values, append or update fields, remove keys, rename keys, and evaluate literal values.

**Use in a flow**

1. Add a **Data Operations** node to the canvas.
2. Connect a **Data** input.
3. Select one **Operation** from the list.
4. Fill in the fields that appear for that operation.
5. Connect **Data** output downstream.

.. code-block:: text

   Webhook (Data) ──► Data Operations ──► Agent / Output

**Parameters**

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Parameter
     - Description
   * - Data
     - The Data object (or list of Data objects for **Combine**) to transform. Required.
   * - Operations
     - The transformation to apply. Choose one: **Select Keys**, **Literal Eval**, **Combine**, **Filter Values**, **Append or Update**, **Remove Keys**, or **Rename Keys**.

Fields below appear based on the selected operation (open **Controls** to access them):

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Operation
     - Additional fields
   * - Select Keys
     - **Select Keys** — list of keys to keep.
   * - Literal Eval
     - Evaluates string values that look like Python literals (lists, dicts, numbers, booleans).
   * - Combine
     - Requires multiple Data inputs. Merges keys from all inputs into one Data object.
   * - Filter Values
     - **Filter Key**, **Comparison Operator** (equals, not equals, contains, starts with, ends with), **Filter Values**.
   * - Append or Update
     - **Append or Update** — key-value pairs to add or overwrite.
   * - Remove Keys
     - **Remove Keys** — list of keys to delete.
   * - Rename Keys
     - **Rename Keys** — map of old key → new key.

**Outputs**

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Output
     - Description
   * - Data
     - The transformed Data object.

**Limitations**

- **One operation at a time** — Only a single operation runs per execution.
- **Combine** — Requires multiple Data inputs wired to **Data**.
- **Filter Values** — The target key must contain a list of dictionaries.

----

.. _component-ref-dataframe-operations:

DataFrame Operations
--------------------

Applies operations on a DataFrame — sort, filter, select columns, add or drop columns, rename columns, replace values, and take head or tail rows.

**Use in a flow**

1. Add a **DataFrame Operations** node to the canvas.
2. Connect a **DataFrame** input.
3. Select an **Operation**.
4. Fill in the fields that appear for that operation.
5. Connect **DataFrame** output downstream.

.. code-block:: text

   Data → DataFrame ──► DataFrame Operations ──► Agent / Output

**Parameters**

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Parameter
     - Description
   * - DataFrame
     - The input table to operate on.
   * - Operation
     - The operation to perform.

**Available operations** (open **Controls** to access operation-specific fields):

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Operation
     - Fields shown
   * - Filter
     - **Column Name**, **Filter Value** — keep rows where the column equals the value.
   * - Sort
     - **Column Name**, **Sort Ascending**.
   * - Drop Column
     - **Column Name**.
   * - Rename Column
     - **Column Name**, **New Column Name**.
   * - Add Column
     - **New Column Name**, **New Column Value** — same value for every row.
   * - Select Columns
     - **Columns to Select** — list of column names to keep.
   * - Head
     - **Number of Rows** — first N rows (default 5).
   * - Tail
     - **Number of Rows** — last N rows (default 5).
   * - Replace Value
     - **Column Name**, **Value to Replace**, **Replacement Value**.

**Outputs**

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Output
     - Description
   * - DataFrame
     - The resulting table after the operation.

**Limitations**

- **One operation at a time** — Each run applies a single selected operation.
- **Exact match filter** — **Filter** uses equality comparison only.

----

.. _component-ref-document-parser:

Document Parser
---------------

Extracts text and structure from documents (PDF, DOCX, PPTX, XLSX, Markdown, and images) using configurable parsing strategies.

**Use in a flow**

1. Add a **Document Parser** node to the canvas.
2. Connect an **Input Message** that includes file attachments (for example, from **Input**).
3. Choose a **Parsing Strategy**.
4. Connect **Parsed Content** downstream.

.. code-block:: text

   Input (Message with files) ──► Document Parser ──► Agent / Split Text

**Parameters**

.. list-table::
   :widths: 30 10 60
   :header-rows: 1

   * - Parameter
     - Default
     - Description
   * - Input Message
     - —
     - Message containing file paths to parse. Required.
   * - Parsing Strategy
     - LLM
     - **LLM** (AI model), **Unstructured** (unstructured.io), or **Textract** (AWS Textract).

When **LLM** is selected, model fields appear:

.. list-table::
   :widths: 30 10 60
   :header-rows: 1

   * - Parameter
     - Hidden
     - Description
   * - Model Provider
     - No
     - Provider for the parsing model.
   * - Model ID
     - No
     - Model to use for LLM parsing.
   * - Region Name
     - No
     - Region where the model is available.
   * - Model Kwargs
     - Yes
     - Extra keyword arguments for the model. Open **Controls** to edit.
   * - Language Model (External)
     - Yes
     - Optional external Language Model component instead of built-in selection.

**Supported file types by strategy:**

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Strategy
     - Extensions
   * - LLM
     - ``.pdf``, ``.docx``, ``.pptx``, ``.xlsx``, ``.md``, ``.jpg``, ``.jpeg``, ``.png``, ``.tif``, ``.tiff``
   * - Textract
     - Same as LLM
   * - Unstructured
     - ``.pdf``, ``.docx``, ``.pptx``, ``.xlsx``, ``.md``

**Outputs**

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Output
     - Description
   * - Parsed Content
     - Extracted text and metadata as a Data object.

**Limitations**

- **File attachments** — The input message must include files; empty input returns an empty result.
- **Strategy-specific formats** — Unsupported extensions fail for the selected strategy.
- **LLM strategy** — Requires a configured or connected language model.

----

.. _component-ref-ingest-approved-response:

Ingest Approved Response
------------------------

Captures the approved response from a Human In The Loop step and injects it back into the flow by embedding and indexing the post/response pair for future retrieval.

**Use in a flow**

1. Add an **Ingest Approved Response** node after a Human In The Loop approval step.
2. Wire **Post** (original user message) and **Post Response** (approved bot reply).
3. Configure embedding model and OpenSearch connection details.
4. Connect **Result** output downstream.

.. code-block:: text

   Human In The Loop ──► Ingest Approved Response ──► Output / downstream processing

**Parameters**

.. list-table::
   :widths: 30 10 60
   :header-rows: 1

   * - Parameter
     - Default
     - Description
   * - Post
     - —
     - The original user message.
   * - Post Response
     - —
     - The human-approved bot response.
   * - Model Provider
     - —
     - Embedding model provider.
   * - Model ID
     - —
     - Embedding model used to vectorize the post/response pair.
   * - Region Name
     - —
     - Region where the embedding model is available.
   * - Model Kwargs
     - —
     - Additional model keyword arguments. Open **Controls** to edit.
   * - Channel Type
     - FACEBOOK
     - Source channel type.
   * - Customer Name
     - —
     - Customer identifier; used to derive the OpenSearch index name.
   * - OpenSearch Host
     - —
     - OpenSearch domain hostname (no ``https://`` prefix).
   * - OpenSearch User
     - —
     - OpenSearch username.
   * - OpenSearch Password
     - —
     - OpenSearch password.

**Outputs**

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Output
     - Description
   * - Result
     - Data object with ingestion status. On success, includes ``ingested`` and the index name. On failure, includes an error message.

**Limitations**

- **OpenSearch required** — Valid host, credentials, and network access are needed.
- **Embedding model** — A workspace-configured embedding model must be selected.
- **Index naming** — Index name is derived from **Customer Name** with a ``_confidence_score`` suffix.

----

.. _component-ref-lambda-filter:

Lambda Filter *(Beta)*
----------------------

Filters a list of items using a custom expression. A connected language model generates a Python lambda function from your natural-language instructions and applies it to the input data.

**Use in a flow**

1. Add a **Lambda Filter (Beta)** node to the canvas.
2. Connect **Data** from an upstream component.
3. Connect a **Language Model** from an LLM component.
4. Write **Instructions** describing how to filter or transform the data.
5. Connect **Filtered Data** or **DataFrame** output downstream.

.. code-block:: text

   Data source ──► Lambda Filter ──► downstream processing
   LLM ──► Language Model

Example instruction: *Filter the data to only include items where the status is active.*

**Parameters**

.. list-table::
   :widths: 30 10 10 50
   :header-rows: 1

   * - Parameter
     - Default
     - Hidden
     - Description
   * - Data
     - —
     - No
     - Structured data to filter or transform.
   * - Language Model
     - —
     - No
     - Connect the Language Model output from an LLM component.
   * - Instructions
     - Filter the data to...
     - No
     - Natural-language description of the filter or transformation.
   * - Sample Size
     - 1000
     - Yes
     - For large datasets, number of characters to sample from the head and tail when building the lambda.
   * - Max Size
     - 30000
     - Yes
     - Character threshold above which the dataset is treated as large and sampled.

**Outputs**

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Output
     - Description
   * - Filtered Data
     - Transformed data as a list of Data objects.
   * - DataFrame
     - Filtered result as a table.

**Limitations**

- **Beta** — Behavior may change; lambda generation depends on the connected LLM.
- **LLM required** — A Language Model must be connected; instructions are not executed directly.
- **Large data** — Very large payloads are sampled before the LLM generates the lambda.

----

.. _component-ref-llm-router:

LLM Router
----------

Routes the flow to one of several downstream paths based on the output of an LLM classification. A judge model selects the best language model for the input, then runs that model and returns its response.

**Use in a flow**

1. Add an **LLM Router** node to the canvas.
2. Connect multiple LLM components to **Language Models**.
3. Connect a judge LLM to **Judge LLM**.
4. Connect the user message to **Input**.
5. Connect **Output** to downstream steps.

.. code-block:: text

                   ┌── LLM A ──┐
   Input ──► LLM Router ── Judge LLM ──► Output
                   └── LLM B ──┘

**Parameters**

.. list-table::
   :widths: 30 10 60
   :header-rows: 1

   * - Parameter
     - Default
     - Description
   * - Language Models
     - —
     - List of LLM components to route between. Connect multiple Language Model outputs. Required.
   * - Input
     - —
     - The input message to route and process. Required.
   * - Judge LLM
     - —
     - LLM that evaluates and selects the most appropriate model. Required.
   * - Optimization
     - balanced
     - Selection preference: ``quality``, ``speed``, ``cost``, or ``balanced``.

**Outputs**

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Output
     - Description
   * - Output
     - The response message from the selected model.
   * - Selected Model
     - Name of the model chosen by the judge. Requires **Output** to run first.

**Limitations**

- **All inputs required** — Language Models, Input, and Judge LLM must all be connected.
- **Fallback** — If the judge returns an invalid index, the first model in the list is used.

----

.. _component-ref-message-to-data:

Message to Data *(Beta)*
------------------------

Converts a conversation message object into a structured data record.

**Use in a flow**

1. Add a **Message to Data (Beta)** node to the canvas.
2. Connect a **Message** from an upstream component (for example, **Input** or an Agent).
3. Connect **Data** output downstream.

.. code-block:: text

   Input (Message) ──► Message to Data ──► Data Operations / Agent

**Parameters**

.. list-table::
   :widths: 30 70
   :header-rows: 1

   * - Parameter
     - Description
   * - Message
     - The Message object to convert. Required.

**Outputs**

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Output
     - Description
   * - Data
     - Structured data extracted from the message. Returns an error field if the input is not a valid Message.

**Limitations**

- **Beta** — Behavior may change.
- **Message input only** — Input must be a Message object; other types produce an error in the output.

----

.. _component-ref-parser:

Parser
------

Formats structured data into text using a template, or converts input into a readable string. Use it to turn Data or DataFrame content into prompt-ready text for downstream components.

**Use in a flow**

1. Add a **Parser** node to the canvas.
2. Connect **Data or DataFrame** from an upstream component.
3. Choose **Mode** — **Parser** (template) or **Stringify** (plain text conversion).
4. Connect **Parsed Text** downstream.

.. code-block:: text

   SQL Query (DataFrame) ──► Parser ──► Prompt / Agent

**Template example:**

.. code-block:: text

   Name: {Name}, Age: {Age}, Country: {Country}

**Parameters**

.. list-table::
   :widths: 30 10 10 50
   :header-rows: 1

   * - Parameter
     - Default
     - Hidden
     - Description
   * - Mode
     - Parser
     - No
     - **Parser** uses a template; **Stringify** converts input to plain text.
   * - Template
     - Text: {text}
     - No
     - Format string with ``{variable}`` placeholders. Required in **Parser** mode.
   * - Data or DataFrame
     - —
     - No
     - Input to format. Accepts Data or DataFrame. Required.
   * - Separator
     - \\n
     - Yes
     - String used to join multiple rows or items.
   * - Clean Data
     - true
     - Yes
     - In **Stringify** mode, remove empty rows and extra blank lines.

**Outputs**

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Output
     - Description
   * - Parsed Text
     - Formatted text as a Message.

**Limitations**

- **Template variables** — Placeholders must match column or key names in the input.
- **Stringify mode** — DataFrames are converted to markdown-style text, not parsed into new fields.

----

.. _component-ref-regex-extractor:

Regex Extractor
---------------

Extracts values from text using a regular expression pattern.

**Use in a flow**

1. Add a **Regex Extractor** node to the canvas.
2. Enter **Input Text** or connect text from an upstream component.
3. Enter a **Regex Pattern**.
4. Connect **Data** or **Message** output downstream.

.. code-block:: text

   Input (Message) ──► Regex Extractor ──► Data Operations / Agent

**Parameters**

.. list-table::
   :widths: 30 70
   :header-rows: 1

   * - Parameter
     - Description
   * - Input Text
     - The text to search. Required.
   * - Regex Pattern
     - Regular expression pattern. Uses Python regex syntax. Required.

**Outputs**

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Output
     - Description
   * - Data
     - List of Data objects, one per match (``match`` field). Empty list if no matches.
   * - Message
     - All matches joined by newlines, or an error/no-match message.

**Limitations**

- **Valid regex** — Invalid patterns return an error in the output.
- **findall behavior** — Uses ``findall``; capture groups affect what is returned per match.
- **No matches** — Returns an empty Data list and a "No matches found" message.

----

.. _component-ref-split-text:

Split Text
----------

Splits text or a DataFrame into chunks using configurable size and overlap settings.

**Use in a flow**

1. Add a **Split Text** node to the canvas.
2. Connect **Data or DataFrame** containing text.
3. Set **Chunk Size**, **Chunk Overlap**, and **Separator**.
4. Connect **Chunks** or **DataFrame** output downstream.

.. code-block:: text

   File (Data) ──► Split Text ──► Embeddings / Vector Store

**Parameters**

.. list-table::
   :widths: 30 10 10 50
   :header-rows: 1

   * - Parameter
     - Default
     - Hidden
     - Description
   * - Data or DataFrame
     - —
     - No
     - Input containing text to split. Required.
   * - Chunk Overlap
     - 200
     - No
     - Number of characters shared between consecutive chunks.
   * - Chunk Size
     - 1000
     - No
     - Maximum characters per chunk after merging splits.
   * - Separator
     - \\n
     - No
     - Character(s) to split on first. Use ``\n`` for newlines, ``\n\n`` for paragraphs.
   * - Text Key
     - text
     - Yes
     - Field name used for text when input is a DataFrame.
   * - Keep Separator
     - False
     - Yes
     - Whether to keep the separator in chunks: ``False``, ``True``, ``Start``, or ``End``.

**Outputs**

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Output
     - Description
   * - Chunks
     - List of Data objects, one per chunk, with text and metadata.
   * - DataFrame
     - Chunks as a table.

**Limitations**

- **Empty input** — Empty DataFrame or missing data raises an error.
- **Separator** — Splits larger than **Chunk Size** are not subdivided further.
- **Text key** — For DataFrames, the column named by **Text Key** is used as the source text.

----

.. _component-ref-streaming-data-to-message:

Streaming Data to Message
-------------------------

Converts a streaming data response into a standard message format.

**Use in a flow**

1. Add a **Streaming Data to Message** node to the canvas.
2. Connect streaming or static **Text** input (Data, DataFrame, Message, string, list, or generator).
3. Connect **Message** output downstream (for example, **Output**).

.. code-block:: text

   LLM (streaming) ──► Streaming Data to Message ──► Output
   Agent ──► Streaming Data to Message

**Parameters**

.. list-table::
   :widths: 30 10 10 50
   :header-rows: 1

   * - Parameter
     - Default
     - Hidden
     - Description
   * - Text
     - —
     - No
     - Input to convert. Accepts Data, DataFrame, Message, string, list, or streaming generator. Required.
   * - Session ID
     - —
     - Yes
     - Chat session identifier. Uses the current session if empty.

**Outputs**

.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Output
     - Description
   * - Message
     - The collected or converted text as a standard message.

**Limitations**

- **Supported input types** — Data, DataFrame, Message, string, list, and generator inputs are supported.
- **Streaming** — Generator inputs are fully consumed before the message is returned.
- **Empty Data** — Data objects with no text content raise an error.