# Processing

Processing components transform, filter, parse, and route data as it moves through the flow.

---

## Combine Data

Merges outputs from multiple upstream components into a single data object.

**Use in a flow**

1. Add a **Combine Data** node to the canvas.
2. Connect at least two **Data Inputs** from upstream components.
3. Choose an **Operation Type**.
4. Connect **DataFrame** output downstream.

```
Component A ──► Data Inputs ──┐
Component B ──► Data Inputs ──┼──► Combine Data (DataFrame) ──► ...
Component C ──► Data Inputs ──┘
```

**Parameters**

| Parameter | Default | Description |
| --- | --- | --- |
| Data Inputs | — | Two or more Data objects to combine. Required. |
| Operation Type | Concatenate | How to merge the inputs: **Concatenate**, **Append**, **Merge**, or **Join**. |

**Operation types**

| Operation | Behavior |
| --- | --- |
| Concatenate | Merge keys from all inputs into one row. Duplicate string keys are joined with newlines. |
| Append | Each input becomes its own row in the output table. |
| Merge | Merge keys; duplicate string values become lists. |
| Join | Merge keys; duplicate keys from later inputs are renamed with a suffix (`_doc2`, `_doc3`, etc.). |

**Outputs**

| Output | Description |
| --- | --- |
| DataFrame | Combined result as a table. Returns empty if fewer than two inputs are connected. |

**Limitations**

- **Minimum inputs** — Requires at least two Data inputs to produce a result.
- **Single output type** — Output is always a DataFrame, not a raw Data object.

---

## Data → DataFrame

Converts raw data into a tabular DataFrame structure for further processing.

**Use in a flow**

1. Add a **Data → DataFrame** node to the canvas.
2. Connect one or more **Data** objects to **Data or Data List**.
3. Connect **DataFrame** output to downstream steps.

```
API Request (Data) ──► Data → DataFrame ──► DataFrame Operations
```

**Parameters**

| Parameter | Description |
| --- | --- |
| Data or Data List | One or more Data objects to convert. Accepts a single Data object or a list. |

**Outputs**

| Output | Description |
| --- | --- |
| DataFrame | A table with one row per Data object. |

**Limitations**

- **Data input only** — Input must be Data objects; other types raise an error.
- **Column names** — Column names come from keys in each Data object’s data fields.

---

## Data Operations

Applies transformations to structured data — select keys, combine, filter values, append or update fields, remove keys, rename keys, and evaluate literal values.

**Use in a flow**

1. Add a **Data Operations** node to the canvas.
2. Connect a **Data** input.
3. Select one **Operation** from the list.
4. Fill in the fields that appear for that operation.
5. Connect **Data** output downstream.

```
Webhook (Data) ──► Data Operations ──► Agent / Output
```

**Parameters**

| Parameter | Description |
| --- | --- |
| Data | The Data object (or list of Data objects for **Combine**) to transform. Required. |
| Operations | The transformation to apply. Choose one: **Select Keys**, **Literal Eval**, **Combine**, **Filter Values**, **Append or Update**, **Remove Keys**, or **Rename Keys**. |

Fields below appear based on the selected operation (open **Controls** to access them):

| Operation | Additional fields |
| --- | --- |
| Select Keys | **Select Keys** — list of keys to keep. |
| Literal Eval | Evaluates string values that look like Python literals (lists, dicts, numbers, booleans). |
| Combine | Requires multiple Data inputs. Merges keys from all inputs into one Data object. |
| Filter Values | **Filter Key**, **Comparison Operator** (equals, not equals, contains, starts with, ends with), **Filter Values**. |
| Append or Update | **Append or Update** — key-value pairs to add or overwrite. |
| Remove Keys | **Remove Keys** — list of keys to delete. |
| Rename Keys | **Rename Keys** — map of old key → new key. |

**Outputs**

| Output | Description |
| --- | --- |
| Data | The transformed Data object. |

**Limitations**

- **One operation at a time** — Only a single operation runs per execution.
- **Combine** — Requires multiple Data inputs wired to **Data**.
- **Filter Values** — The target key must contain a list of dictionaries.

---

## DataFrame Operations

Applies operations on a DataFrame — sort, filter, select columns, add or drop columns, rename columns, replace values, and take head or tail rows.

**Use in a flow**

1. Add a **DataFrame Operations** node to the canvas.
2. Connect a **DataFrame** input.
3. Select an **Operation**.
4. Fill in the fields that appear for that operation.
5. Connect **DataFrame** output downstream.

```
Data → DataFrame ──► DataFrame Operations ──► Agent / Output
```

**Parameters**

| Parameter | Description |
| --- | --- |
| DataFrame | The input table to operate on. |
| Operation | The operation to perform. |

**Available operations** (open **Controls** to access operation-specific fields):

| Operation | Fields shown |
| --- | --- |
| Filter | **Column Name**, **Filter Value** — keep rows where the column equals the value. |
| Sort | **Column Name**, **Sort Ascending**. |
| Drop Column | **Column Name**. |
| Rename Column | **Column Name**, **New Column Name**. |
| Add Column | **New Column Name**, **New Column Value** — same value for every row. |
| Select Columns | **Columns to Select** — list of column names to keep. |
| Head | **Number of Rows** — first N rows (default 5). |
| Tail | **Number of Rows** — last N rows (default 5). |
| Replace Value | **Column Name**, **Value to Replace**, **Replacement Value**. |

**Outputs**

| Output | Description |
| --- | --- |
| DataFrame | The resulting table after the operation. |

**Limitations**

- **One operation at a time** — Each run applies a single selected operation.
- **Exact match filter** — **Filter** uses equality comparison only.

---

## Document Parser

Extracts text and structure from documents (PDF, DOCX, PPTX, XLSX, Markdown, and images) using configurable parsing strategies.

**Use in a flow**

1. Add a **Document Parser** node to the canvas.
2. Connect an **Input Message** that includes file attachments (for example, from **Input**).
3. Choose a **Parsing Strategy**.
4. Connect **Parsed Content** downstream.

```
Input (Message with files) ──► Document Parser ──► Agent / Split Text
```

**Parameters**

| Parameter | Default | Description |
| --- | --- | --- |
| Input Message | — | Message containing file paths to parse. Required. |
| Parsing Strategy | LLM | **LLM** (AI model), **Unstructured** (unstructured.io), or **Textract** (AWS Textract). |

When **LLM** is selected, model fields appear:

| Parameter | Hidden | Description |
| --- | --- | --- |
| Model Provider | No | Provider for the parsing model. |
| Model ID | No | Model to use for LLM parsing. |
| Region Name | No | Region where the model is available. |
| Model Kwargs | Yes | Extra keyword arguments for the model. Open **Controls** to edit. |
| Language Model (External) | Yes | Optional external Language Model component instead of built-in selection. |

**Supported file types by strategy:**

| Strategy | Extensions |
| --- | --- |
| LLM | `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.md`, `.jpg`, `.jpeg`, `.png`, `.tif`, `.tiff` |
| Textract | Same as LLM |
| Unstructured | `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.md` |

**Outputs**

| Output | Description |
| --- | --- |
| Parsed Content | Extracted text and metadata as a Data object. |

**Limitations**

- **File attachments** — The input message must include files; empty input returns an empty result.
- **Strategy-specific formats** — Unsupported extensions fail for the selected strategy.
- **LLM strategy** — Requires a configured or connected language model.

---

## Ingest Approved Response

Captures the approved response from a Human In The Loop step and injects it back into the flow by embedding and indexing the post/response pair for future retrieval.

**Use in a flow**

1. Add an **Ingest Approved Response** node after a Human In The Loop approval step.
2. Wire **Post** (original user message) and **Post Response** (approved bot reply).
3. Configure embedding model and OpenSearch connection details.
4. Connect **Result** output downstream.

```
Human In The Loop ──► Ingest Approved Response ──► Output / downstream processing
```

**Parameters**

| Parameter | Default | Description |
| --- | --- | --- |
| Post | — | The original user message. |
| Post Response | — | The human-approved bot response. |
| Model Provider | — | Embedding model provider. |
| Model ID | — | Embedding model used to vectorize the post/response pair. |
| Region Name | — | Region where the embedding model is available. |
| Model Kwargs | — | Additional model keyword arguments. Open **Controls** to edit. |
| Channel Type | FACEBOOK | Source channel type. |
| Customer Name | — | Customer identifier; used to derive the OpenSearch index name. |
| OpenSearch Host | — | OpenSearch domain hostname (no `https://` prefix). |
| OpenSearch User | — | OpenSearch username. |
| OpenSearch Password | — | OpenSearch password. |

**Outputs**

| Output | Description |
| --- | --- |
| Result | Data object with ingestion status. On success, includes `ingested` and the index name. On failure, includes an error message. |

**Limitations**

- **OpenSearch required** — Valid host, credentials, and network access are needed.
- **Embedding model** — A workspace-configured embedding model must be selected.
- **Index naming** — Index name is derived from **Customer Name** with a `_confidence_score` suffix.

---

## Lambda Filter *(Beta)*

Filters a list of items using a custom expression. A connected language model generates a Python lambda function from your natural-language instructions and applies it to the input data.

**Use in a flow**

1. Add a **Lambda Filter (Beta)** node to the canvas.
2. Connect **Data** from an upstream component.
3. Connect a **Language Model** from an LLM component.
4. Write **Instructions** describing how to filter or transform the data.
5. Connect **Filtered Data** or **DataFrame** output downstream.

```
Data source ──► Lambda Filter ──► downstream processing
LLM ──► Language Model
```

Example instruction: *Filter the data to only include items where the status is active.*

**Parameters**

| Parameter | Default | Hidden | Description |
| --- | --- | --- | --- |
| Data | — | No | Structured data to filter or transform. |
| Language Model | — | No | Connect the Language Model output from an LLM component. |
| Instructions | Filter the data to… | No | Natural-language description of the filter or transformation. |
| Sample Size | 1000 | Yes | For large datasets, number of characters to sample from the head and tail when building the lambda. |
| Max Size | 30000 | Yes | Character threshold above which the dataset is treated as large and sampled. |

**Outputs**

| Output | Description |
| --- | --- |
| Filtered Data | Transformed data as a list of Data objects. |
| DataFrame | Filtered result as a table. |

**Limitations**

- **Beta** — Behavior may change; lambda generation depends on the connected LLM.
- **LLM required** — A Language Model must be connected; instructions are not executed directly.
- **Large data** — Very large payloads are sampled before the LLM generates the lambda.

---

## LLM Router

Routes the flow to one of several downstream paths based on the output of an LLM classification. A judge model selects the best language model for the input, then runs that model and returns its response.

**Use in a flow**

1. Add an **LLM Router** node to the canvas.
2. Connect multiple LLM components to **Language Models**.
3. Connect a judge LLM to **Judge LLM**.
4. Connect the user message to **Input**.
5. Connect **Output** to downstream steps.

```
                ┌── LLM A ──┐
Input ──► LLM Router ── Judge LLM ──► Output
                └── LLM B ──┘
```

**Parameters**

| Parameter | Default | Description |
| --- | --- | --- |
| Language Models | — | List of LLM components to route between. Connect multiple Language Model outputs. Required. |
| Input | — | The input message to route and process. Required. |
| Judge LLM | — | LLM that evaluates and selects the most appropriate model. Required. |
| Optimization | balanced | Selection preference: `quality`, `speed`, `cost`, or `balanced`. |

**Outputs**

| Output | Description |
| --- | --- |
| Output | The response message from the selected model. |
| Selected Model | Name of the model chosen by the judge. Requires **Output** to run first. |

**Limitations**

- **All inputs required** — Language Models, Input, and Judge LLM must all be connected.
- **Fallback** — If the judge returns an invalid index, the first model in the list is used.

---

## Message to Data *(Beta)*

Converts a conversation message object into a structured data record.

**Use in a flow**

1. Add a **Message to Data (Beta)** node to the canvas.
2. Connect a **Message** from an upstream component (for example, **Input** or an Agent).
3. Connect **Data** output downstream.

```
Input (Message) ──► Message to Data ──► Data Operations / Agent
```

**Parameters**

| Parameter | Description |
| --- | --- |
| Message | The Message object to convert. Required. |

**Outputs**

| Output | Description |
| --- | --- |
| Data | Structured data extracted from the message. Returns an error field if the input is not a valid Message. |

**Limitations**

- **Beta** — Behavior may change.
- **Message input only** — Input must be a Message object; other types produce an error in the output.

---

## Parser

Formats structured data into text using a template, or converts input into a readable string. Use it to turn Data or DataFrame content into prompt-ready text for downstream components.

**Use in a flow**

1. Add a **Parser** node to the canvas.
2. Connect **Data or DataFrame** from an upstream component.
3. Choose **Mode** — **Parser** (template) or **Stringify** (plain text conversion).
4. Connect **Parsed Text** downstream.

```
SQL Query (DataFrame) ──► Parser ──► Prompt / Agent
```

**Template example:**

```
Name: {Name}, Age: {Age}, Country: {Country}
```

**Parameters**

| Parameter | Default | Hidden | Description |
| --- | --- | --- | --- |
| Mode | Parser | No | **Parser** uses a template; **Stringify** converts input to plain text. |
| Template | Text: {text} | No | Format string with `{variable}` placeholders. Required in **Parser** mode. |
| Data or DataFrame | — | No | Input to format. Accepts Data or DataFrame. Required. |
| Separator | \n | Yes | String used to join multiple rows or items. |
| Clean Data | true | Yes | In **Stringify** mode, remove empty rows and extra blank lines. |

**Outputs**

| Output | Description |
| --- | --- |
| Parsed Text | Formatted text as a Message. |

**Limitations**

- **Template variables** — Placeholders must match column or key names in the input.
- **Stringify mode** — DataFrames are converted to markdown-style text, not parsed into new fields.

---

## Regex Extractor

Extracts values from text using a regular expression pattern.

**Use in a flow**

1. Add a **Regex Extractor** node to the canvas.
2. Enter **Input Text** or connect text from an upstream component.
3. Enter a **Regex Pattern**.
4. Connect **Data** or **Message** output downstream.

```
Input (Message) ──► Regex Extractor ──► Data Operations / Agent
```

**Parameters**

| Parameter | Description |
| --- | --- |
| Input Text | The text to search. Required. |
| Regex Pattern | Regular expression pattern. Uses Python regex syntax. Required. |

**Outputs**

| Output | Description |
| --- | --- |
| Data | List of Data objects, one per match (`match` field). Empty list if no matches. |
| Message | All matches joined by newlines, or an error/no-match message. |

**Limitations**

- **Valid regex** — Invalid patterns return an error in the output.
- **findall behavior** — Uses `findall`; capture groups affect what is returned per match.
- **No matches** — Returns an empty Data list and a “No matches found” message.

---

## Split Text

Splits text or a DataFrame into chunks using configurable size and overlap settings.

**Use in a flow**

1. Add a **Split Text** node to the canvas.
2. Connect **Data or DataFrame** containing text.
3. Set **Chunk Size**, **Chunk Overlap**, and **Separator**.
4. Connect **Chunks** or **DataFrame** output downstream.

```
File (Data) ──► Split Text ──► Embeddings / Vector Store
```

**Parameters**

| Parameter | Default | Hidden | Description |
| --- | --- | --- | --- |
| Data or DataFrame | — | No | Input containing text to split. Required. |
| Chunk Overlap | 200 | No | Number of characters shared between consecutive chunks. |
| Chunk Size | 1000 | No | Maximum characters per chunk after merging splits. |
| Separator | \n | No | Character(s) to split on first. Use `\n` for newlines, `\n\n` for paragraphs. |
| Text Key | text | Yes | Field name used for text when input is a DataFrame. |
| Keep Separator | False | Yes | Whether to keep the separator in chunks: `False`, `True`, `Start`, or `End`. |

**Outputs**

| Output | Description |
| --- | --- |
| Chunks | List of Data objects, one per chunk, with text and metadata. |
| DataFrame | Chunks as a table. |

**Limitations**

- **Empty input** — Empty DataFrame or missing data raises an error.
- **Separator** — Splits larger than **Chunk Size** are not subdivided further.
- **Text key** — For DataFrames, the column named by **Text Key** is used as the source text.

---

## Streaming Data to Message

Converts a streaming data response into a standard message format.

**Use in a flow**

1. Add a **Streaming Data to Message** node to the canvas.
2. Connect streaming or static **Text** input (Data, DataFrame, Message, string, list, or generator).
3. Connect **Message** output downstream (for example, **Output**).

```
LLM (streaming) ──► Streaming Data to Message ──► Output
Agent ──► Streaming Data to Message
```

**Parameters**

| Parameter | Default | Hidden | Description |
| --- | --- | --- | --- |
| Text | — | No | Input to convert. Accepts Data, DataFrame, Message, string, list, or streaming generator. Required. |
| Session ID | — | Yes | Chat session identifier. Uses the current session if empty. |

**Outputs**

| Output | Description |
| --- | --- |
| Message | The collected or converted text as a standard message. |

**Limitations**

- **Supported input types** — Data, DataFrame, Message, string, list, and generator inputs are supported.
- **Streaming** — Generator inputs are fully consumed before the message is returned.
- **Empty Data** — Data objects with no text content raise an error.