# Pipelines and runs

When you send content to Engram, it runs through an asynchronous pipeline that extracts, transforms, and commits memories. Pipelines are defined as a directed acyclic graph (DAG) of steps — different [input content types](input-data-types.md) enter the DAG at different extraction steps, and downstream steps like transform and commit are often shared between them.

:::callout{intent="info" title="Pipeline configuration"}
Configurable pipelines are available on enterprise plans.
:::

## Pipeline steps

Each pipeline is a DAG with multiple entrypoints — one per content type — that converge into shared downstream steps:

1. **Extract** — The entrypoint into the pipeline. Each [content type](input-data-types.md) has its own extraction step (`ExtractFromString`, `ExtractFromConversation`, or `ExtractFromPreExtracted`). These extraction steps often feed into shared downstream transform and commit steps, though that is not required — a pipeline can route different content types through different downstream steps.
2. **Transform** — Refines extracted memories using existing context. Steps like `TransformWithContext`, `TransformOperations`, `TransformConcatenate`, and `TransformAggregate` deduplicate, merge, consolidate, and resolve conflicts with existing memories. `TransformAggregate` and `TransformWithContext` also honor [bounded topics](topics.md), consolidating multiple extracted facts into the single memory that exists for the topic's scope.
3. **Buffer** — Pauses the pipeline and accumulates memories (or raw inputs) until a trigger fires — by count, time since the first item, or time since the last item. Buffers can appear anywhere in the pipeline, not just at the start. Memories from different content types that route into the same buffer are aggregated together.
4. **Commit** — Finalizes the operations (create, update, delete) and persists them to storage.

A pipeline can chain these steps in different ways. For example, a pipeline for aggregated daily summaries might look like:

`[extract] → [transform] → [commit] → [buffer] → [transform] → [commit]`

In this case, memories are immediately extracted, transformed, and committed. They then enter a buffer that accumulates all memories added with the same [scope](scopes.md) (user or conversation). When the buffer triggers (e.g. after 24 hours), the accumulated batch continues through a second transform step — which could combine everything into a single "daily activity" memory — followed by a second commit.

![Weaviate Engram Pipeline](/assets/docs/engram/_includes/pipeline.png)

## Runs

Each call to [store memories](../engram-guides/store-memories.md) creates a **run** — a trackable unit of pipeline execution. Runs have four possible states:

| Status      | Meaning                                                           |
| ----------- | ----------------------------------------------------------------- |
| `running`   | Pipeline is actively processing                                   |
| `in_buffer` | Run is paused at a buffer step, waiting for a trigger to continue |
| `completed` | All operations committed successfully                             |
| `failed`    | An error occurred during processing                               |

When a run completes, its [`committed_operations`](../engram-guides/check-run-status.md) field shows exactly which memories were created, updated, or deleted.

## Questions and feedback

Have a question or feedback? Here's how to reach us.

::::card-grid
:::card{title="Community Forum" href="https://forum.weaviate.io/c/support" icon="messages-square"}
Ask questions and connect with other developers on our **Community forum**.
:::

:::card{title="Support" href="/guides/support-overview" icon="life-buoy"}
Weaviate Cloud user or customer? Find the right channel on the **Support page**.
:::
::::

## Related pages

- [Agents](./agents-index.md)
- [AI-assisted Weaviate code generation](./ai-assisted-vibe-coding-index.md)
- [APIs](./apis-index.md)
- [Authorization and authentication](./authorization-and-authentication-index.md)
- [Benchmarks](./benchmarks-index.md)
- [Best practices](./best-practices-index.md)
- [Client libraries](./clients-index.md)
- [Client Libraries / SDKs](./client-libraries-index.md)
- [Cloud](./cloud-index.md)
- [Cloud account management](./cloud-account-management-index.md)

# Agent Instructions

This portal answers questions programmatically. To receive a synthesized,
source-cited answer instead of crawling page by page, append the `?ask=`
query parameter to any page URL on this site:

    /guides/quickstart?ask=how+do+I+authenticate

Optional parameters:

- `&goal=<what-you-are-trying-to-do>` steers the answer toward your
  objective (e.g. `&goal=write+a+python+client`).
- `&version=<label>` scopes the answer to a mounted version when the
  portal publishes more than one.

The response is `text/markdown`: the answer followed by a `# Sources` list
of the portal pages it was grounded in. Status codes are the contract:

- `200` — the answer; `402` — the portal owner’s plan or answer credits are
  exhausted (surface this to your operator; do NOT retry); `429` — you are
  rate-limited; back off for the `Retry-After` seconds; `503` — the answer
  lane is temporarily unavailable; fall back to crawling the `.md` pages.

For the full corpus map read `llms.txt` at the site root; for the tool
surface (search + page fetch as MCP tools) see `/mcp`.