Retrieve

slateRead lets a step read declared files at config time. The author writes from: { slate: ..., folder: ..., file: ... } and that file's contents arrive in the prompt. This works when the author knows which file matters.

But the whole point of a slate is that files accumulate over time. A research slate might have 100 documents after a week. A lessons slate might have 50 entries after a hundred calls. The author cannot know at config time which of those files will matter for this particular call.

Two bad options without retrieve:

• Read every file every call. Token-expensive. Most files are irrelevant to most calls. The slate becomes useless above a few dozen files.
• Read no files. The slate becomes write-only. The ladder learns nothing from accumulated state.

Retrieve is the third option. The step sees the folder index (small: just file names and sizes) and the tree (recursive structure). Based on those, the LLM reasons about what is relevant and requests specific files on demand. The runtime fetches them, injects them as a field, and the LLM continues with the enriched context.

This is the "browse → reason → retrieve → read → continue" pattern, modeled on how a human reads a filesystem. The LLM's retrieval reasoning is visible in the trace (useful for debugging). No embedding pipeline is needed (slates are bounded). Termination is bounded by maxRounds. Composes with structured metatags for deterministic queries.

Better than RAG for this case because:

• Interpretable. The LLM's retrieval reasoning is in the trace, not opaque similarity scores.
• No embedding pipeline. Slates are bounded; structured indexes are enough.
• Composable with metatags. Deterministic queries ({"metatag": {"name": "tags", "contains": "calculus"}}) coexist with LLM-driven choices.
• Termination-guaranteed. maxRounds caps the fetch loop. A ladder author cannot cause infinite retrieval.

A retrieve block is declared as an optional field on a normal or sequential step.

- id: research
  type: normal
  fields:
    - { name: Question,   type: text,      from: input.context }
    - { name: FileIndex,  type: slateRead, from: { slate: Knowledge, folder: docs, metatag: index } }
    - { name: FolderTree, type: slateRead, from: { slate: Knowledge, folder: docs, metatag: tree, depth: 2 } }

  retrieve:
    to: { slate: Knowledge, folder: docs }
    as: RetrievedDocuments              # field name injected on subsequent rounds
    maxRounds: 5                        # hard cap → guarantees termination
    requestMatch:                       # JSON Schema; if output matches → retrieval request
      type: object
      required: [request]
      properties:
        request:
          type: object
          properties:
            files:      { type: array, items: { type: string } }
            subfolders: { type: array, items: { type: string } }
            metatag:
              type: object
              properties:
                name: { type: string }
                contains: { type: string }
            glob: { type: string }
          additionalProperties: false
      additionalProperties: false

  systemPrompt: |
    You can see FileIndex and FolderTree. To retrieve, emit exactly:
    {"request": {"files": ["path.md"]}}
    {"request": {"subfolders": ["notes"]}}
    {"request": {"metatag": {"name": "tags", "contains": "calculus"}}}
    {"request": {"glob": "notes/*.md"}}
    When you have enough, output your final answer in prose.

Properties:

The requestMatch schema declares what counts as a retrieval request. The four canonical modes (files, subfolders, metatag, glob) are conventional; the schema can be tightened or loosened per ladder. Multiple modes can combine in one request.

Within a single step's execution:

1. First round. The runtime assembles the step's fields (including the declared FileIndex and FolderTree via slateRead) and calls the LLM.

2. Inspect output. The runtime checks whether the output matches requestMatch:

   • Match → retrieval request. The runtime parses the request, fetches the requested content from the target slate, and appends a new field named after as (e.g. RetrievedDocuments) to the field list.
   • No match → final answer. The step completes; the output is the step's result.

3. Subsequent rounds. If retrieval happened, the runtime re-assembles the prompt with the new RetrievedDocuments field included, and calls the LLM again. Round counter increments.

4. Repeat until the LLM emits a non-matching output (final answer) OR maxRounds is hit.

5. maxRounds hit. The runtime injects: "Retrieval budget exhausted. Produce your final answer now." The next call must produce the final answer.

Termination is guaranteed by maxRounds. A ladder author cannot cause infinite retrieval loops. The cost is bounded: at most maxRounds + 1 LLM calls per step (the +1 is the forced-final call if budget is exhausted).

Each round's prompt grows. The first round's prompt is the original fields. The second round's prompt includes RetrievedDocuments 1: (whatever was fetched in round 1). The third round includes RetrievedDocuments 1: and RetrievedDocuments 2:. And so on. The accumulation is monotonic; nothing previously retrieved disappears.

Four modes can be requested, combinable in one request.

Multiple modes combine in one request:

{
  "request": {
    "files": ["overview.md", "intro.md"],
    "subfolders": ["examples"],
    "metatag": {"name": "tags", "contains": "important"},
    "glob": "drafts/*.md"
  }
}

The runtime fetches from all four modes and merges the results into one RetrievedDocuments block.

A planned fifth mode: semantic similarity via the auto-maintained embedding metatag:

{"request": {"metatag": {"name": "embedding", "similarTo": "query text"}}, "topK": 5}

Same requestMatch schema, new mode. No grammar change — slots in behind the existing interface.

The runtime tracks what has already been retrieved across rounds in this step. If the LLM requests a file it already has, the runtime returns "(already in context)" instead of re-fetching.

This serves two purposes:

• Token economy. Re-requesting the same file bloats the prompt without adding information. Dedup keeps the prompt focused on what's new.
• Loop prevention. Without dedup, an LLM could get stuck requesting the same files every round. Dedup forces it to either request something new or produce a final answer.

Dedup is per-step, not per-call. Each step with a retrieve: block starts with a fresh dedup tracker.

Dedup compares on file path. Two requests for notes/calculus.md in the same step yield one fetch and one "(already in context)". A request for notes/calculus.md and a request for the glob notes/*.md that happens to include notes/calculus.md will fetch the file once via the first request and note it as already-in-context in the glob's response.

Retrieve composes with every other LADR primitive.

With if/then/else: retrieval happens first (rounds complete), then the gate fires on the final output. A step can research → reason → gate in one place.

- id: research
  retrieve: { ... }
  systemPrompt: "..."
  if:
    jsonMatches: { ... }
    then: continue
    else: { jump: { stepId: fallback } }

The retrieval loop runs to completion (or maxRounds), then the gate evaluates against the final output. If the gate fails, control jumps; the retrieved documents are not carried over to the jump target.

With slateWrite: retrieval and write coexist. A step can read → reason → write structured findings, all in one.

- id: research_and_extract
  retrieve: { ... }
  systemPrompt: "..."
  slateWrite:
    to: { slate: "Memory", folder: facts, file: core.md }
    on: append
    match: { ... }

The retrieval loop runs; the step's final output is checked against slateWrite.match; conforming JSON gets persisted. Read-reason-write in one step.

With jumps: retrieval completes before any jump fires. The retrieved documents stay in the step's prompt context; they don't follow the cursor to the jump target. To carry retrieved context across steps, persist it via slateWrite and re-read via slateRead downstream.

With multi-node fanout (nodes: N + retrieve:): each node runs its own retrieval loop independently. Useful for parallel research (multiple investigators each pulling different documents); expensive (cost scales with nodes). Knob-control the node count for cost management.

With recursion: a recursive child can have its own retrieve: block against the same shared slate. The child's retrieval is independent of the parent's; the slate state is shared, so anything the parent wrote is visible to the child's retrieval.

Retrieve-specific rules checked at config load time.

• maxRounds positive. Zero or negative fails: Step "X": retrieve.maxRounds must be > 0 (got 0).
• as non-empty. Empty field name fails.
• requestMatch non-empty valid JSON. Malformed JSON fails; empty schema fails.
• to.slate and to.folder must be declared. Same rule as slateRead and slateWrite.

These rules ensure the retrieval loop is well-formed before any execution. The runtime can rely on the schema being valid; it does not re-check at execution time.

Research step. Investigate a question against a slate of documents.

- id: research
  fields:
    - { name: Question,   type: text,      from: input.context }
    - { name: FileIndex,  type: slateRead, from: { slate: Knowledge, folder: docs, metatag: index } }
  retrieve:
    to: { slate: Knowledge, folder: docs }
    as: RetrievedDocuments
    maxRounds: 5
    requestMatch:
      type: object
      required: [request]
      properties:
        request:
          type: object
          properties:
            files: { type: array, items: { type: string } }
      additionalProperties: false
  systemPrompt: |
    Research the Question. Use FileIndex to find relevant docs.
    Emit {"request": {"files": [...]}} to retrieve.
    When ready, output your synthesis.

Targeted retrieval via metatag. Find files matching a category.

retrieve:
  to: { slate: Knowledge, folder: docs }
  as: RetrievedDocuments
  maxRounds: 3
  requestMatch:
    type: object
    properties:
      request:
        type: object
        properties:
          metatag:
            type: object
            properties:
              name: { type: string, enum: [tags] }
              contains: { type: string }

The LLM can only request via metatag query — files where the tags metatag contains some string. Useful for category-driven retrieval ("calculus", "integration", "u-substitution").

Single-shot retrieval (maxRounds: 1). Force the LLM to ask for everything it needs in one round.

retrieve:
  maxRounds: 1
  # ...

The LLM gets one chance to request files. After the response, it must produce a final answer. Cheaper than multi-round; less adaptive. Use when the folder structure is simple enough that one round is sufficient.

Parallel research (multi-node). Multiple investigators each research independently.

- id: parallel_research
  nodes: "{{knobs.investigators}}"
  fields:
    - { name: Question, type: text, from: input.context }
    - { name: NodeNumber, type: nodeInfo }
    - { name: FileIndex, type: slateRead, from: { slate: Knowledge, folder: docs, metatag: index } }
  retrieve: { ... }
    systemPrompt: "You are the investigator whose number is in the Node Number field above. Find evidence for the Question."

Each node runs its own retrieval loop. Downstream multi_ingest: { nodeRef: accumulate } reads all nodes' findings. Useful for coverage; expensive — cost scales with nodes × maxRounds.

Forgetting to declare FileIndex or FolderTree fields. Without them, the LLM has no way to know what files exist. The retrieval loop becomes a blind guess. Always include at least metatag: index as a field on a retrieve step.

Setting maxRounds too high. Each round is an LLM call. maxRounds: 20 means up to 21 LLM calls per step (20 retrieval + 1 forced final). For most use cases, 3-5 rounds is plenty.

Setting maxRounds too low. maxRounds: 1 forces single-shot retrieval. If the LLM cannot figure out what it needs from just the file index, it will produce a poorly-grounded answer. Use higher limits when the folder structure is complex or the LLM needs to browse.

Tight requestMatch schema that the LLM can't satisfy. If your requestMatch requires exact field names the LLM doesn't emit, every output fails to match and the LLM never retrieves anything. Look at the trace to see what the LLM is emitting; loosen the schema or improve the prompt.

Loose requestMatch that matches final answers. If your requestMatch is too permissive, the LLM's final answer might match and trigger another retrieval round instead of ending the step. Use required: [request] and a structured request object to keep the schemas distinct.

Expecting retrieve to read across slates. retrieve.to specifies one slate + folder. To read from multiple folders, use multiple steps chained together, each with its own retrieve: block.

Combining with multi-node without thinking about cost. nodes: 5 with maxRounds: 5 is up to 30 LLM calls per step. Knob-control the node count and bound the budget via executionBudget.maxLlmCalls to avoid runaway cost.