How to Debug a Ladder

Every ladder execution publishes a stream of events. In Foundry, the Trace View shows these events in real time. Via the API, events are available on the trace endpoint.

The event sequence for a normal execution:

pipeline_start          → pipeline begins
  loop_start            → loop 0 begins
    step_start          → step "draft" begins
    step_complete       → step "draft" finishes
    step_start          → step "verify" begins
    node_survivors      → 3 of 5 nodes survived gate
    step_complete       → step "verify" finishes
    step_start          → step "answer" begins
    step_complete       → step "answer" finishes
  loop_complete         → loop 0 finishes
pipeline_complete       → exit output returned

Key events for debugging:

• node_survivors — shows which nodes passed the gate. If all nodes were pruned, check the gate condition.
• node_count_resolved — shows if the requested node count was clamped by the budget.
• node_failed — a specific node's LLM call failed. Check the message for the provider error.
• error — something went wrong. Check ceiling_exceeded for budget hits, nonFatal for slate write failures.

Symptom: The ladder returns an empty string or unexpected content.

Diagnosis:

1. Check which step is the exit: step. The returned output is the exit step's output from the last completed loop.

2. Check if the exit step ran. Look for step_start and step_complete events for the exit step ID. If the exit step was skipped (by a forward jump that jumped past it), the output comes from the previous loop iteration.

3. Check field resolution. If an ingest field references a step that was pruned to zero survivors, the field resolves to empty. Look for node_survivors events showing zero survivors.

4. Check for fallback. If a field has a fallback:, it resolves to the fallback when the primary source misses. An unexpected fallback value suggests the primary source didn't resolve.

Fix: Ensure the exit step's fields reference steps that actually produce output. Use fallback: to handle missing sources gracefully.

Symptom: Every node is pruned by the gate; the else: action always fires.

Diagnosis:

1. Check the gate condition type. If using integerRange: [4, 5] but the LLM outputs "Score: 4" (prose around the number), the parse fails. The integer predicates parse the entire string, not embedded numbers.

2. Check the system prompt. Ensure it instructs the LLM to output the bare value (just the number, just the token, just the JSON).

3. Check jsonMatches schemas. If the schema requires fields the LLM isn't producing, validation fails. Use the trace to see what the LLM actually outputted.

4. Use contains instead of equals for prose-wrapped outputs. contains: "VERDICT: ACCEPT" handles surrounding text; equals: "ACCEPT" requires exact match.

Fix: Align the system prompt with the gate condition. If the prompt says "output JSON", use jsonMatches. If it says "output a number", use integerRange or integerEquals. If it says "output yes or no", use in: ["yes", "Y", "YES"].

Symptom: The ladder stops mid-execution with an error event containing ceiling_exceeded: true.

Diagnosis:

1. Check which ceiling was hit. The error message includes the ceiling name (spend, hops, llmCalls, or recursion) and the used vs. limit values.

2. Spend ceiling: The ladder is making too many expensive LLM calls. Reduce node counts, reduce loop iterations, or use a cheaper model.

3. Hop ceiling: The ladder is looping too many times (jumps or loops). Check for backward jumps that create tight loops. Increase executionBudget.maxHops if the loop count is intentional.

4. LLM call ceiling: Similar to spend but counts calls, not dollars. Reduce node counts or parallel branches.

5. Recursion ceiling: The ladder recurses too deeply. Reduce recursion.maxDepth or restructure to avoid deep nesting.

Fix: Declare an executionBudget that matches your expected usage. The runtime returns best-so-far output on ceiling exhaustion, so the caller still gets a response.

executionBudget:
  maxSpend: 15.0       # raise from $5 default
  maxHops: 500         # raise from 200 default

Remember: platform maximums are $50 spend, 10,000 hops, 5,000 LLM calls, 20 recursion depth. You cannot exceed these.

Symptom: The slate file is empty after a call that should have written to it.

Diagnosis:

1. Check for error events with nonFatal: true and "slateWrite failed" in the message. This means the write was attempted but failed.

2. Check the match schema. If the LLM's output doesn't match the schema, the write is a silent no-op (not an error). Look at what the LLM actually produced vs. what the schema requires.

3. Check JSON extraction. The slateWrite extracts the first JSON block from the output. If the output has no JSON (prose only), extraction fails silently.

4. Check the field: extraction. If field: fact is declared but the JSON doesn't have a fact key, extraction fails silently.

Fix: Temporarily remove the match schema to see if the write succeeds without validation. If it does, the schema is too strict. Adjust the schema to match what the LLM actually produces, or adjust the system prompt to produce schema-conforming output.

# Debug: log what the LLM produces before adding schema matching
slateWrite:
  to: { slate: "Debug", folder: raw, file: output.md }
  on: overwrite
  match:
    type: string       # accept any string output

Once you see what the LLM produces, tighten the schema to match.