How to Optimize Cost and Latency

LADR cost is fundamentally simple: total cost = sum of all LLM call costs. Each step dispatches N LLM calls (where N = nodes:); each loop iteration repeats the steps array.

Total calls = loops × Σ(step node counts)
Total cost  = Σ(call cost)

For a 3-step ladder with nodes [5, 5, 1] running 1 loop:

• Step 1: 5 calls
• Step 2: 5 calls (minus pruned)
• Step 3: 1 call
• Total: 11 calls

For the same ladder running 3 loops:

• Total: 33 calls (if no pruning)

Pruning reduces cost. If step 2's gate prunes 3 of 5 nodes, step 3 sees only 2 survivors (via multi_ingest). The synthesize step still makes 1 call, but its prompt is shorter (2 inputs instead of 5), reducing per-call token cost.

Backward jumps multiply cost. A reflexion loop that runs 3 iterations effectively triples the steps array. The hop ceiling bounds this, but each iteration is a full pass.

1. Use knobs to offer cheap and expensive modes.

knobs:
  branches:
    name: Branches
    type: nodes
    input: slider
    steps:
      - { title: Fast, value: 3, default: true }    # 7 calls
      - { title: Balanced, value: 5 }                # 11 calls
      - { title: Wide, value: 8 }                    # 17 calls

Callers choose based on their budget. The default (Fast) is the cheapest useful mode.

2. Prune aggressively. A tight gate is a cost saver. If 5 candidates cost 5 verify calls but only 1 survives, the synthesize step processes 1 input instead of 5.

# Tight gate: only scores 4-5 survive
if:
  integerRange: [4, 5]
  then: continue
  else: abort            # don't waste downstream cost on weak candidates

3. Use early exit jumps. If the first candidate is excellent, skip the rest.

- id: draft
  if:
    jsonMatches: { properties: { confidence: { minimum: 0.9 } } }
    then: { jump: { stepId: answer } }    # skip verify + synthesize
    else: continue

On high-confidence inputs, the ladder costs 2 calls instead of 11.

4. Reduce loop iterations. maxLoops is a knob and an API parameter. Default to 1 for most ladders; increase only when cross-loop refinement is needed.

5. Use cheaper models for scoring. Verification doesn't need the best model — a cheaper model can score candidates adequately. Use allowedTargets to let callers choose, or document which models work for which step.

Latency = wall-clock time from API call to response. LADR latency depends on:

1. Sequential depth: how many steps run one after another.
2. Parallel width: how many nodes run at once (parallel = lower latency per step).
3. Loop iterations: how many times the steps array repeats.
4. Retrieval rounds: how many rounds the retrieve loop runs.

Strategies:

Use group steps for parallel independent work. Three reviewers in a group step take the latency of the slowest one, not the sum.

# Parallel: latency = max(security, style, correctness)
- id: reviewers
  type: group
  steps: [security, style, correctness]

# Sequential would be: latency = security + style + correctness

Reduce sequential depth. Fewer steps = lower latency. Can you combine draft + verify into one step with self-assessment in the prompt?

Use early exit. Forward jumps skip steps, reducing latency on easy inputs.

Avoid unnecessary loops. Each loop adds a full pass of latency. Use maxLoops: 1 unless cross-loop refinement is essential.

Cache results via memory. If the same question is asked repeatedly, a slate with prior answers lets the ladder return cached results without recomputing.

The executionBudget caps the worst case. Set it based on what you're willing to spend per call, not what you expect to spend.

executionBudget:
  maxSpend: 2.0       # $2 per call — generous for most ladders
  maxHops: 100        # 100 step executions — enough for 5-10 reflexion iterations
  maxLlmCalls: 200    # 200 API calls — plenty for wide fan-out

How to choose values:

The spend cap is your guarantee. No matter what happens — infinite loops, deep recursion, wide fan-out — the caller will never pay more than maxSpend. The runtime exits gracefully with the best-so-far output when the cap is hit.

Monitor actual spend. The trace includes spend tracking. Check the error event with ceiling_exceeded: true to see if your ladder is hitting the cap. If it is, either raise the cap or optimize the ladder.