How to Test and Validate Ladders

Every ladder config is validated at load time. The validator checks ~25 rules covering structure, references, and semantics.

Validate via the API:

curl -X POST https://api.redeo.ai/v1/validate \\
  -H "Authorization: Bearer $REDEO_API_KEY" \\
  -H "Content-Type: application/json" \\
  -d '{
    "config": { ...your ladder config... }
  }'

Response:

{
  "valid": true,
  "errors": []
}

Or with errors:

{
  "valid": false,
  "errors": [
    "steps[1].fields[0].from.stepId not found: did you mean \"draft\"?",
    "executionBudget.maxSpend exceeds platform max ($50)"
  ]
}

The validator does not stop at the first error — it reports all problems found. Fix them all, then re-validate.

Validate in Foundry: The editor validates automatically on save. Errors appear inline with the step ID and field name.

Common validation errors:

• Field references to non-existent step IDs.
• Gate jump targets that don't exist.
• Slate write targets (slate/folder/file) not declared in slates:.
• integerRange with min > max.
• if: and continueIf: declared on the same step (mutually exclusive).
• executionBudget values exceeding platform maximums.

After execution, the trace shows every event in order. Use it to verify:

1. Expected steps ran. Look for step_start and step_complete events for each step you expected.

2. Gates evaluated correctly. node_survivors events show which nodes passed. Compare against expected survival.

3. Jumps fired at the right time. step_start events with jump: data show when a jump occurred and where it went.

4. Slate writes succeeded. Check for error events with nonFatal: true and "slateWrite failed" — these indicate schema validation failures (silent no-ops).

5. No ceiling exhaustion. Check for error events with ceiling_exceeded: true. If present, the ladder hit a budget limit.

Programmatic trace access:

# Get the trace for a completed instance
curl https://api.redeo.ai/v1/pipelines/instances/$INSTANCE_ID/trace \\
  -H "Authorization: Bearer $REDEO_API_KEY"

The trace is a JSON array of events, each with type, data, and timestamp. Filter by event type to find specific behaviors.

For published ladders, regression testing ensures changes don't break existing behavior.

Pattern 1: Fixed test cases.

Maintain a set of test inputs with expected behaviors (not exact outputs — LLM outputs vary). Check:

• The ladder completes without error.
• The expected steps ran (via trace).
• The output passes a quality gate (e.g., jsonMatches on the response).

# Test script
for test_case in test-cases/*.json; do
  INPUT=$(cat "$test_case")
  RESPONSE=$(curl -s ... -d "$INPUT")
  
  # Verify the response is valid
  echo "$RESPONSE" | jq '.choices[0].message.content' > /dev/null
  
  # Verify the trace shows expected steps
  INSTANCE_ID=$(echo "$RESPONSE" | jq -r '.id')
  TRACE=$(curl -s ... "/trace")
  echo "$TRACE" | jq '[.[] | select(.type == "step_complete")] | length'
done

Pattern 2: A/B comparison.

When modifying a ladder, run both versions on the same inputs and compare:

• Did the output quality improve?
• Did the cost (LLM calls, spend) change?
• Did the latency change?

# Run old version
OLD=$(curl -s .../v1/you/ladder-v1/chat/completions ...)

# Run new version
NEW=$(curl -s .../v1/you/ladder-v2/chat/completions ...)

# Compare
diff <(echo "$OLD" | jq '.usage') <(echo "$NEW" | jq '.usage')

Pattern 3: Budget verification.

Verify the ladder stays within budget on adversarial inputs:

# Edge case: very long input
curl ... -d '{ "messages": [{ "role": "user", "content": "'$(python3 -c "print('x' * 10000)")'" }] }'

# Check that spend didn't exceed the cap
TRACE=$(curl -s .../trace)
echo "$TRACE" | jq '[.[] | select(.type == "error" and .data.ceiling_exceeded)] | length'

The spend cap guarantees the ladder won't exceed its budget. But you should verify the output is still useful when the cap is hit.