docs(learnings): add /eval/simulation/* API endpoint reference#13
Merged
dhruva-reddy merged 1 commit intomainfrom Apr 28, 2026
Merged
docs(learnings): add /eval/simulation/* API endpoint reference#13dhruva-reddy merged 1 commit intomainfrom
dhruva-reddy merged 1 commit intomainfrom
Conversation
Appends a full API reference section to simulations.md covering all
six controllers: simulation, scenario, personality, suite, run +
run items, and AI scenario generation. Includes request/response
shapes, query params, auth scopes, and validation rules grounded in
the DTOs at libs/core/src/types/simulation.types/.
Also fixes the stale POST /eval/simulation/suite/{suiteId}/run curl
example — it's been replaced by the unified POST /eval/simulation/run
with discriminated simulations[] entries.
dhruva-reddy
added a commit
that referenced
this pull request
May 2, 2026
#14) ## ELI5 **Problem.** Two of our customer-fork repos (`gitops-mudflap`, `gitops-amazon3p`) kept their own running notes about engine quirks ("man, this is annoying when X happens"). Those notes never made it back upstream, so every new customer hit the same friction. There was also no convention for *anyone* — human or AI — to leave behind a "this should be better" trail. **What this fix does.** Adopts the customer-log format upstream (severity-ranked, evidence-tagged) and seeds it with 20 entries catalogued from both customer logs. Adds a voice-provider cheat-sheet under `docs/learnings/` so the most common 400-rejection class (`voice.speed` on Cartesia) becomes a one-page lookup. Updates `.gitignore` to stop sweeping AI agent handoff scratch (`.agent/`, `.claude/handoffs/`) into commits via `git add -A`. Adds a CLAUDE.md section telling future contributors how to log new entries. **Outcome you'll notice.** Every fresh customer clone of this template inherits the running log on day one. When you hit something annoying, you append an entry in the same change instead of carrying it as folklore. As later stacks land, rows in the triage table flip from `Open` to `RESOLVED` so the file becomes a living changelog. --- Land all the zero-engine-change cleanups in one small PR so the rest of the stack starts from a clean docs surface. - improvements.md (NEW, repo root): adopt the severity-ranked, evidence- tagged catalog format from the Amazon3p customer log. Seeds 20 entries catalogued from gitops-mudflap and gitops-amazon3p. Triage table rows flip from Open → RESOLVED as later stacks land. - docs/learnings/voice-providers.md (NEW): per-provider voice block cheat-sheet (Cartesia vs 11labs vs OpenAI/Azure/Rime/LMNT/Minimax/ Neuphonic/SmallestAI). Closes the manual-lookup half of #9. - docs/learnings/README.md: route the new entry from the index. - AGENTS.md: document multi-file push (closes #14) + voice-providers routing row. - CLAUDE.md: add Improvements log section instructing future contributors (humans + AI agents) to append entries when they hit friction. - .gitignore: cover .agent/, .agent/handoffs/, .claude/handoffs/ so git add -A doesn't sweep PII handoff scratch (closes #13). Drop the legacy "requested improvements.md" line since the local-only convention is superseded by upstream's improvements.md. Closes improvements.md #13, #14. Partial #9 (doc cheat-sheet half). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/learnings/simulations.mdcovering all six simulation controllers in the Vapi monorepo (apps/api/src/simulation/):/eval/simulation— pairs scenario + personality/eval/simulation/scenario— what the tester does + how it's evaluated/eval/simulation/personality— tester assistant config (voice, model, persona)/eval/simulation/suite— named bundle of simulations/eval/simulation/run(+ run items sub-resource) — unified executor/eval/simulation/scenario/generate— AI-generated scenario draftslibs/core/src/types/simulation.types/), query params, auth scopes (TEST_SUITE/TEST_SUITE_RUN), and validation rules.POST /eval/simulation/suite/{suiteId}/runwas replaced by the unifiedPOST /eval/simulation/runwith discriminatedsimulations[].typeentries.README.mdindex entry forsimulations.mdto mention the new API reference.Why
Existing
simulations.mdwas gotcha-focused. FDEs running customer simulations were jumping between Swagger and the wiki to assemble payloads. Putting the reference next to the gotchas makes one-stop context for things like the chat-mode evaluation gotcha, primitive-only schema rule, and 2-slot voice concurrency — each of which is now hyperlinked in spirit between the gotcha section and the API section.What I called out that isn't obvious from the code
/concurrencymath.metadata.{assistant,scenario,personality,squad,simulation}is an immutable snapshot at run-creation time. Editing the source after a run does NOT change historical items.RequestContext.orgId, so a foreign UUID is indistinguishable from a missing one./eval/simulation/scenarioto save.Test plan
simulation.controller.ts,simulationRun.controller.ts,simulationSuite.controller.ts,scenario.controller.ts,personality.controller.ts,simulationGenerate.controller.ts) and verified routes, methods, scopes, and response types match the markdown.simulation.types.ts,scenario.types.ts,personality.types.ts,simulationSuite.types.ts,simulationRun.types.ts,simulationRunItem.types.ts,simulationRunConfiguration.types.ts,evaluationPlan.types.ts) and verified field names, types, optionality, and constraints.🤖 Generated with Claude Code