This is how the project ended up with a toggle in `cdk.json`: ```json "useBedrockKnowledgeBase": false ``` The bundled JSON keyword scoring isn't as semantically rich as vector search, but it runs in-memory inside Lambda and costs nothing at idle. For a conference demo with a 16KB lore file, it's the right call. I'd benchmark both before deciding for a real production system, but I'd check the pricing page first. > **Side note:** if you're using AOSS for dev/test, you can disable redundancy to drop to 2 OCUs instead of 4, which cuts the floor to ~$350/month. Still not free, but worth knowing. The day after I made this decision and committed to the keyword approach, AWS announced that OpenSearch Serverless now supports scaling to zero. The demo was already written. I wasn't going back. But if you're starting fresh today, that changes the math significantly. **Always check the pricing page.** A quick read now saves a very awkward conversation with finance later. --- ## Retries, idempotency, and the dead letter queue The `execute-tools` stage is where game state actually mutates: damage is applied, XP is awarded, inventory changes. These are Lambda invocations, and Lambda invocations fail. Step Functions handles the retry strategy entirely in the state machine definition: ```json "Retry": [{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2, "JitterStrategy": "FULL" }] ``` Full jitter is deliberate. Without it, multiple concurrent workflows retrying the same error pile onto Bedrock in sync. Full jitter staggers them. But retries create a second problem: if `apply-damage` succeeds and the response gets lost, a retry would apply that damage again. The fix is idempotency: cache the tool result in DynamoDB and return the cached version on retry instead of re-executing: ```typescript // lambda/shared/idempotency.ts const key = `${campaignId}:${turnId}:${toolName}:${purpose}`; const cached = await dynamo.get({ Key: { pk: key } }); if (cached.Item) return cached.Item.result; const result = await executeTool(args); await dynamo.put({ Item: { pk: key, result, ttl: now + 3600 } }); return result; ``` The key is scoped to `turnId`. The same tool called in two different turns should produce different outcomes (dice are random), so the cache can't be global. Executions that exhaust all retries route to an SQS dead letter queue. The player gets a safe fallback narrative. Nothing crashes. The DLQ depth metric fires a CloudWatch alarm. No data is lost and the failure is isolated to one turn. --- ## The system controls execution, not the model When the LLM controls its own tool invocation loop (decides which tool, calls it, reads the result, decides what to do next) you get infinite loops, runaway API costs, and failure states that are nearly impossible to debug. Letting the model govern its own execution is like letting your database write its own SQL. In this system, **the model reasons. Step Functions and Lambda execute.** Those are different jobs. After Bedrock returns a narrative and a list of tool calls, the `validate-and-route` Lambda filters them against an explicit allowlist before anything runs: ```typescript // lambda/workflow/validate-and-route.ts const ALLOWED_TOOLS = new Set([ 'roll-dice', 'apply-damage', 'update-inventory', 'award-xp', 'update-location', 'apply-effect', 'use-special-ability', 'update-quest-log' ]); const validated = dmOutput.toolCalls.filter(call => { if (!ALLOWED_TOOLS.has(call.name)) { logger.warn({ toolName: call.name, reason: 'not in allowlist' }); return false; } return true; }); ``` The model cannot invoke a tool that isn't on the list. Business logic never lives inside a prompt; it lives in typed Lambda functions with unit tests. --- ## Structured observability from day one Traditional debugging doesn't translate to AI systems. A stack trace doesn't tell you whether the model misread the lore context, whether the tool arguments were semantically wrong, or whether a retry loop doubled a game effect. Every Lambda in this system emits exactly one log line per invocation: one structured JSON object, queryable in CloudWatch Logs Insights without any parsing overhead: ```typescript console.log(JSON.stringify({ timestamp: new Date().toISOString(), requestId, campaignId, toolName, inputTokens: usage.inputTokens, outputTokens: usage.outputTokens, latencyMs: Date.now() - start, retryCount, success: true })); ``` CloudWatch metric filters extract fields from those log lines into custom metrics in the `NeonScratch` namespace: dice roll totals, Bedrock token usage, monsters defeated per hour, DLQ depth. No code change needed to add a dashboard widget for a new field you're already logging. The alarms are defined in CDK, not clicked in the console: ```typescript new cloudwatch.Alarm(this, 'ControllerLatency', { metric: controllerFn.metricDuration({ statistic: 'p99', period: Duration.minutes(5), }), threshold: 10_000, evaluationPeriods: 2, }); ``` The game's mechanics panel shows these log lines live for the current campaign, each Lambda invocation appearing in order as the turn executes. For the Summit demo, I added a failure-injection shortcut, `Ctrl+Shift+F` in the browser, that forces the tool-execution Lambda to fail, so the audience could watch the retry animation in the workflow trace and see the DLQ alarm flip red in real time. --- ## State lives in DynamoDB, not in the prompt A stateless AI agent is a frustrating one. Without memory of the previous turn, the Dungeon Master doesn't know your HP, what's in your inventory, where you are, which enemies are still alive, or what quests you've accepted. Campaign state (character stats, inventory, location, quest log, active effects, conversation history) lives in DynamoDB. Every turn reads it before invoking Bedrock and writes it back after tool execution: ```typescript // lambda/workflow/persist-campaign.ts await dynamo.update({ Key: { campaignId }, UpdateExpression: ` SET character.hp = :hp, character.inventory = :inventory, currentLocation = :location, conversationHistory = :history `, ExpressionAttributeValues: { ':hp': state.hp, ':inventory': state.inventory, ':location': state.location, ':history': trimmedHistory, } }); ``` Bedrock charges per token, and conversation history grows without bound. After 20 turns, `persist-campaign` calls Bedrock to summarize the narrative so far, stores the summary, and trims the raw turn history. The next turn assembles: system context + summary + last N turns + current action. Continuity survives the context window. --- ## What I'd add before calling it truly production-ready The game is a demo, not a product, and I want to be honest about what's missing: - **Authentication:** there's none. Any request to the API can start or mutate a campaign. - **Rate limiting:** API Gateway has basic throttling configured, but there are no per-user limits. - **Request signing:** the frontend calls the API directly with no auth header. - **Encryption at rest:** DynamoDB tables don't have customer-managed KMS keys. - **Multi-region failover:** single region only. For a real application, I'd add Cognito or a JWT check at the API Gateway layer first. Everything else is addressable without changing the core architecture. **What's production-worthy:** - Retry strategy - Idempotency pattern - Structured logging discipline - Alarm definitions - CDK infrastructure - Separation between model reasoning and system execution Those patterns transfer directly to any AI agent system, game, or otherwise. --- ## The source The full project is on GitHub at [github.com/chaotictoejam/neon-scratch-lounge](https://github.com/chaotictoejam/neon-scratch-lounge). Deploy it with `cdk deploy --all` from the `infra/` directory, point `VITE_API_GATEWAY_URL` at the output URL, and you have a working skeleton. The lore JSON files in `lore/` are the easiest place to adapt it. Swap out Neo-Pawsburg for your own domain, and the DM will narrate your world instead. The five core patterns stay exactly the same. --- *Find me at [linkedin.com/in/jlskiles](https://linkedin.com/in/jlskiles) or [youtube.com/@DrJoanneSkiles](https://youtube.com/@DrJoanneSkiles)*