OpsPilot — A Multi-Agent AI System for Autonomous Product Operations

I tested OpsPilot on 40 real inputs because the only honest way to know whether an AI system works is to measure it on inputs it didn't know it would see. Results by category: • PRD requests: 80 to 90 percent, structured inputs, explicit intent, consistent language patterns • Client email queries: around 83 percent, similar reasons • Informational and FYI messages: 65 to 70 percent, occasional ambiguity creates classification uncertainty • Knowledge base alerts: 40 to 45 percent, rigid system-generated language overlaps semantically with other categories • Overall: approximately 70 percent The knowledge base failure is a label definition problem, not a model problem. The classifier doesn't have enough signal to distinguish a knowledge base alert from an FYI because the category boundaries I defined weren't sharp enough. The fix is prompt refinement and better label definitions, not a bigger model or more training data. That distinction is exactly what an AI PM needs to be able to make. What I'd Build Next The current version processes each message independently with no memory of previous messages and no awareness that Tuesday's PRD request connects to Thursday's feedback thread. Agent memory is the highest-value next problem: persistent context that lets the system understand active workstreams and connect related signals automatically. After that, expanded output types including Jira tickets, meeting summaries, and Slack thread digests. How I'd Measure Success in Production • Time to first draft • Reduction in manual triage • Output acceptance rate: how often users use the generated artifact with minimal editing That last metric is the most honest signal. 70 percent classification accuracy means nothing if users are rewriting 80 percent of outputs. What Building This Taught Me I understand why multi-agent architectures outperform monolithic prompts on complex reasoning tasks, not because I read it, but because I watched a single-prompt system fail in a specific, instructive way and had to redesign around it. I know what transparent reasoning logs do for user trust because I built one and felt the difference. I know what 70 percent accuracy feels like when you thought the system was working. Most candidates applying for AI PM roles have used LLMs. I built a system with them, got the architecture wrong the first time, understood exactly why, redesigned it, measured the output honestly, and documented what I found. That is the difference between knowing about agentic AI and knowing how it actually behaves.