What Auditing a Production AI Platform Taught Me About Engineering
I spent two weeks systematically evaluating an AI platform's production behavior. What I found contradicted most of what the documentation claimed - and changed how I think about building systems.
The Moment the Documentation Stopped Matching Reality
I found the first real problem on day two, when I traced a production API response back to the model serving layer and discovered the latency documentation claimed 40 milliseconds. Production was 340 milliseconds - almost nine times slower - and nobody had noticed because no one was measuring it.
The documentation didn't lie. It was just measuring something different from what production was doing.
That gap - between what the system was supposed to do and what it was actually doing - was the theme of the entire audit. I'd been hired to evaluate an AI platform's production readiness. What I got was a two-week lesson in why documentation is never the source of truth, why observability is not optional, and why the most important engineering skill is the willingness to go look.
Why The Audit Happened
The organization had grown quickly. A small team had built an AI-powered platform over eighteen months - RAG-based retrieval, a custom model serving layer, multiple API integrations, a React frontend. It worked. Users were on it. But the engineering team had rotated twice, and the second set of engineers had inherited a system they didn't fully understand.
They asked me to answer a specific question: can this platform scale, and if not, where does it break first?
What I expected to find: a standard architecture evaluation. Mismatched resource allocations, missing autoscaling, maybe some N+1 query patterns in the retrieval layer.
What I actually found: a system where the documented behavior and the production behavior had diverged so significantly that "read the docs" was actively harmful advice.
The constraints were real. I had two weeks. No write access to production. Limited access to logs - what existed was unstructured and inconsistently labeled. No metrics dashboard; the team used logs plus intuition. The codebase was large enough that no single person could hold all of it in their head.
This is the situation most production audits happen in: incomplete information, real time pressure, and a system whose behavior was a combination of what was built and what had accumulated.
What I Investigated
Architecture and Request Flow
I started by reconstructing the actual request path - not from documentation but from code plus production logs. The critical path looked straightforward on paper: API gateway → Auth middleware → Retrieval service → Model inference → Response formatting.
In practice, there were two additional hops that weren't documented and a retry layer that wasn't in any diagram. The retry layer was returning cached responses for failed inference requests without marking them as cached in the response headers. So failed requests were silently returning stale data and looking like successes.
Observability and Monitoring
There was no metrics dashboard. There were logs - dense, unstructured logs that required manual grep to make sense of. Error rates were inferred from complaints, not measured.
I set up a temporary monitoring layer over a weekend. What I found: the error rate was 6.3 percent, not the "less than 1 percent" the team had estimated from user complaints.
Six percent of AI inference requests were silently failing. Most users didn't complain because the failures looked like slow responses, not errors.
Model Behavior Under Load
I ran structured load tests against the inference layer. The model serving infrastructure was undersized for sustained load - under sustained concurrent requests, latency degraded nonlinearly. The degradation wasn't from the model itself but from the token queue, which had no size limit and no timeout policy.
Requests would queue, wait, and either complete slowly or timeout silently. The timeouts didn't surface as errors - they surfaced as users complaining the system "felt slow today."
Documentation Audit
I went through every internal document, README, and API reference. Then I tested every documented behavior against production. The hit rate: about 60 percent of documented behaviors matched production. The other 40 percent were either subtly wrong - the documented behavior was the original behavior, not the current one - or missing entirely.
The most consequential gap: the documentation described a three-tier retrieval system. The vector search tier had been replaced with a simpler embedding lookup eight months prior. Nobody had updated the docs. The rules-based fallback, which the documentation called "legacy," was actually the primary retrieval path in production.
Assumptions That Turned Out Wrong
Assumption 1: "The docs are close enough"
I assumed the documentation was approximately accurate - the kind of gap that happens with any evolving system.
The gap was 40 percent. Not minor drift - a fundamental misrepresentation of how the system worked. You can't make good engineering decisions about a system you don't understand, and you can't understand a system whose documentation actively misleads you.
Assumption 2: "Real-time means real-time"
The API had a documented response time of under 200 milliseconds. I assumed this was measured from request receipt to first byte.
It was measured from after the retrieval step. The retrieval step, which could take 300-600 milliseconds, was not included in the measurement. So the user-facing latency was 500-800 milliseconds, not under 200. The docs weren't lying about the number - they were measuring a different thing.
Assumption 3: "The model is the bottleneck"
I assumed the inference layer would be the constraint. Model inference is expensive and slow - it's the obvious bottleneck in any AI platform.
The actual bottleneck was the retrieval layer. Every inference request waited for the retrieval to complete before the model could start. The model sat idle for 60 percent of the total request time while retrieval happened. Optimizing the model would have yielded maybe 20 percent improvement. Optimizing retrieval would have yielded 60 percent.
Lessons Learned
Production is the source of truth
Everything I thought I knew about the system came from reading code and documentation. Production was the thing that actually mattered, and I didn't have access to it until I went and looked.
This is why observability isn't an optional nice-to-have. It's the only thing that lets you close the gap between your mental model and reality. When you don't measure production behavior continuously, you're running blind.
Observability before optimization
I found three separate places where engineers had tried to optimize something based on assumptions about production behavior. None of the optimizations had been validated against actual production data. One of them had made things worse.
The correct sequence is: measure first, understand second, optimize third. Skipping steps one and two and going straight to three is how you get confident wrong answers.
Documentation drift compounds
The 40 percent mismatch wasn't accumulated dishonesty - it was accumulated neglect. Every small architecture change, every undocumented behavior, every "we'll update the docs later" decision compounded. Eighteen months of compounding left a system whose documentation was not just incomplete but actively misleading.
The only way to prevent this is to make documentation part of the definition of done. If documentation isn't current at the moment of shipping, it never gets updated.
Systems thinking isn't optional at scale
The most important finding of the audit wasn't any single issue - it was the discovery that the system couldn't be understood by examining any one layer. The retrieval layer's synchronous behavior was a problem because of how it interacted with the inference layer's queuing policy. The inference layer's queuing policy was a problem because of how it interacted with the API gateway's retry logic.
None of these looked broken in isolation. Together they produced a system that was fragile in ways nobody had measured.
What I Would Do Differently Today
First, I would instrument before I optimized. Not "add some logging" - add structured metrics with cardinality labels, histogram distributions for latency, and error categorization. The data should be in a format you can query, not just dump to a file you grep later.
Second, I would define and track SLOs from the first week. Not just availability - latency SLOs, error rate SLOs, retrieval latency SLOs. An SLO is a contract with your users. If you don't have one, you don't know when you're violating it.
Third, I would make documentation part of code review. Not a separate doc step - the PR includes the documentation update, or it doesn't merge. The alternative is a growing gap between what the system does and what the docs say.
Advice For Other Engineers
If you're inheriting a production system, audit it before you optimize it. Not a quick look - a real audit. Structured logging for at least one week. Trace at least fifty requests end-to-end. Test at least five documented behaviors against production. The goal is to close the gap between your mental model and the actual system.
If you're evaluating an AI platform - whether as an engineer building on top of it or as someone auditing it - treat the model as the one part you can rely on being documented correctly. Everything around the model - retrieval, prompting, routing, caching, retry logic - is where the actual complexity lives and where documentation is most likely to be wrong.
Final Thoughts
The most dangerous state in engineering is being confident about something you haven't measured.
I've seen systems that everyone on the team "knew" were stable, performant, and well-understood - until someone ran a load test, or read the production logs carefully, or traced a single request through every hop. The confident understanding was a fiction maintained by the absence of information.
The gap between "I think this works" and "this works in production" is the gap that kills systems at the worst moment. The only bridge across that gap is observability: knowing what your system is actually doing, not what you intended it to do.
Documentation, intuition, and code review are necessary. They're not sufficient. At some point you have to go look.
Related Articles
The Hidden Cost of LLM Context
Every RAG system I've worked with had the same silent failure mode: retrieval that looked correct but wasn't. Here's what I learned about managing context quality in production AI systems.
What 283 Production Requests Taught Me About Assumptions
I analyzed 283 production API requests to understand how users actually interact with an AI platform. What I found contradicted several assumptions our team had made about how the system would be used.