Kaan · The Officer's Manual
2026-02-15
AI Makes your Systems Reliable
This type of technology comes every 150 years, if not more. Make use of it as a starving man would make use of food.
I talk a lot about AI, the ways that you can spin up agents, make it so you evaluate different parts of your response systems. Ultimately AI comes in on problems that are tough enough that getting an algorithm for said problem is time consuming, but verifiable enough that it can be consistently evaluated.
Here I will be talking about how AI makes your system reliable. →
Introduction to the problem statement: The thing about current day code is that it is nearly infinitely replicatable. The market is filled with builders and some of these builders can close sales. Reliability becomes a good part of the arsenal in our advance.
Problem constraints Modern AI & Agents still have problems. They struggle with numerical precision, they fail at complex tool execution, and they hallucinate. However, the enterprise sales environment operates under strict constraints:
- Predictability:The system must behave the same way twice.
- Uptime: The system cannot fail when it matters.
Rigorous proof:
We can derive the solution through simple logical deduction:
- As AI lowers the cost of code generation, the number of competitors producing "good enough" software increases exponentially.
- As the number of competitors increases, market noise maximizes. Enterprise buyers are overwhelmed with undifferentiated choices.
- Reliability is important for enterprise and customers
- Enterprise buyers are fundamentally risk averse. They will always choose a boring, reliable system over a volatile one.
- Therefore, as features become commoditized and indistinguishable; Reliability becomes a strong differentiation coefficient.
Have you heard about a smoke test? Hardware builders came up with it; rule of thumb goes that if the circuit smokes, turn it off. Similar situation with AI:
- Your AI assisted systems would make use of a strict stop condition
- Adding more and more QA into your systems make it so it maintains better
- A lot of people use the word “self healing” around, you can make sure your systems actually are that by good unit tests
The SDLC might have changed on the surface but the depths have spoils that stay relevant. I always go for depth.
Eval Driven Development:
Lots of people ship AI demos. Actual professionals ship AI systems. The difference is Evals.
You need to stop the "vibes based" testing and move to metric based, as soon as possible
The Two Types of Evals you need:
Deterministic Evals:
- Did the agent output valid JSON?
- Did the tool call contain the correct arguments?
- Use case: Preventing your code from crashing because the LLM forgot a closing bracket.
Probabilistic Evals:
- This is where you use a smarter model (Claude 4.5 Sonnet) to grade the homework of your smaller production model.
- Prompt: "You are a senior QA engineer. Grade the following response on a scale of 1-10 for Relevance to the context."
- Use case: Ensuring your sales bot isn't promising features you don't have.
Why this matters for GTM:
If you are selling AI, your customer’s fear is "What if it says something wrong?" Your answer is not "It won't." Your answer is "We have a continuous evaluation pipeline that runs 500 adversarial test cases before every deployment."
That is how you close the deal.
Agent skills dropped Q4 last year
Agent skills make excellent use of QA and evals, you should use it to the furthest extremity. By whipping up skills to evaluate, regulate and maintain your AI systems, your code would be good without the need for further manual changes.
Takeaways
Spin up a sandbox
Sandboxes make it so the models have a place to do more computation heavy tool calls in a space where it would not mess up your entire system. I’m sure you have seen the post where an LLM apologizes for deleting prod database. Better test your models so you don’t get that
Add more evals
I use evaluations in data enrichment pipelines quite a bit. Mostly thanks to the modern tools having built in evals help with this tremendously. But having a consistent evaluator scheme for most tasks will make your systems more reliable.
Get your skills up
QA related parts of the SDLC are the most boring part of the job. That’s why they give those jobs to interns and juniors, no sane person would work in QA after a year or two. But with usage of skills (Claude, Cursor, whichever works for you), it is possible to skip some of the more annoying steps of the workflow.
Keep up the intensity,