Nothing crashed. No test failed. The answer was still wrong.
Our peer-reviewed paper on testing an AI platform where the answer can change every run and still be right. A request goes to the wrong agent, the wrong agent answers confidently, and thousands of unit tests pass. They were never going to catch it.
- Published in the PNSQC 2026 proceedings
- Written from two years of testing Swa in production
Get the paper summary
We'll email you the summary and the link to the proceedings.
You're on the list.
Watch your inbox for the summary and the proceedings link. Until then, the short version is on our blog.
Four questions, asked in order
Each one is harder to automate than the one before it.
- Did the code execute?
- Did the right agent answer?
- Did it do the right thing?
- Was the output any good?
The testing pyramid comes out upside down
Automation is thick at the base and human judgment is heaviest at the top. The paper covers what our AI review agent checks on every pull request, why the safety layers fail closed, and the half we're still building.
The three of us wrote it
“Testing Intelligence: A Journey of Quality for a Multi-Modal AI Orchestration Platform”, 44th Pacific Northwest Software Quality Conference, Portland, Oregon
Mike Sirchuk
Founder and CEO, Swa
Ben Pickett
Co-founder and COO, Swa
Rahul Ravel
Advisor, Swa