AI portfolio / Case study
I set out to learn evals and ended up mapping the whole AI stack
I wanted to learn how to test AI systems properly. One evals guide grew into a connected reference site of seven guides, from language models to guardrails, each with its sources cited. I built it with Claude over a couple of nights, and shared it so others can learn from my journey.
- Role
- Solo: structure, technology, spot-checking sources
- Audience
- Me, and anyone learning how AI products are built
- Built with
- Claude, HTML, GitHub and Vercel
- Time
- A few hours over a couple of nights, 2026
- 7guides, one per layer of the AI application stack
- 1pilot guide first, to test the format before writing the rest
- 4guides researched in parallel and shipped in one batch
- 6 / 7deployment problems solved on the way to a live site
Why evals
The harder my projects get, the more quality assurance matters
AI models aren't deterministic, so you can't assume that what you built works as intended. Evals give you hard data on system quality instead of guesswork.
They also protect what already works. Changing a system prompt, updating a retriever or switching the underlying model can silently break other parts of an app. Evals run automated checks that catch these regressions before a bad update reaches users.
The insight
Evals measure a whole system, so learning them properly meant understanding the layers they measure.
So the scope widened from one guide to a hub with a guide for each layer of the stack. The seven-layer framing follows a widely shared breakdown by Brij Kishore Pandey, credited on the site.
How it was built
One guide, then a pilot, then a batch
-
An evals guide for a beginner
I started by asking Claude to teach me AI evals from the basics to advanced methods, including how to build them into coding projects. The result was a walkthrough and a reference guide, with a Sources page, plus Word and PDF versions.
-
Live on Vercel
The first guide went online through a drag-and-drop upload.
-
Widen to the whole stack, starting with a pilot
Before writing six more guides, I had one pilot written first, on language models, to test the format. RAG followed as the second guide.
-
Move to Git-based deploys
To stop re-uploading folders by hand, the site moved to a GitHub repository connected to Vercel, so every push publishes automatically. This took several rounds of troubleshooting.
-
The last four guides in one batch
Vector databases, agents, MCP and guardrails were researched in four parallel passes, written to the same template, cross-linked so no link says "coming soon", checked in light and dark mode, and shipped as one package.
Key decisions
Prove the format once, and always publish the whole package
- Pilot before scale. One guide first meant any problem with the format only had to be fixed once, not seven times.
- One template for everything. Every guide shares the same stylesheet, navigation and eleven-section rhythm, ending with a Sources page.
- No partial updates. After any rework, the complete site is published as one package through Git and Vercel, so pages never fall out of step with each other.
What broke
The content was the easy part; getting it online wasn't
Nearly everything that went wrong was between the files on my computer and the live site:
| Problem | What fixed it | Status |
|---|---|---|
| The guide looked unstyled on a second computer | Delivered Word and PDF versions instead | Unresolved |
| Blank table of contents in the Word and PDF files | Replaced the automatic contents with a fixed list | Fixed |
| Navigation elements crammed onto one line | Spotted in a screenshot review, fixed in the shared stylesheet | Fixed |
| Git not recognized on Windows | Installed Git and reopened the terminal | Fixed |
| Windows PowerShell rejected chained commands | Ran each Git command on its own line | Fixed |
| Vercel never redeployed after connecting GitHub | Deleted the Vercel project and imported it again from the repository | Fixed |
| Vercel blocked the deployment | Used an email address verified on GitHub for commits | Fixed |
What readers get
Seven layers, each solving a gap the one below leaves open
My role
Claude researched and wrote; I set the direction and checked
I decided how the site is structured, which technology to use, and how updates work. I spot-checked source material against the guides. Claude did the research and writing, and every guide cites what it draws on. Benchmark figures quoted in the guides come from each source's own tests, and the site says so: they show direction and rough scale, not guarantees.
Outcome
A much better understanding, ready to apply
Building the guides gave me a much better understanding of how AI products are put together and how to measure them. I haven't run evals on a project yet; that comes with my next, more demanding builds. So far I'm the main user.
I shared it because I learn best by doing, I enjoy hearing about other people's learning journeys, and I like helping others get closer to their goals. Publishing my own journey is one way to do that.
What I'd do differently
Write the style guide before the next project
The styling problem on my second computer was never solved. And across my guide projects I've learned that a reference link is not enough for an AI to match a format. Next, I plan to create a style guide in Claude Design so every future project starts from the same written rules.