AI portfolio / Case study

I set out to learn evals and ended up mapping the whole AI stack

I wanted to learn how to test AI systems properly. One evals guide grew into a connected reference site of seven guides, from language models to guardrails, each with its sources cited. I built it with Claude over a couple of nights, and shared it so others can learn from my journey.

Role
Solo: structure, technology, spot-checking sources
Audience
Me, and anyone learning how AI products are built
Built with
Claude, HTML, GitHub and Vercel
Time
A few hours over a couple of nights, 2026
  • 7guides, one per layer of the AI application stack
  • 1pilot guide first, to test the format before writing the rest
  • 4guides researched in parallel and shipped in one batch
  • 6 / 7deployment problems solved on the way to a live site

Why evals

The harder my projects get, the more quality assurance matters

AI models aren't deterministic, so you can't assume that what you built works as intended. Evals give you hard data on system quality instead of guesswork.

They also protect what already works. Changing a system prompt, updating a retriever or switching the underlying model can silently break other parts of an app. Evals run automated checks that catch these regressions before a bad update reaches users.

The insight

Evals measure a whole system, so learning them properly meant understanding the layers they measure.

So the scope widened from one guide to a hub with a guide for each layer of the stack. The seven-layer framing follows a widely shared breakdown by Brij Kishore Pandey, credited on the site.

How it was built

One guide, then a pilot, then a batch

  1. An evals guide for a beginner

    I started by asking Claude to teach me AI evals from the basics to advanced methods, including how to build them into coding projects. The result was a walkthrough and a reference guide, with a Sources page, plus Word and PDF versions.

  2. Live on Vercel

    The first guide went online through a drag-and-drop upload.

  3. Widen to the whole stack, starting with a pilot

    Before writing six more guides, I had one pilot written first, on language models, to test the format. RAG followed as the second guide.

  4. Move to Git-based deploys

    To stop re-uploading folders by hand, the site moved to a GitHub repository connected to Vercel, so every push publishes automatically. This took several rounds of troubleshooting.

  5. The last four guides in one batch

    Vector databases, agents, MCP and guardrails were researched in four parallel passes, written to the same template, cross-linked so no link says "coming soon", checked in light and dark mode, and shipped as one package.

Key decisions

Prove the format once, and always publish the whole package

  • Pilot before scale. One guide first meant any problem with the format only had to be fixed once, not seven times.
  • One template for everything. Every guide shares the same stylesheet, navigation and eleven-section rhythm, ending with a Sources page.
  • No partial updates. After any rework, the complete site is published as one package through Git and Vercel, so pages never fall out of step with each other.

What broke

The content was the easy part; getting it online wasn't

Nearly everything that went wrong was between the files on my computer and the live site:

ProblemWhat fixed itStatus
The guide looked unstyled on a second computerDelivered Word and PDF versions insteadUnresolved
Blank table of contents in the Word and PDF filesReplaced the automatic contents with a fixed listFixed
Navigation elements crammed onto one lineSpotted in a screenshot review, fixed in the shared stylesheetFixed
Git not recognized on WindowsInstalled Git and reopened the terminalFixed
Windows PowerShell rejected chained commandsRan each Git command on its own lineFixed
Vercel never redeployed after connecting GitHubDeleted the Vercel project and imported it again from the repositoryFixed
Vercel blocked the deploymentUsed an email address verified on GitHub for commitsFixed

What readers get

Seven layers, each solving a gap the one below leaves open

The AI Factory Stack hub page, showing the seven layers from LLM to Evals as a sequence
The hub presents the seven layers as one connected stack, and credits the framing it builds on.
The Evals field guide, with its eleven sections listed in a side menu, from 'Why evals' to 'Sources'
The original evals guide: eleven sections from "why evals" to a two-week starter plan, ending with its sources.

My role

Claude researched and wrote; I set the direction and checked

I decided how the site is structured, which technology to use, and how updates work. I spot-checked source material against the guides. Claude did the research and writing, and every guide cites what it draws on. Benchmark figures quoted in the guides come from each source's own tests, and the site says so: they show direction and rough scale, not guarantees.

Outcome

A much better understanding, ready to apply

Building the guides gave me a much better understanding of how AI products are put together and how to measure them. I haven't run evals on a project yet; that comes with my next, more demanding builds. So far I'm the main user.

I shared it because I learn best by doing, I enjoy hearing about other people's learning journeys, and I like helping others get closer to their goals. Publishing my own journey is one way to do that.

What I'd do differently

Write the style guide before the next project

The styling problem on my second computer was never solved. And across my guide projects I've learned that a reference link is not enough for an AI to match a format. Next, I plan to create a style guide in Claude Design so every future project starts from the same written rules.

Related: the Claude Code Field Guide case study

See it yourself

Open the AI Factory Stack

Back to the AI portfolio

Let's talk about your product

Hiring for a product role, or need someone to take an AI initiative from idea to production? Send a note and I'll reply within two working days.

Based in
Hungary