YOUR SOURCE PDF
Returns policy
Items can be returned within 30 days of purchase.
Source reference · §2
HUMAN JUDGMENT. TRACEABLE EVIDENCE.
Your AI sounds right.
Let’s make sure
it is.
EvalAnchor evaluates AI systems that answer from retrieved documents, cached context, or long-term memory. Our reviewers check each answer against the knowledge it was given and report what is correct, what is unsupported, and what to fix before you release.
For product and AI teams shipping knowledge assistants, support tools, and personalized AI
YOUR AI’S ANSWER
“You have 60 days to return your purchase.”
[1] Returns policy · §2 EVALANCHOR REVIEW
The source says 30.
The AI says 60.
Correct source. Unsupported answer.
Fictional example · Method illustration
A FOCUSED STARTYour evaluation pilot
50–100evaluation cases
2reviewers per answer
3architectures covered
See what’s included THE TWO-MINUTE OVERVIEW
Watch one answer
get checked.
Follow a single refund question through a RAG system, from retrieval to a reviewed finding.
THREE ARCHITECTURES, ONE STANDARD
See where your system
gets its knowledge.
Switch between retrieval, cache, and memory to follow how a question becomes an answer, and where our evaluation steps in.
RAG
Retrieval-augmented generation
The system searches a document index when a question arrives and passes the best-matching passages to the model.
Where it tends to fail
- Relevant document never retrieved
- Off-topic passage shapes the answer
- Claim not supported by any passage
- Citation points to the wrong source
What we measure
- Retrieval precision
- Context relevance
- Faithfulness
- Citation support
- Answer completeness
- Retrieval-failure analysis
WHAT WE EVALUATE
Three evaluation services.
One review standard.
Each service uses private evaluation cases, expert human review, and failure analysis. We evaluate the system you run; we don’t build or tune it.
RAG evaluation
Retrieval-augmented generation
For assistants that search your documents before answering. We check whether the right passages were found and whether the answer stays true to them.
What we evaluate
- Retrieval precision
- Context relevance
- Faithfulness
- Citation support
- Answer completeness
- Retrieval-failure analysis
CAG evaluation
Cache-augmented generation
For assistants that answer from knowledge preloaded into context. We check coverage and freshness, and what happens when the answer isn’t in the cache.
What we evaluate
- Cache coverage
- Context selection
- Cache freshness
- Unsupported-answer behavior
- Large-context reliability
- Latency
- Cost-quality tradeoffs
MAG evaluation
Memory-augmented generation
For assistants that remember users and ongoing work across sessions. We check what is stored, when it is recalled, and who can see it.
What we evaluate
- Memory correctness
- Recall timing
- Memory updates
- Contradiction handling
- Personalization consistency
- Cross-session behavior
- Privacy and memory leakage
Also in preparation: language evaluation (starting with Bangla and English) and data labeling. Register interest
SIDE BY SIDE
Different architectures
fail in different ways.
No architecture is better in every case. What matters is how your knowledge changes, how fast answers must arrive, and how personal they need to be.
| RAGRetrieval-augmented generation | CAGCache-augmented generation | MAGMemory-augmented generation | |
|---|---|---|---|
| Knowledge source | Documents retrieved at question time from an index or search system. | A fixed knowledge set preloaded into the model’s context or key-value cache. | Memories written from earlier interactions and recalled in later sessions. |
| Best use cases | Large or frequently updated document collections, and answers that need citations. | Smaller, stable knowledge sets where response time matters, such as manuals or policy sets. | Assistants that should remember people, preferences, or ongoing work across sessions. |
| Primary failure modes | Missed or irrelevant passages, unsupported claims, mismatched citations. | Stale content, wrong section used, invented answers when information is absent. | Wrong or outdated memories, badly timed recall, conflicts, cross-user leakage. |
| Evaluation priorities | Retrieval quality, groundedness, citation accuracy. | Coverage, freshness, behavior when the answer is missing, cost against quality. | Memory correctness, updates and deletion, consistency, privacy boundaries. |
| Example metrics | Retrieval precision, context relevance, faithfulness, citation support. | Cache coverage, context selection, freshness, large-context reliability, latency. | Recall timing, update accuracy, contradiction handling, memory leakage. |
| Human-review requirements | Reviewers verify each important claim against the passage it cites. | Reviewers check answers against the source of truth behind the cache and judge refusals. | Reviewers follow multi-session scenarios and judge whether each recalled memory is correct, relevant, and permitted. |
Scroll sideways to compare all three.
LOOK PAST THE CONFIDENCE
A convincing answer
can still be wrong.
Whether knowledge comes from retrieval, a cache, or memory, we separate three questions so your team can see exactly where an answer breaks down.
Explore our evaluation methodIs the answer correct?
Compare the claim with the expected answer.
Does the evidence support it?
Find the exact passage or memory behind each claim.
Does the citation check out?
Confirm the reference exists and applies.
SAMPLE EVALUATION WORKFLOW
Seven steps from risk
to regression set.
The same structure applies to RAG, CAG, and MAG. Only the cases and the checks change.
A good fit if you
- Run an assistant that answers from documents, a preloaded knowledge set, or user memory
- Need to decide whether a new version is ready to release
- Want to know why answers fail, not just how often
Not included
- Building, training, or fine-tuning your AI system
- Changing your implementation for you
- Ongoing production monitoring
- Together
Step 1: Define the use case and risks
Agree which users, questions, and failure risks matter, and the rubric each answer will be judged against.
OutputScope, risk list, review rubric
- EvalAnchor
Step 2: Build private evaluation cases
We write realistic questions with reference answers from your authorized knowledge. Your cases stay private and separate from any public sample set.
OutputPrivate case set
- Your team
Step 3: Run your system
You run the cases through your system as it is configured today. We record versions and settings; we never change the system.
OutputFrozen responses and configuration record
- EvalAnchor
Step 4: Apply automated metrics
Automated checks, including model-based scoring, flag likely problems at scale. They decide what to look at first, not the outcome.
OutputTriage signals for every case
- EvalAnchor
Step 5: Conduct expert human review
Two reviewers judge each answer against the evidence, record their reasoning, and resolve disagreements explicitly.
OutputJudged cases with rationale
- EvalAnchor
Step 6: Analyze failures
Findings are grouped into a failure taxonomy, so patterns such as missed retrieval or stale memory are visible rather than hidden in a score.
OutputFailure taxonomy and patterns
- Delivered to you
Step 7: Deliver results and regression-ready datasets
You receive the report, machine-readable results, and a case set you can rerun after every change.
OutputReport, CSV + JSONL, regression set
A FOCUSED PILOT
Start with one system.
Leave with evidence.
A fixed-scope pilot on the RAG, CAG, or MAG system you run today. Scope and price are agreed after we review your use case.
- 50–100 evaluation cases built for your use case
- An agreed review rubric
- Human-reviewed outputs with documented reasoning
- A failure taxonomy for your system
- An executive summary for decision makers
- Machine-readable results (CSV + JSONL)
- Prioritized final recommendations
Executive summary
What works, what fails, and what to fix first.
01Reviewed results
Every case, judgment, and rationale.
02Regression set
Your private cases, ready to repeat.
03THE BIGGER PICTURE
Better AI starts with
better evaluation.
Leading research teams test more than how an answer sounds. They test how well it stands up to the evidence.
INDUSTRY EXAMPLES · NO AFFILIATION OR ENDORSEMENT
824challenging questions
FRAMES · 2024
Tests whether AI can retrieve information and reason across multiple documents.
Read the researchMicrosoft
4classes of questions
BenchmarkQED · 2025
Examines different kinds of questions, from specific facts to whole-dataset understanding.
Read the researchSalesforce AI Research
7contextual benchmarks
ContextualBench · 2024
Evaluates how well answers use their context, including difficult and conflicting evidence.
Read the researchPublished benchmark sizes and categories, not EvalAnchor performance results. Different benchmarks measure different things. These companies are not our customers or partners.
GOOD QUESTIONS
A little more clarity.
New to AI evaluation?
Start here.
What are RAG, CAG, and MAG?+
Three ways an AI system brings in knowledge. Retrieval-augmented generation (RAG) searches documents for each question. Cache-augmented generation (CAG) preloads a fixed knowledge set into the model’s context. Memory-augmented generation (MAG) stores and recalls information across sessions.
Which architecture should we use?+
There is no universal answer. It depends on how large and changeable your knowledge is, how fast answers must arrive, and how personal they need to be. We evaluate the system you run and show where it holds up and where it does not.
Do you build or fix the system?+
No. We evaluate your system as it is configured. The pilot provides evaluation cases, reviewed findings, and recommendations. Building, training, implementation changes, and ongoing monitoring are outside its scope.
Do automated judges replace your reviewers?+
No. Automated metrics, including model-based scoring, help us decide which cases to look at first. Every reported finding is judged by people, and disagreements between reviewers are resolved explicitly.
What do we need to provide?+
Authorized knowledge for the use case, a way to run your system on our cases, and the configuration you want tested. We agree the cases, rubric, and access boundaries before work starts.
How long does a pilot take, and what does it cost?+
We agree the timeline during scoping. A focused RAG pilot takes about three weeks once scope, access, and the review rubric are agreed. We quote after reviewing your use case, case complexity, and acceptance criteria.
Are the companies above your customers?+
No. Google, Microsoft, and Salesforce are industry research examples only. The statistics describe their published benchmarks, not EvalAnchor results or relationships.
START WITH THE ANSWERS THAT MATTER
Evaluate your AI system
before your next release.
Tell us whether it retrieves, caches, or remembers. We’ll scope a
focused pilot around your users and your risks.