LLM applications & assistants
Assess answer correctness, instruction adherence, multi-turn behaviour, refusal boundaries and structured output.
Judge whether the user completed the task, not whether the response sounded convincing.
Services / AI Safety & Reliability
AI testing &
evaluation.Evidence before release.
We test AI applications against the jobs they need to do and the boundaries they must respect. From representative datasets and failure analysis to repeatable evaluations, release checks and a clear view of what remains uncertain.
Task-specific coverage. Reproducible evidence. Accountable decisions.
The business case
A good demo is not a release criterion.
An AI application can answer one question well and fail on the next phrasing, permission boundary or dependency outage. We help your team understand those failure modes and make a reasoned decision about what to fix, what to constrain and what is ready for a bounded rollout.
Questions this service helps answer
Is the new model actually better for our task? Which users and inputs are poorly served? Can the assistant take an unauthorised action? What happens when retrieval or a tool fails? What evidence should be attached to the release?
What testing cannot promise
No test suite proves that every future answer will be correct or that every attack is covered. Findings depend on the agreed scope, data and access. We make those limitations visible rather than presenting a single score as a guarantee.
What we evaluate
The application. The workflow. The outcome.
We can evaluate an existing system, a candidate change or a new pilot. Scope follows the actual product risks; not every engagement needs every category below.
Retrieval & knowledge systems
Test retrieval relevance, evidence coverage, citation support, freshness and permission-aware search.
Separate a search failure from a generation failure so the right layer gets fixed.
Agents & tool workflows
Evaluate tool selection, arguments, authorisation, approvals, state transitions and recovery across complete trajectories.
A correct final answer does not excuse an unauthorised intermediate action.
Document & multimodal AI
Check critical fields, layout variation, missing values, source evidence and review routing across representative inputs.
Report field-level and document-level results; aggregate accuracy can hide costly errors.
Voice & conversation systems
Test transcription, interruptions, noisy audio, turn-taking, action confirmation and human transfer.
Evaluate the end-to-end conversation as well as the individual speech and language components.
Model & prompt changes
Compare candidates with the accepted baseline on task quality, safety, latency and total operating cost.
A model upgrade is a release decision, not an assumption that newer is better.
A defensible comparison
Same cases. Same conditions. Clear differences.
Baseline and candidate runs use the same versioned evaluation set and relevant execution conditions. We separate quality, safety, reliability and cost, then inspect failures instead of hiding them behind an overall score. The diagram illustrates the process, not measured customer results.
01
Run a controlled comparison
Record the application and model configuration, data snapshot and tool environment. Repeat stochastic tasks where needed and account for run-to-run variation.
02
Inspect evidence & failures
Review outputs, retrieved sources, trajectories and confirmed side effects. Classify the error and its severity so remediation targets the actual cause.
03
Make a scoped decision
Compare against pre-agreed acceptance criteria, document exceptions and require the right owner to accept residual risk. Fixes are retested before promotion.
Risk-based coverage
Start with what must not go wrong.
We map the application's jobs, users, data and authority before choosing metrics. The severity of an error determines how it is tested and who can accept the residual risk.
Task & outcome contracts
Define acceptable answers, actions and escalation behaviour with domain owners. Identify where multiple answers are valid and where an exact business rule applies.
Failure severity
Separate inconvenient mistakes from disclosure, financial loss or unsafe actions. Establish non-negotiable boundaries and escalation requirements rather than averaging every error together.
Coverage matrix
Map intents, languages, source types, user permissions, tools and edge cases to tests. Record gaps explicitly; a passing suite does not establish coverage outside its scope.
Decision ownership
Name the people responsible for product acceptance, specialist review, security and release approval. Testing provides evidence; it does not replace accountable decisions.
Evaluation data
A representative test set is a product asset.
We build an authorised, versioned set of cases that reflects the real workload, including hard and uncommon cases. Synthetic examples can supplement it but cannot prove performance on their own.
Realistic examples
Use permitted support questions, documents, workflow traces and known incidents. Minimise personal information and agree storage, access and retention before collecting examples.
Labels & rubrics
Define expected outcomes, supporting evidence and review guidance. Use qualified reviewers for specialist tasks, record disagreement and adjudicate unclear examples.
Clean holdouts
Separate development, validation and held-out acceptance data. Deduplicate related cases and avoid tuning repeatedly against the final test set; refresh it as usage changes.
Slices & difficult cases
Include rare failures, ambiguous requests, missing evidence, long conversations and relevant language or user groups. Report subgroup results with sample sizes and uncertainty.
Scoring & evidence
Use the right judge for the question.
No single score captures an AI system's quality. We combine deterministic checks, task metrics and calibrated human or model-assisted review, with each method's limitations documented.
Deterministic assertions
Check schemas, required fields, allowed actions, permission enforcement and known calculations in code. Use exact checks for exact requirements rather than asking another model to guess.
Task-specific metrics
Measure precision and recall, critical-field accuracy, citation support, completion and escalation quality where relevant. Show denominators, error severity and slice-level results.
Human assessment
Use a documented rubric and blind comparisons where practical. Review subjective quality and specialist correctness with people qualified to judge them, recording disagreement.
Model-assisted judging
Calibrate judges against human-labelled examples, check position and verbosity bias and retain human review for consequential disagreements. A judge score is not independent ground truth.
End-to-end system testing
Test the path. Not only the final response.
We inspect the entire application boundary: context, model output, tool requests, side effects and user-visible outcomes. Where relevant, repeat trials reveal variability that a single successful run misses.
Retrieval & grounding
Check whether the right sources were available, retrieved and used correctly. Test stale or contradictory content, missing answers, revoked access and citations that do not support the claim.
Tools & action integrity
Validate arguments, identity scope, approval requirements and idempotency. Inspect intermediate actions and confirm that failures do not become fabricated success messages.
Multi-turn & memory
Test corrections, changed intent, context truncation, session expiry and cross-user isolation. Verify that conversation state does not silently grant privileges or retain inappropriate data.
Failure recovery
Inject unavailable tools, model timeouts, malformed results, duplicate callbacks and partial execution in an agreed test environment. Validate retry limits, compensation and human fallback.
Safety & adversarial testing
Challenge the boundaries deliberately.
We include risk-based adversarial cases within an authorised scope. Targeted testing helps expose weaknesses, but it is not a promise that every attack has been found or a substitute for a specialist security assessment.
Prompt injection
Test malicious instructions in user messages, retrieved documents and tool output. Assess whether they alter authority, bypass intended workflow or trigger data exfiltration.
Privacy & permissions
Probe cross-tenant retrieval, cached answers, session memory and tool access using approved fixtures. Verify that secrets and personal information are not unnecessarily exposed or logged.
Unsafe or unsupported requests
Test refusal, safe alternatives and human escalation for the application's prohibited and high-impact use cases. Include benign requests that should not be incorrectly refused.
Remediation & retesting
Document evidence, affected boundaries and reproduction steps. Retest fixes and keep confirmed failure cases in the regression suite; passing a test is not a compliance certification.
Performance & cost
Quality at the workload you actually have.
A system that passes isolated examples can still fail under production conditions. We assess latency, capacity, failures and cost with representative context lengths, traffic and integrations.
User-visible latency
Measure time to a useful response and completion, including retrieval and tools. Examine tail latency and long-running workflows instead of reporting only an average.
Load & resilience
Test concurrency, queues, rate limits, cold starts and dependency outages in a controlled scope. Identify where traffic should be bounded or routed to a fallback.
Cost per outcome
Include tokens, retries, tool calls, infrastructure and human review. Compare costs at similar quality and workload; a cheaper response may create more downstream work.
Candidate comparison
Run baseline and candidate against the same versioned cases and conditions. Repeat stochastic tasks, report variability and avoid unsupported claims from small samples.
Release gates & regression
A report is useful. A repeatable decision is better.
We help integrate evaluations into the development and release process. Criteria are agreed before inspecting candidate results, with critical boundaries kept separate from aggregate quality scores.
Layered checks
Run fast contract tests on changes, broader scenario suites on release candidates and deeper adversarial or load tests at agreed intervals. Control runtime and provider spend.
Versioned runs
Record application revision, prompts, models, retrieval settings, source snapshot and tool configuration. Preserve enough privacy-safe evidence to reproduce or explain the result.
Release decisions
Block agreed critical failures, compare against the baseline and document unresolved risks. Thresholds depend on the task and consequences; no universal pass percentage is assumed.
Pilot & rollback
Use a bounded pilot, shadow evaluation or staged rollout where suitable. Define stop conditions, accountable approval and a tested rollback path before wider exposure.
Production evaluation
Passing before launch is not the end.
Real users, changing sources and provider updates can expose new failure modes. We connect offline tests with privacy-aware production review and a process for adding new regression cases.
Outcome sampling
Review authorised samples, unresolved tasks, complaints and handoff quality. Avoid assuming user ratings or lack of complaints reliably establish correctness.
Drift & change
Monitor changes in intents, sources, tool behaviour and model revisions. Investigate quality shifts and re-evaluate material changes before increasing exposure.
Privacy-aware traces
Capture the minimum evidence needed, redact sensitive fields and set access and retention rules. Detailed observability must not become an uncontrolled copy of user data.
Operating responsibility
Agree who reviews failures, maintains labels, approves releases and responds to incidents. Evaluation infrastructure needs ongoing ownership as the application evolves.
Technology stack
Tools that support the evidence.
These are options we can assess, not a required bundle. We favour a maintainable harness that fits your application, data-handling requirements and delivery process over adding every evaluation framework.
Application & workflow tests
pytestVitestPlaywrightJSON Schema / Pydantic
Assert exact contracts, integrations, action states and user workflows. Combine conventional software tests with AI-specific evaluations.
LLM & RAG evaluation
promptfooRagasDeepEvalCustom task harnesses
Choose supported metrics and adapters for the task. Validate framework scores against known cases; built-in metrics do not establish correctness automatically.
Agent evaluation
LangSmithLangfuseOpenTelemetryTrajectory assertions
Inspect prompts, retrieval and tool sequences with appropriate redaction. Select hosted or self-managed tooling according to processing and access requirements.
Datasets & review
Label StudioDVCPostgreSQLVersioned JSONL datasets
Maintain authorised examples, reviewer guidance, holdouts and dataset revisions. An existing internal review process may be preferable to new annotation software.
Adversarial assessment
promptfoo red teaminggarakCustom attack fixtures
Run bounded, authorised probes for relevant threats. Tool findings require review and reproduction; automated scanners are not a complete security audit.
Load & operational signals
k6LocustPrometheus / GrafanaOpenTelemetry
Assess capacity, latency, failures and cost on representative workloads. Use test accounts and environments to avoid unintended live actions.
Release automation
GitHub Actions / GitLab CIDockerEvaluation reportsStaged deployments
Attach versioned results and agreed gates to releases, with scoped credentials, bounded spend and an accountable approval path.
Framework capabilities, licences and provider-processing terms are reviewed for the engagement. Hosted tracing or model-assisted scoring may send content to another provider; that path must be authorised and controlled.
Delivery & outputs
From an assessment to a working evaluation practice.
We agree a practical scope with your engineering and domain owners. The objective is not simply to deliver a report, but to leave repeatable evidence and a clearer process for the next change.
- 01
Map risk & baseline
Review the workflow, current tests, incidents, data access and release process. Agree what will be assessed and which boundaries matter most.
Output: A coverage plan, baseline definition and acceptance owners.
- 02
Build the evaluation set
Curate authorised cases, define labels and rubrics, establish holdouts and identify coverage gaps with domain reviewers.
Output: A versioned dataset, scoring guidance and known limitations.
- 03
Implement & run the harness
Connect task metrics, deterministic assertions, trajectory tests and selected adversarial or load cases. Compare the current system with agreed candidates.
Output: Repeatable runs, failure evidence and prioritised findings.
- 04
Remediate & gate releases
Support root-cause analysis, retest fixes and wire appropriate evaluations into delivery. Separate critical blockers from accepted residual risks.
Output: Regression cases, release checks and a reviewed decision report.
- 05
Hand over & maintain
Document how to run tests, review cases, interpret results and respond to production failures. Agree ownership and ongoing evaluation scope.
Output: Code, datasets within agreed rights, runbooks and an operating plan.
A bounded first sprint
Start with a paid two-week sprint and one agreed goal, without a required long-term commitment. Establish a baseline, prioritise a critical workflow and build an agreed first slice. Comprehensive testing and remediation may require further work.
What your team brings
Authorised access to the application, representative examples, known failures and reviewers who can judge outcomes. We also need system constraints, a safe environment for action tests and named owners for acceptance decisions.
Practical tradeoffs
Spend testing effort where failure costs most.
Evaluation has its own costs: dataset preparation, specialist review, model calls, test environments and maintenance. We prioritise consequential boundaries and representative workflows, then broaden coverage with evidence.
Useful reporting, not false precision
Findings include the evaluated scope, dataset size, configuration, failures and limitations. Where appropriate we report variability or confidence intervals, rather than presenting a small or unrepresentative sample as a universal performance claim.
Ownership after the engagement
Your code belongs to you from day one. Dataset access depends on source rights and privacy rules, and third-party tools retain their own terms. We agree who maintains test cases, reviews results and funds ongoing runs.
Related engineering & guides
Connect testing to the system it protects.
Explore agent development and detailed guides for evaluation and evidence-led operations.
Questions before you assess
The practical details.
Can you evaluate an AI system another team built?
Yes, subject to authorised access and an agreed scope. We can assess an existing application, a candidate release or a pilot. Limited access may constrain what we can test; black-box findings cannot establish internal permission or deployment controls.
Is this the same as ordinary QA?
It includes conventional software QA but adds evidence-grounded answer assessment, stochastic behaviour, retrieval quality, agent trajectories and model-specific adversarial cases. Traditional tests remain essential for exact business rules and permissions.
What pass rate should we target?
There is no universal number. Targets depend on the task, dataset, error severity and fallback. Critical boundaries are assessed separately from aggregate quality; sample size and uncertainty must accompany reported scores.
Can an LLM judge another LLM reliably?
It can assist with selected rubric-based assessments, but may share errors or exhibit bias. We calibrate against human-labelled cases, inspect disagreement and do not use judge scores as the sole evidence for consequential correctness.
Will evaluation eliminate hallucinations or certify compliance?
No. Evaluation measures behaviour within a defined scope and exposes failures to address. It cannot prove the absence of hallucinations or future attacks, and this service is not a regulatory certification or legal opinion.
How much test data do we need?
Enough to cover the agreed risks and workload credibly; there is no useful universal minimum. We assess available cases, rare failure coverage and reviewer capacity, report uncertainty and identify where more evidence is needed.
Will you use our production data?
Only with authorisation and agreed handling rules. We minimise or de-identify examples where possible, review processing terms and use controlled fixtures for unsafe actions. Synthetic cases can supplement real coverage but are labelled and reviewed.
What do we receive and how do we start?
Start with a paid two-week sprint, one agreed goal and no required long-term commitment. Your code belongs to you from day one. The scoped outputs may include a harness, dataset, coverage map and findings; a complete audit or production rollout may need further work. Data access and third-party licences are agreed explicitly.
Start with the uncertainty
Bring the system.
Let's test what matters.
Tell us what the application does, what you are worried about and what decision you need to make. We'll define a sensible evaluation scope and the evidence it should produce.