Evaluation
In developmentEvaluation suites and graders exist for some agents and are run by hand; automatic scorecards and release gates are still being built.
Lamiak Core · Evaluation and learning
Measure first, then decide what changes.
Evaluation suites and graders run by hand today. Scorecards, promotion gates, dataset snapshots and training from outcomes are designed and not built.
The question
How do you know a change is an improvement?
Today, by running an evaluation suite by hand and reading the result. Graders exist for some agents. There is no standing scorecard, and no evaluation suite runs in our continuous integration, so a change is judged by a person looking at the output rather than by a gate. The machinery that would turn that judgement into a rule is designed and not built.
How it works today
In developmentWhat is built, stated as built.
Suites and graders exist for some agents and are started by a person. Feedback is captured in a few places, not yet as one stream.
Suites run by hand
An evaluation suite is started by a person and its result read by a person. Nothing holds a release on it.
Graders for some agents
Where a grader exists, it scores an output against what the suite expects. Where one does not, the reading is a person's.
Feedback where it is given
Edits, rejections and whether a suggestion was useful are recorded in a few places today, rather than gathered into one stream.
What is planned
PlannedWhat is designed, stated as designed.
The rest is on the registry as planned. Writing it here is not a claim that it runs.
Scorecards
A standing view of how each agent scores, rather than a suite somebody has to remember to start.
Promotion gates
A candidate would have to do better on evaluations before it is given authority in production.
Dataset snapshots
Versioned snapshots of the feedback and outcome data, so an evaluation can be repeated against the same input.
Training from outcomes
Adapting models on outcome data a customer has agreed to share, and recording which snapshot they came from.
A design, not a description
Three speeds, written as a design.
None of the three runs today: this is a design, written down so it can be argued with before it is built.
Fast
At run time
Context and instructions would adjust only within the permissions the task already holds. Learning would change how a system behaves, not what it is allowed to reach.
Medium
Between runs
Routing and workflow changes would be compared on evaluations before they are adopted, against a snapshot rather than against memory.
Slow
Over time
Models adapted from outcome data would be promoted only through a gate, and a candidate that does not do better would stay a candidate.
What this leans on
The registry entries behind this page.
Each card carries its own status, taken from the registry rather than written here. How the labels are used is set out on the trust page.
Feedback capture
In developmentRecord how people respond to AI suggestions, including edits and rejections, and whether they found them useful; captured in a few places today, not yet as one feedback stream.
Dataset snapshots
PlannedVersioned snapshots of the feedback and outcome data used for training and evaluation.
Model registry
PlannedA registry of models, adapters and prompts with their evaluation history.
Model promotion gates
PlannedA candidate must prove better on evaluations before it receives authority in production.
Training from outcomes
PlannedAdapt models on outcome data a customer has agreed to share.
The rest of Lamiak Core
The parts this one sits beside.
- ContextWhat the system knows before it acts.Assembling the records, documents and permissions a task needs into one packet, before an agent is allowed to do anything with them.
- AgentsAgents that propose, inside limits you set.Bounded agents that draft and propose rather than act, with each step sent to a model or to plain code depending on what the step is.
- Workflows and approvalsSteps that run the same way, and the person who decides.Ordinary repeatable steps, with a person approving where the policy for that workflow says so, and each decision recorded with who made it and when.
- Deployment and governanceWhere your records sit, and what governs a change.How records are kept apart per organisation, which outside services receive data and why, and what would have to be true before something changes.
Start with one workflow that matters.
We map the work, the systems it touches and the approvals it needs before anything is automated.