DataCourt AI
Put your dataset on trial before your model pays the price.
Find the data hurting your model, understand why, review what matters first, and prove whether the fix actually helps.
Image classification datasets as ZIP folders. Private by default. No model training on your data outside your workspace.
train/circle/img_0412.png is labelled “circle”.
Out-of-fold baseline predicts “triangle” (p = 0.91); 9 of 10 nearest neighbours are triangles.
Image quality is normal; the sample is not isolated, so a rare-but-valid explanation is weak.
Models inherit every flaw in their data, and most tools stop at a list of suspicious images.
Flagging is the easy part. What teams actually need is to know which findings are real, which matter for the model, which to look at first with limited review time, and whether fixing them improves anything measurable.
A resized copy of a training image in the test split makes the model look better than it is.
A few percent of wrong labels move decision boundaries and hide real errors.
If every “star” has a blue border, the model can learn the border and still score well.
Unusual but valid examples look like outliers. Deleting them makes the model more brittle.
Every number says where it came from.
DataCourt labels each output with its type. Scores and verdicts come from deterministic, versioned algorithms. An optional language model can phrase an explanation, but it never produces a score or a verdict, and numbers it adds that are not in the evidence get flagged.
Audits record the exact algorithm versions and configuration they ran with, so results can be reproduced and compared across releases.
From suspicion to a proven, versioned fix.
- 01Find
Quality defects, duplicate families, split leakage, suspicious labels, shortcuts, coverage gaps and privacy flags.
- 02Explain
Every finding carries its evidence: which witness said what, how reliable that witness is, and the thresholds used.
- 03Prioritise
A deterministic jury turns evidence into cases, and the Review Budget orders them by expected value per minute.
- 04Review
Humans decide with keyboard-first review, multi-reviewer consensus and expert adjudication.
- 05Test
The What-if Lab retrains the baseline on the changed dataset and measures a preserved holdout set.
- 06Prove
Hash-chained evidence ledger, versioned algorithms, Dataset Debt, preflight gate and an audit report.
- 07Export
A new dataset version built from final human decisions, with manifest, action log and checksums.
Findings become cases with a prosecution, a defence and a rule-based jury.
A case shows the evidence for the problem and the evidence against it, including the rare-but-valid hypothesis. A deterministic jury weighs witnesses by how reliable they proved to be on this dataset and records which rule fired. Nothing is deleted or relabelled automatically.
Spend review time where it changes the most.
Given a budget in minutes or items, the optimiser picks cases by expected impact per minute, with diminishing returns so one duplicate family or one class cannot absorb the whole budget. You can optimise for label errors, leakage or a balance.
Measured on DataCourt’s synthetic benchmark with injected, known errors. Real-world gains depend on the dataset. See evaluation.
Trace model failures back to the training data that caused them.
For each evaluation error, DataCourt estimates which training samples pushed the model toward the wrong answer, and connects them to open cases. Influence is an approximation on a linear head, labelled as a model estimate. It is not a causal proof.
Test a cleanup before you commit to it.
Apply proposed changes to a copy, retrain the baseline and compare metrics on a holdout set that no action touched. DataCourt says plainly when an improvement is within training noise instead of claiming a win.
A running balance of what is wrong with your data.
Debt has published formulas and thresholds. It goes down as reviewers resolve cases, is snapshotted per version, and feeds a preflight gate (ready, ready with warnings, or blocked) and data contracts that CI can query.
Know what to collect next, not just what to delete.
A coverage map groups the embedding space into regions and compares classes and conditions such as lighting, resolution and background. Under-covered combinations become concrete collection requests.
Every export is a new version, compared against the last one.
Clean exports are registered and audited automatically, so you can see what a cleanup actually changed, whether debt fell, and whether the data drifted in ways nobody intended.
Your data stays in your workspace.
Originals are immutable and every change is traceable in a hash-chained ledger. Reports are technical audit aids, not legal or compliance certifications. Read the security model.
Measured on a benchmark with known answers.
A seeded generator plants duplicates, cross-split leakage, label errors, rare valid examples, a colour shortcut, quality defects and coverage gaps, then scores DataCourt against the ground truth. The numbers come from the built-in CPU descriptor backend. They show the method works on controlled data, not how it will perform on any given real dataset.
Open, inspectable, reproducible.
The source, algorithms, benchmark generator and evaluation scripts are on GitHub. Run the benchmark yourself and check every number on this page.