Verification
What we check, and what stays declared.
You cannot judge a dataset responsibly until its origin, rights, consent position, structure, quality evidence and known limitations are clear.
Suitability is assessed against the intended use.
Six dimensions
Each is reviewed separately, and each produces two answers: what was established, and what was not.
A technically strong dataset can still be unsuitable if its provenance, licensing authority, consent or permitted use cannot support the buyer's intended use.
-
01
Provenance
Where the data came from, through every hand it passed.
- What we review
- Collection route, originating party, date range, and whether the chain can be evidenced rather than asserted.
- What stays unverified
- Who collected it and under what arrangement, where the supplier cannot produce records.
-
02
Rights
Who holds them and what they permit.
- What we review
- Ownership or licensing authority, permitted use, territory, duration, exclusivity, prior licensing, and whether rights extend to commercial AI training and to the licence the buyer needs.
- What stays unverified
- Whether an upstream agreement a supplier describes but cannot show actually says what they say it says.
-
03
Consent
What the people recorded were told and agreed to.
- What we review
- The consent instrument and version, the scopes it grants, whether it covers the intended use, and whether withdrawal and deletion can be traced.
- What stays unverified
- Whether a contributor understood the instrument, and anything a supplier asserts without producing the record.
-
04
Metadata
What is labelled and how consistently.
- What we review
- Field coverage, label consistency across the set, whether labels mean the same thing throughout, and whether metadata travels with the data through delivery.
- What stays unverified
- Accuracy of demographic self-declaration, and any field recorded only as free text.
-
05
Quality control
How defects are found before delivery.
- What we review
- Technical integrity, format and checksum verification, language confirmation, speaker structure, transcript-to-audio review, and what happens to material that fails.
- What stays unverified
- Anything outside the reviewed sample. A review of six files is evidence about six files.
-
06
Intended-use fit
Whether it suits your specific workflow.
- What we review
- Suitability assessed separately for training, evaluation, benchmarking, ASR, TTS and voice agents. Never one score.
- What stays unverified
- Fit against a specification we have not been given.
Speaker structure, reviewed as its own axis
“Conversational speech” is not one thing. A set that suits a single-speaker recogniser can be the wrong shape for a two-party voice agent, and the difference is structural rather than acoustic. Where the requirement turns on it, these are established separately and reported separately.
- Number of speakers per item, established from the material rather than from the label.
- Speaker attribution: whether turns are attributed, and by what method.
- Turn structure: how turns are segmented and timed.
- Overlap: whether simultaneous speech occurs, and whether it is marked.
- Channel topology: mixed single channel, or separated per speaker.
- Diarisation status: present, absent, or asserted but not evidenced.
- Transcript attribution: whether the transcript carries the speaker labels the audio structure implies.
Evidence classes
We distinguish between what the supplier declared, what documents NaraDatum reviewed, what NaraDatum independently tested, and what still remains unverified. The classes below split those states further.
Every finding carries the class of evidence it rests on. A dataset does not get one state; each dimension gets the state its evidence actually supports, and they are reported separately.
Buyer-ready means the reviewed evidence supports the stated intended use for the dataset that was reviewed. It is not a warranty of fitness, not a guarantee of legal compliance, and not a statement that the dataset suits every model or every use.
If the evidence does not support the intended use, we don’t treat the dataset as buyer-ready.
Review scope must match the claim.
- Supplier declared
- Stated by the holder. No evidence reviewed. Recorded as a declaration and treated as one.
- Document reviewed
- A contract, consent instrument or record was produced and read. What it says is reported; what it does not cover is named.
- Sample tested
- Established by inspecting delivered material. The finding is evidence about the sample, and the sample size is stated with it.
- Independently established
- Confirmed against a source that does not depend on the holder’s account, such as re-measuring files or re-hashing against a manifest. Where NaraDatum is itself the holder, re-verification is reported as NaraDatum’s own, not as independent.
- Conditional or missing evidence
- Review opened, evidence outstanding or incomplete. Named as outstanding rather than rounded up or left out.
- Not reviewed
- Outside the scope that was reviewed. Reported as not reviewed, not as satisfactory.
Markers are a reading aid only. The evidence class is always stated in words next to the marker; no colour carries meaning on its own, and no marker is a certification.
Review outcomes
The five outcomes below roll the evidence classes up into one outcome per dataset. Only the first two, Reviewed, full scope and Reviewed with limitations, can be presented to a buyer as reviewed supply, and any limitations travel with the offer. Those two outcomes are the ones we treat as buyer-ready.
- Reviewed, full scope
- Reviewed across all six dimensions with evidence produced, and the review scope stated with the dataset. Can be presented as reviewed supply.
- Reviewed with limitations
- Reviewed with named gaps. The limitations are disclosed with the offer, not on request. Can be presented as reviewed supply where the evidence supports the intended use.
- Conditional
- Review incomplete or evidence outstanding. Not offered while it stands.
- Not reviewed
- No review has taken place. Declared information only.
- Rejected
- Failed review. Not offered, and not offered later without a new review.
How supplier review works
We review supplier identity and authority first. Each dataset is then assessed separately across provenance, rights, consent, metadata, quality control and intended-use fit.
Supplier approval does not make every dataset approved. Each dataset has to clear its own NaraDatum review before we treat it as buyer-ready.
NaraDatum sources only from organisations that represent that they hold, or can lawfully license, human data, and that representation is reviewed before any dataset is offered. An approved supplier is an organisation we are willing to review datasets from.
Applying is not verification. Everything an applicant writes is a declaration. Until evidence has been reviewed, it stays a declaration and is treated as one internally.
Consent, where NaraDatum collects
For NaraDatum-controlled collection, consent records are linked to the relevant contributor and data, with permitted-use scope retained as part of the project record.
- Consent is task-specific, not a blanket platform agreement.
- The consent position and permitted-use scope are reviewed against the project requirement.
- Withdrawal and deletion requests are tracked, and the account of what could and could not be deleted is written.
- Sensitive actions are recorded in an append-only audit log.
See what this produces
A Dataset Passport records what was reviewed, what was established, and what was not. One illustrative specimen is published in full.
Common questions
- How does NaraDatum verify a dataset?
- Six dimensions are reviewed separately: provenance, rights, consent, metadata, quality control and intended-use fit. Each produces two answers: what was established, and what was not. NaraDatum does not issue a single quality score, because suitability differs by use case.
- What does NaraDatum leave unverified?
- Anything a supplier asserts but cannot evidence stays labelled as declared. That includes who collected the data where records cannot be produced, whether an upstream agreement says what a supplier says it says, the accuracy of demographic self-declaration, and anything outside the reviewed sample. A review of six files is evidence about six files.
- Does an approved supplier mean an approved dataset?
- No. Applying is not verification. An approved supplier is an organisation NaraDatum is willing to review datasets from, and approval of the organisation does not approve any dataset it holds. Each dataset is reviewed on its own provenance, rights, consent, metadata, quality control and intended-use fit before it can be used.
- When can NaraDatum describe something as reviewed supply?
- Only once the dataset itself has cleared review against the six dimensions, including any disclosed limitations, with the evidence class recorded for each fact. A supplier clearing organisation review does not make anything it holds reviewed supply. NaraDatum does not publish an open catalogue. Availability is confirmed against each requirement after rights, provenance, quality and commercial availability are reviewed.