Skip to content
NOQTIS

noqtis-data(5)

noqtis-data — the noqtis data record, noqtis-data-1

A data record is the file noqtis.data.json at the top of a capsule folder (noqtis-capsule(5)). It says where each data file of the capsule came from, what shape it has, how it was summarised, and what was done and checked with it. The record is an ordinary file of the capsule: the manifest lists it with its digests and the seal covers it, so noqtis-check(1) fails when it changes, and noqtis-data(1) (noqtis data check) checks the files against what it says.

The key words MUST, MUST NOT, SHOULD and MAY are to be read as in RFC 2119 and RFC 8174.

The record MUST be JSON (RFC 8259) in the canonical form of RFC 8785: UTF-8, no whitespace outside strings, object members sorted by the UTF-16 code units of their names, the minimal string escapes, and numbers as IEEE 754 doubles in the shortest form that round-trips, as ECMAScript writes them. A reader MUST reject a record that is not byte-identical to its own canonical form, has a duplicate member name, a string with U+0000 or not valid UTF-8, a number that is not finite, objects nested deeper than 32 levels, or more than 64 MiB. The same record is therefore the same bytes whichever program wrote it.

string
at most 65,536 bytes.
path
a string naming a file of the capsule as the manifest does: relative, ‘/’ between folders, no empty, ‘.’ or ‘..’ segment, no backslash, colon or control character, at most 4,096 bytes, and not noqtis.json, noqtis.sig, noqtis.ers or noqtis.data.json.
hex
64 lower-case hex digits.
count
a whole number from 0 to 2^53.
time
‘YYYY-MM-DDTHH:MM:SSZ’, a valid UTC time.
date
‘YYYY-MM-DD’, a valid date of the proleptic Gregorian calendar.

An object with format, the string ‘noqtis-data-1’, and any of the members below; no other member. A reader MUST reject a member it does not know, so that a record of a later format is refused rather than half-read. Arrays keyed by path (sources, schemas, stats, chunks) or name (checks) MUST be sorted by the bytes of the key, with no key twice; steps and commits keep the order they were written in. An array of objects MUST NOT be empty: a member with nothing in it is left out, so that each record has one form.

Where each file came from. Each element: path (path, required), sha256 (hex, required: the file's SHA-256), size (count, required: its size in bytes), and optionally url (where it was fetched), resolver (an object: kind, one of ‘direct’, ‘ckan’, ‘socrata’, ‘arcgis’, ‘wayback’, required; and the strings base (the portal), id (the dataset), resource, format, and the count layer), retrieved (time), etag, modified (the server's ETag and Last-Modified, as sent), licence, publisher, title (strings), archived (an object: the archived copy's url, required, and its time), mirrors (an array of URLs with the same bytes), synthetic (true when the file is a generated stand-in for the real data; every report line about it says so).
The shape of a CSV file. Each element: path (required), kind (required: ‘csv’), delim (one byte, not a quote or line break; default ‘,’), header (boolean; default true), na (an array of strings that mean a missing value; default ‘’ and ‘NA’), rows (an object with counts min and max), columns and rules.

columns lists the columns in file order, each with name (required, no name twice), type (required: ‘int’, ‘real’, ‘text’, ‘date’ or ‘bool’), nulls (boolean: whether a missing value is allowed; default true), min and max (numbers for ‘int’ and ‘real’, dates for ‘date’), and set (for ‘text’: the values allowed).

rules are checks over a column or the table, each with kind and its arguments: ‘unique’ (col or cols: no value, or combination, twice), ‘notnull’ (col), ‘range’ (col, min and/or max), ‘set’ (col, set), ‘incr’ (col: each value at least the one before), ‘complete’ (col, share: at least that share not missing), ‘distinct’ (col, min and/or max distinct values), ‘regex’ (col, pattern: each value matches the pattern), ‘mad’ (col, max: no value further from the median than max times the scaled MAD), ‘rows’ and ‘cols’ (min and/or max).

A ‘mad’ rule's scaled MAD is 1.4826 times the median of the absolute deviations from the median (R's ()), and max is above 0; a value that is not a number fails, and when the scaled MAD is 0 every value other than the median fails. A ‘regex’ rule's pattern (at most 1,024 bytes) passes a value when it matches some part of it; anchor it with ‘^’ and ‘$’ to match the whole value. Its syntax is this subset, over Unicode code points (a byte that is not UTF-8 is one character, matched only by ‘.’ and a negated class), and a reader MUST refuse anything else: a character stands for itself; ‘.’ is any character, a line break too; ‘^’ and ‘$’ match at the start and the end of the value; ‘[...]’ is a class of characters and ranges ‘a-z’, ‘[^...]’ its complement, ‘-’ first or last is itself, and ‘\\’, ‘\]’, ‘\-’, ‘\[’ and ‘\^’ are those characters in a class; ‘\d’, ‘\w’ and ‘\s’ are the ASCII digits, word characters (digits, letters, ‘_’) and white space (tab, line feed, vertical tab, form feed, carriage return, space), in a class too; a backslash before one of ‘\.[](){}|*+?^$-/’ makes it itself; ‘*’, ‘+’ and ‘?’ repeat what is before them zero or more times, once or more and at most once; ‘|’ separates alternatives and ‘(...)’ groups, at most 64 deep. ‘{’, ‘}’ and an unescaped ‘]’ are refused, as are an empty pattern, group, alternative or class.

Summaries kept to detect drift later without keeping the data (noqtis-drift(1)). Each element: path (required), rows (count, required), delim and header (how the file was read, as for a schema, when not the default) and columns, each with name and n (required: the values not missing), nulls, and for a numeric column q (the quantile sketch: [value, count] pairs, value increasing, count the number of values at most it), edges and counts (the binning of the population stability index: increasing interior edges, bins closed on the right, one count more than edges), digits (nine counts of first significant digits 1 to 9); for any column cal (its calibration on files known not to have drifted: an object with n, the number of files, at least 1, psi, the largest PSI seen, at least 0, p, the smallest KS or chi-square p-value, and for a numeric column bp, the smallest Benford p-value, both from 0 to 1; noqtis-drift(1)); for a categorical column cats ([value, count] pairs sorted by value) and other (the count of values beyond those kept). noqtis-drift(1) defines how they are made and the error of each test made from them.
Digests of the runs of one large file, so that a piece of it can be checked without the rest (noqtis-chunk(1)). Each element, every member required: path; size (the file's bytes); chunk (the bytes in a run, 1024 to 1073741824); digests (the SHA-256 of each run in order, 64 lower-case hex digits each, the last run short: exactly ‘ceil(size / chunk)’ of them, none for an empty file).
The environment of the run: strings os, arch, kernel, cpu, locale, tz, noqtis, build; runtime (an object: language required, version, packages, an array of {name, version, source} sorted by name); locks (lock files: an array of {path, sha256} sorted by path); rng (an object: kind required, and seed, a number or string).
What was done to the data, in order: each with kind (required: ‘recode’, ‘filter’, ‘derive’, ‘script’ or ‘other’), and the strings from, to, note, script (a path) and sha256 (of a mapping or script; with script, the SHA-256 of that file, which noqtis data check checks).
Results of checks made elsewhere (a published total, a falsification test): each with name (required), status (required: ‘pass’, ‘warn’ or ‘fail’), expected and observed (a number or string), tol (a number, at least 0) and note. A reader can recompute the result: with expected and observed numbers it holds when ‘|observed - expected| <= tol’ (tol 0 when not given), with strings when they are equal. A check recorded as ‘fail’ fails its row in noqtis data check.
Sealed commitments (a randomisation list, a pre-registration): each with kind (required: ‘randomisation’, ‘preregistration’ or ‘other’), created (time, required), alg (required: ‘sha3-256’), digest (hex, required), after (time: not to be revealed before) and label.
How to cite the data: an object with title (required), authors (an array of strings), version, doi, url, licence and the count year.

{"format":"noqtis-data-1","sources":[{"licence":"OGL-Canada-2.0",
"path":"data/a.csv","publisher":"City of Example",
"retrieved":"2026-01-01T00:00:00Z","sha256":"7a1668480cde19db0b76c7a36e6
fee9a82afd9937f5d03cec3dae1705ce173e2","size":89,
"url":"https://example.org/a.csv"}]}

(one line in the file; broken here to fit).

noqtis-data(1), nqdchk(3), nqdnew(3), noqtis-capsule(5)

The JavaScript Object Notation (JSON) Data Interchange Format, RFC 8259. JSON Canonicalization Scheme (JCS), RFC 8785. Common Format and MIME Type for Comma-Separated Values (CSV) Files, RFC 4180.