noqtis-data —
make, show and check a capsule's data record
noqtis data init |
[-f | --force]
dir |
noqtis data add |
[-u | --url
url] [--title
text] [--publisher
text] [--licence
text] [--retrieved
time | now]
[--etag text]
[--modified text]
[--archived url]
[--mirror url ...]
[--synthetic] dir file
... |
noqtis data put |
dir member json-file |
- |
noqtis data del |
dir member [path |
name] |
noqtis data show |
[-j | --json]
dir |
noqtis data check |
[-j | --json]
dir |
noqtis data syn |
--seed n
[--rows n]
dir file |
The data record, noqtis.data.json in a
capsule folder
(noqtis-data(5)),
says where each data file came from, its schema and summaries, and the steps
and checks made with it. It is a file of the capsule like any other: seal
the folder
(noqtis-seal(1))
after changing the record, and the seal covers it. Each command that changes
the record of a sealed folder says so.
init- Writes a record that holds only its format.
-f
replaces a record that is there.
add- Records the SHA-256 and size of each file (a path in
dir, ‘
/’
between folders) as a source, keeping what the record already says of it,
so a file fetched again keeps its URL. A folder with no record gets a new
one. The options describe where one file came from:
--url, --title,
--publisher, --licence,
--retrieved (a UTC time
‘YYYY-MM-DDTHH:MM:SSZ’, or
now: the clock, or
SOURCE_DATE_EPOCH when set),
--etag and --modified (the
server's ETag and Last-Modified), --archived (the
URL of an archived copy), --mirror (another URL
with the same bytes, up to 32), and --synthetic
(the file is a generated stand-in: every report line about it says
so).
put- Adds the element in json-file
(
- for standard input) to
member: sources,
schemas, stats,
chunks (replacing the element with the same path),
checks (the same name),
steps, commits (appended),
or sets env or cite. The
element is checked as the format defines it; a refusal names the member
and what is wrong.
del- Removes the element of member with that path or
name, or the whole member.
fmt- Reads a record edited by hand (any member order, keyed arrays in any
order) and writes it in canonical form.
show- Prints the record, one element a line;
-j prints
it as it is.
check- Checks the record: that it is canonical and of the format, that each
source file is there with its recorded SHA-256 and size, that each
schema's file matches it, that each file's summaries
(noqtis-drift(1))
are made again the same from it, that the script or mapping of each step
that names one with its SHA-256 still has it, and that no recorded check
failed: one with
expected and
observed numbers is recomputed against its
tol (0 when not given), one with strings must have
them equal, and one recorded as
‘pass’ that does not hold fails. It
prints one row per check and, on standard error, what to do about each
kind of failure. -j prints one line of JSON:
{"folder":"<dir>","ok":<bool>,"checks":[{"check":"<name>",
"ok":<bool>,"detail":"<text>"[,"fix":"<text>"]},...]}
syn- Writes a synthetic file in dir
from the record's schema of it, the recipe (columns, types, ranges, sets,
missing values, delimiter, header), with
--rows
rows (default the schema's rows.min, else 100),
the same bytes for the same --seed (required, from
0 to 2^64 - 1), and records it as a source marked
synthetic, so every check of it says
‘[synthetic]’. It never writes in
place of a file the record does not mark synthetic.
nqdsyn(3) defines the
values.
noqtis data check does not check the seal,
and noqtis-check(1)
does not read the record: run both.
- 0
- success; for
check, every check passed
- 1
check: a
check failed- 2
- a usage or file error, or a record that is not valid
$ noqtis data add --url https://example.org/a.csv \
--retrieved now results data/a.csv
$ noqtis data check results
check detail
ok record noqtis-data-1, canonical: 1 sources, 0 schemas
ok source data/a.csv: SHA-256 and size match
results: OK (2 checks passed)
$ noqtis seal -k key results
