Skip to content
NOQTIS

noqtis-schema(1)

noqtis-schema — pin and check the schema of CSV files

noqtis schema infer [-d | --delim c] [--no-header] [--ranges] [--put] dir file ...

noqtis schema check [-j | --json] dir [file ...]

infer reads each CSV file in dir (RFC 4180: a delimiter byte, LF or CRLF line ends, quoted fields with ‘""’ for a quote and line breaks kept inside, a UTF-8 byte order mark skipped) and prints its schema, a schemas element of the data record (noqtis-data(5)), as one line of JSON: the columns in order with their names, the narrowest type every value fits (‘bool’ (true, false, TRUE, FALSE, True, False), ‘int’ (an optional minus and at most 18 digits, no leading zero, so codes such as ‘007’ stay text), ‘real’ (a JSON number), ‘date’ (‘YYYY-MM-DD’), else ‘text’), whether each has missing values (the empty field and ‘NA’), and the number of rows, pinned as its minimum and maximum.

, --delim c
the delimiter, one byte (default ‘,’; tab for a tab)
the first line is data; the columns are named V1, V2, ...
pin each number and date column's smallest and largest value too
record the schemas in the data record instead of printing them, replacing any for the same file

check checks the files against the schemas the record pins (every schema, or those of the files named): the header against the column names, the number of fields of each line, each value against its column's type, missing-value codes, nulls, min, max and set, the row count, and each rule (‘unique’, ‘notnull’, ‘range’, ‘set’, ‘incr’, ‘complete’, ‘distinct’, ‘regex’, ‘mad’, ‘rows’, ‘cols’). A file that is not CSV fails its row with the line where it stops being CSV. It shows the first ten problems of a file and how many there are in all, one row per rule, and on standard error what to do. noqtis data check runs the same checks with the rest of the record. Edit a schema, or add rules, with noqtis data put dir schemas file.

Limits: a field of at most 1 MiB, a line of at most 16 MiB and 16,384 fields; a ‘unique’ or ‘distinct’ rule holds at most 2^25 distinct values, a ‘mad’ rule at most 2^23 numbers, a ‘regex’ pattern at most 1,024 bytes.

0
success; for check, every check passed
1
: a check failed
2
a usage or file error, a file that is not CSV (infer), or a folder with no schema to check

$ noqtis schema infer --put results data/a.csv
$ noqtis schema check results
      check   detail
ok    schema  data/a.csv: 3 rows, 4 columns as the schema says
results: OK (1 checks passed)

noqtis(1), noqtis-data(1), nqdinfer(3), noqtis-data(5)