Skip to content
NOQTIS

noqtis-drift(1)

noqtis-drift — summarise CSV files, and say later whether they drifted

noqtis drift pin [-d | --delim c] [--no-header] dir file ...

noqtis drift check [-j | --json] [--psi-warn x] [--psi-fail x] [--alpha p] [--min-n n] [--benford] [-d | --delim c] [--no-header] dir [file[=current-file] ...]

noqtis drift cal [-d | --delim c] [--no-header] dir file=unchanged-file ...

pin reads each CSV file in dir and records its summaries in the data record (stats, noqtis-data(5)), of a fixed size whatever the file's, so that a later file can be compared with it without the old data being kept. check compares each file (all summarised files, or those named; with file=current-file, that file instead of the one in dir, such as a new download of the same source) with its summary, one row per column. cal calibrates the warnings of the summary of each file on the files given after it, which are known not to have drifted from it (a re-fetch of unchanged data, the other half of a random split of the same data); a file may be named more than once. Each is compared as check would; every column then records, as cal, the number of files and the largest PSI, the smallest KS or chi-square p-value and the smallest Benford p-value seen on them. A later check warns of that column only for a PSI above that largest one or a p-value below that smallest one, besides the thresholds below; when a column fails does not change. A file that lacks a column of the summary, or has text in a numeric column, cannot calibrate it and is refused. Pinning the file again drops its calibration.

Values that are empty or ‘NA’ are missing; a column is numeric when every other value is a JSON number, categorical otherwise. Of n values of a numeric column, sorted x(1) <= ... <= x(n):

, the quantile sketch
pairs (v, c): the value x(r) and c, the number of values at most it, for the ranks r = 1 and r = ceil(j n / 1000), j = 1 ... 1000, a value kept once. When n <= 1,001, every distinct value is kept and the sketch is the exact distribution.
, counts
the bins of the population stability index: with at most ten distinct values, each value is an edge; else the type-1 deciles x(ceil(i n /10)), i = 1 ... 9, equal ones merged. Bins are closed on the right, the last open above; counts are exact.
the counts of first significant digits 1 to 9, read from each value's text (so 0.0032 counts as 3).

Of a categorical column, cats counts its 1,000 most frequent values (ties broken by their bytes) and other the rest.

Of a numeric column, with the reference of n values and the current file of m:

Kolmogorov-Smirnov
D = sup |F(x) - G(x)|, with G the current file's distribution function and F the sketch's step function F(x) = c/n for the largest kept v <= x. From the sketch, D is within e = max (c(j+1) - c(j) - 1) / n of the exact statistic, at most about 0.001 (the row and the JSON say e, 0 when the sketch is exact). The p-value is exact, by the recursion R used for () without ties, when the sketch is exact, n m < 10,000 and no value is tied; otherwise it is the Kolmogorov limit P(K > sqrt(nm/(n+m)) D), as R computes it (tolerance 1e-6, so p-values below about 1e-6 are not precise). With ties R 4.6 has an exact p-value too; nqd then gives the asymptotic one.
Population stability index
PSI = Σ (p - q) ln(p/q) over the bins, with p and q the shares of the reference and the current file and an empty bin counted as 1e-6; the Jensen-Shannon divergence ½ Σ p ln(p/m) + ½ Σ q ln(q/m), m = (p + q)/2, in nats, at most ln 2.
Benford
X^2 of the nine first-digit counts against log10(1 + 1/d), on 8 degrees of freedom.

Of a categorical column, PSI and Jensen-Shannon over the kept categories and other (where every value the reference did not keep falls), and the chi-square test of homogeneity of the 2 × k table of reference and current counts (no continuity correction; columns empty in both left out), on k - 1 degrees of freedom.

A column fails when its PSI is at least --psi-fail (default 0.25) and both samples have at least --min-n values (default 1,000: on fewer, binning noise alone can pass 0.25); it warns when its PSI is at least --psi-warn (default 0.1) or at least --psi-fail on small samples, when a KS or chi-square p-value is below --alpha (default 0.001), and with --benford when a Benford p-value is below it. A numeric column with values that are not numbers fails; a column of the reference that is gone fails; a new column warns. The PSI bands come from credit-risk monitoring and are heuristics; calibrate them on data known not to have drifted (cal).

tests/nqd/kserr.c measures the KS statistic from the sketch against the exact one for normal samples of n and n/2 values (five runs each): the difference never passed the stated e; under no shift it reached it (0.0005 at n = 2,000, 0.00096 at 1,000,000), and with a shift of 0.05 or more it stayed under 0.00012 from n = 100,000 on. Since the exact D under no drift is about 1/sqrt(n) and so near 0.001 for a million values, read the KS p-value of very large files with that error in mind, or rely on the PSI.

, --json
One line of JSON: the folder, the level (‘ok, ‘warn, ‘fail’’’) and the rows, each with its numbers under stats: nref, ncur, nullsref, nullscur, kind, and ksd, ksp, kserr, ksexact, psi, js, benfordx2, benfordp, benfordpref (numeric) or psi, js, chisq, df, chisqp, unseen (categorical), calpsi, calp and calbp when calibrated, and level.

0
nothing drifted
1
a column failed
3
a column warned, none failed
2
a usage or file error, or no summary to compare with

cal exits 0 when the record is written and 2 otherwise.

$ noqtis drift pin results data/a.csv
$ curl -o new.csv https://example.org/a.csv
$ noqtis drift check results data/a.csv=new.csv
$ noqtis drift cal results data/a.csv=fetch1.csv data/a.csv=fetch2.csv
data/a.csv: calibrated on 2 files known not to have drifted

noqtis-data(1), noqtis-schema(1), nqdstats(3), noqtis-data(5)

F. J. Massey, The Kolmogorov-Smirnov test for goodness of fit, Journal of the American Statistical Association, 46, 1951. J. Lin, Divergence measures based on the Shannon entropy, IEEE Transactions on Information Theory, 37, 1991.