Skip to content
NOQTIS

nqdstats(3)

nqdstats, nqddrift, nqddcal — summarise a CSV file, compare a later one with the summary, and calibrate it

libnqd, -lnqd -lnoqtis

#include <nqd.h>

typedef struct {
  double psiwarn; /* 0.1 */
  double psifail; /* 0.25 */
  double alpha;   /* 0.001 */
  double minn;    /* 1000 */
  int benford;    /* 0 */
} nqddopt;

int
nqdstats(const char *dir, const char *path, const nqdcsvopt *o, char *out, size_t cap, size_t *outlen, char *why, size_t wcap);

int
nqddrift(const char *stats, size_t n, const char *file, const nqdcsvopt *o, const nqddopt *d, nqdrep **rep);

int
nqddcal(const char *stats, size_t n, const char *const *files, size_t nf, const nqdcsvopt *o, char *out, size_t cap, size_t *outlen, char *why, size_t wcap);

() reads the CSV file path in dir and writes its summaries to out as a ‘stats’ element in canonical JSON, for nqdput(3); o is as for nqdinfer(3) (its ranges unused), and may be NULL.

() compares the CSV file at file (any path) with the ‘stats’ element stats (n bytes): one ‘drift’ row for the row counts, and per column a ‘drift’ row (passed or failed) or a ‘drift.warn’ row (passed, with a warning). Each column's row carries its numbers as JSON, read with nqdrepj(3). d sets the levels (NULL: the defaults above). noqtis-drift(1) defines the summaries, the tests, the levels and the error of the sketch.

() calibrates the warnings of the ‘stats’ element stats (n bytes) on the nf CSV files files, known not to have drifted from it: each is compared with it as nqddrift() does with the default levels, and every column gets a cal object: the number of files (n), the largest PSI (psi), the smallest KS or chi-square p-value (p) and, for a numeric column, the smallest Benford p-value (bp). The element with them is written to out as nqdstats() writes one. nqddrift() then warns of a calibrated column only for a PSI above psi or a p-value below p (bp for Benford), besides the levels of d; when a column fails does not change.

All three hold every number of a column in memory while reading a file, 8 bytes each.

nqdstats() returns NQOK, NQEBUF with *outlen set when cap is too small, NQEDEC for a file that is not CSV or has rows of different lengths, NQEIO for a file that cannot be read, NQELIM over the reader's limits. nqddrift() returns NQOK when no row failed and NQEVRFY when one did (an element that is not ‘stats’, or a file that cannot be read, is a failed row), with *rep set. nqddcal() returns NQOK, NQEBUF with *outlen set when cap is too small, NQEDEC with a sentence in why for an element that is not ‘stats’ or a file that lacks one of its columns, cannot be read or has values of another kind, NQEARG for no files.

noqtis-drift(1), nqdchk(3), nqdinfer(3), noqtis-data(5)