funkpower
ce8332d6d8
A book that arrives as a file often has no identifier to ask about, only a title and perhaps a name. `bibfetch search --title T [--author A] [--limit N]` asks Open Library's index which books it has under those, and Google Books only when Open Library has nothing, and prints one record per candidate in the order the register gave them. From Rust, `bibfetch::search(&agent, title, author, limit, attempts)`. Each candidate is one edition. The index answers with works and, asked for `editions.*`, names the edition of each that best matches; the ISBN, publisher, language, year and cover come from that edition, the title, authors, subjects and length from the work. The work's own ISBN list mixes editions: the first War and Peace work lists four, while the edition matched is a 1917 printing with none. `id` is the first ISBN whose check digit agrees, in thirteen digits, thirteen-digit entries taken first. A candidate with none is named by the register's key: `olid` (OL…M for the edition, OL…W for the work when no edition was named) or `googlebooks` (a volume id). Google's `OTHER` shelfmarks are never read as ISBNs. `cover_id` keeps the meaning it has in a lookup record: the edition's own first cover, taken from `editions.cover_i` and only when the record names that edition. The work's `cover_i` and `cover_edition_key` are not asked for. A work with no edition named, and every Google Books candidate, has none. Open Library having nothing stands when Google Books is down; both down is exit 69. Exit 0 a candidate, 1 none, 65 an empty title. The retrying GET that lookups use is now `ask::get`, shared by both, and no longer sleeps after its last attempt.
4.3 KiB
What these programs promise
Eight things hold for every program here. Together they are what lets a person put any of them in a pipeline without first reading how it works.
Part of catena.
Each one does a single job
No program here has a mode switch that changes what it is. Where two jobs
would otherwise share a name, there are two programs — which is why
cuenorm, cuepack and cueseg are
separate commands rather than one command with a flag.
Input on stdin, results on stdout, diagnostics on stderr
Output and commentary never share a stream, so a result can always be piped somewhere without filtering anything out of it.
Almost all of them read stdin or a path. Where one takes a path only, it says
so in --help and says why: pdftriage reads two specific
page ranges rather than the whole file, which a stream cannot offer.
Anything list-shaped is NDJSON
One record per line, flushed as it is produced — so a slow stage downstream sees each record as it appears rather than when the producer finishes.
Readers are strict, and each strictness prevents a failure that would otherwise be silent. A record from a version the reader does not know is refused rather than parsed into whatever lines up. A bad line ends the stream instead of being skipped, because it means the two ends disagree about the format. Lines are bounded, so a program pointed at a binary file fails instead of allocating it. A byte-order mark and blank lines are ignored, so a stream that was concatenated or written on Windows still reads.
Exit codes carry the answer
| Code | Meaning |
|---|---|
0 |
Yes, or done |
1 |
No |
2 |
Could not tell |
64 |
The command line was wrong |
65 |
The input was not what the program can work on |
69 |
A program it needs is not installed |
70 |
Its own fault |
74 |
Reading or writing failed |
Failures start where sysexits.h puts them, so "no" and "broke" are never the
same number. That is what makes this work:
for f in *.pdf; do pdftriage --quiet "$f" && ocrmypdf "$f" "ocr/$f"; done
A consumer closing the pipe is not a failure. … | head -1 exits 0, the
same as any other filter.
Nothing reaches outside the process
No network, no database, no configuration file, no state kept between runs.
Five exceptions, each declared in --help:
embmakes an HTTP call, which is the whole of what it does.bibfetchasks a register about an identifier or a title, likewise.pdftriagerunspdftotextandpdfinfo.pdfgutterrunspdftoppmto look at a page.avplanrunsffprobeto read a file's streams, andffmpegto ask which encoders the machine has.
The three that call another program exit 69 when it is missing, rather than
guessing at what it would have said.
Nothing is assumed about where it is running
No hostnames, no paths, no private vocabularies are built in. General reference data is — ISBN check digits, the HTML character references, the names of well-known publishers — and anything specific to one collection is supplied in a file rather than compiled in.
The same input gives the same output
On every run and on every machine. No dependence on hash ordering, no reading
of the clock or the locale. Numbers that JSON cannot represent are refused
where they are produced rather than written as null and lost somewhere
downstream.
--json is the stable form
Where a program prints something for a person to read, --json prints the
same answer as a record, and that record does not change shape without a major
version. Programs whose only sensible output is a stream of records write that
stream and have no --json to add.
Every one writes its own manual page
--man writes the program's manual page, in roff, to stdout:
pdfgutter --man | gzip > /usr/local/share/man/man1/pdfgutter.1.gz
It is generated from the same definition --help is, so the two can never
come to say different things.
See also the recipes, for what these rules buy when the tools are put in a row.