catena/crates/booksniff

booksniff

What a file actually is, whatever its name claims.

Part of catena.

What it does

Reads the first few bytes of a file, and an archive's directory at the back, and says what format it really is. It never reads the whole file, so a directory of nine-hundred-megabyte scans costs the same as a directory of anything else.

Writes one doc record per file, with a note for anything worth remarking on: a name that disagrees with the bytes, a file too small to be a book, a name that says "sample".

When you would use it

• Before a conversion step, so a converter is never handed a format it cannot read. Most conversion failures are this, and the error a converter gives for it is rarely the error you want.
• Auditing a library you did not build, where files have been renamed, re-saved, and moved between machines that disagreed about extensions.
• Finding the files that are not books — the samples, the readmes, the cover images that ended up in a folder of PDFs.

A library accumulates files that lie: a DjVu named .pdf, a Word document named .epub, an RTF saved as .doc, an EPUB saved as .zip, a single PDF that somebody zipped and renamed back to .pdf.

Install

cargo install --git https://git.pumpin.online/funkpower/catena booksniff

Example

atlas.pdf is a DjVu that somebody renamed:

$ booksniff --plain book.pdf atlas.pdf chapter.md
pdf	book.pdf
djvu	atlas.pdf
	named .pdf, but the bytes say DjVu
txt	chapter.md
	925 bytes, which is too small to be a book

The record form, which is what a pipeline wants:

$ booksniff book.pdf
{"v":1,"path":"book.pdf","kind":"pdf","declared_ext":"pdf","bytes":17595,"pages":null,"language":null,"text_layer":null,"needs_ocr":null,"notes":[]}

Finding everything in a folder that is not really a PDF:

booksniff library/*.pdf | jq -r 'select(.kind != "pdf") | "\(.kind)\t\(.path)"'

Options

Option	Effect
--plain	Print a line a person can read instead of the record
--man	Write this page as a manual page, in roff

Exit codes

Code	Meaning
0	Every file was identified
1	At least one was not
64	The command line was wrong
74	A file could not be read

A file that cannot be read is reported on stderr and the rest still run, so one bad file does not stop a sweep.

What it recognises

Kind	How
pdf	%PDF-
djvu	The IFF container tag
ole2	The compound-file header every binary Word document starts with
epub	A zip holding mimetype and META-INF/container.xml
zip-wrapped-pdf	A zip whose only real member is one PDF
zip-wrapped-fb2	A zip whose only real member is named .fb2 — the .fb2.zip most FictionBooks are passed around as
cbz	A zip whose every real member is a page image (JPEG, PNG, GIF or WebP), and which carries a ComicInfo.xml or is itself named .cbz or .cbr
azw3, mobi	A Palm database tagged BOOKMOBI, told apart by the format version inside the first record
lit	ITOLITLS, Microsoft Reader's signature
rtf	{\rtf at the front, after any byte-order mark and blank space, followed by the version digit, a space or the next control word
fb2	XML whose root element is FictionBook, found by stepping over the declaration, comments and doctype in the first four kilobytes; UTF-16 with a byte-order mark is decoded first
html	Markup at the front, after any byte-order mark
txt	Text that is none of the above
unknown	Anything else, with a note saying what was seen

An archive holding several PDFs is a collection, not a wrapped book: choosing among them would be a guess, and naming one chapter as the whole work is worse than refusing.

A zipped FictionBook is reported as wrapped rather than as fb2, the same way a zipped PDF is: a converter handed it as FictionBook is handed a zip, and the kind is there to stop exactly that.

A comic is the pages and nothing else. Directories, the ComicInfo.xml a comic reader writes into the archive, and the desktop clutter every archive picks up are passed over; anything more — one PDF among the pages, a readme — and the zip is not a comic, and says what it holds instead.

A zip of pictures is as often a holiday as a comic, so images alone are not enough. It is a comic when it carries a ComicInfo.xml, or when its own name ends .cbz or .cbr. Otherwise it is unknown, with a note saying it is a zip of so many images and would be a comic book if its name said .cbz.

Members are judged by name. Only the archive's directory is read, so no member is ever unpacked and a page called 001.jpg is taken to be an image without its bytes being looked at. The same goes for a wrapped FictionBook: the one member is taken to be a FictionBook because its name ends .fb2.

FictionBook is read as bytes rather than as text. A great many are written in windows-1251, which spells its markup in ASCII but is not UTF-8, and would otherwise be called binary.

As a library

use booksniff::bytes::File;

let file = File::open(std::path::Path::new("atlas.pdf"))?;
let doc = booksniff::sniff(&file, "atlas.pdf");
assert_eq!(doc.kind, corpus_core::Kind::Djvu);

sniff works over anything implementing booksniff::bytes::Bytes, which is implemented for &[u8] as well as for a file — so a caller that already has the bytes need not write them to disk first.

See also

• pdftriage — once you know it is a PDF, whether its text is worth reading.
• The records — the shape this writes.