funkpower/catena
1
0 Fork 0
mirror of https://git.pumpin.online/funkpower/catena.git synced 2026-11-13 23:55:40 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: funkpower/catena:v0.1.0
funkpower/catena:main funkpower/catena:issues-3-6 funkpower/catena:work-edition-metadata funkpower/catena:fixes-040 funkpower/catena:fixes-040-search funkpower/catena:fixes-040-epub funkpower/catena:fixes-040-sniff funkpower/catena:bookmeta-epub-cover funkpower/catena:bibfetch-search funkpower/catena:release-0.4.0 funkpower/catena:booksniff-more-formats funkpower/catena:bibfetch-openlibrary-cover funkpower/catena:bookmeta-mobi-exth
funkpower/catena:v0.5.0 funkpower/catena:v0.4.0 funkpower/catena:v0.3.1 funkpower/catena:v0.3.0 funkpower/catena:v0.2.3 funkpower/catena:v0.2.2 funkpower/catena:v0.2.1 funkpower/catena:v0.2.0 funkpower/catena:v0.1.0
...
compare: funkpower/catena:fixes-040-epub
funkpower/catena:main funkpower/catena:issues-3-6 funkpower/catena:work-edition-metadata funkpower/catena:fixes-040 funkpower/catena:fixes-040-search funkpower/catena:fixes-040-epub funkpower/catena:fixes-040-sniff funkpower/catena:bookmeta-epub-cover funkpower/catena:bibfetch-search funkpower/catena:release-0.4.0 funkpower/catena:booksniff-more-formats funkpower/catena:bibfetch-openlibrary-cover funkpower/catena:bookmeta-mobi-exth
funkpower/catena:v0.5.0 funkpower/catena:v0.4.0 funkpower/catena:v0.3.1 funkpower/catena:v0.3.0 funkpower/catena:v0.2.3 funkpower/catena:v0.2.2 funkpower/catena:v0.2.1 funkpower/catena:v0.2.0 funkpower/catena:v0.1.0
35 Commits
v0.1.0 ... fixes-040-epub
Author SHA1 Message Date
funkpower 0e83c6ffc1 bookmeta: an EPUB cover by the word, not the letters 
The last route took any image whose id or address contained "cover", so
images/discover-ch1.png listed before images/cover.jpg won, and
recovery-map.png alone became the cover. A name now says cover only with
it as a word of its own: words break at anything not a letter or digit
and where a lower case letter meets a capital, so CoverImage and
Book_COVER still do. An image named cover outright comes first, then
manifest order. It is still a guess.

The guide's cover page took its first image, which can be a logo. Where
the page has several, the one whose file name, alt, id or class says
cover is taken; where none does, the first is, with a note saying how
many there were.

Each member was read no further than MAX_COVER, but nothing bounded how
many: two hundred cover-named images of 16 MiB took seven seconds to
inflate and left two hundred notes. One search now reads at most eight
members and stops with one note.

A percent-escape takes two hex digits and nothing else. from_str_radix
also takes a sign, so a%+fb decoded to a, 0x0f, b.
 2026-11-13 15:49:18 -05:00
funkpower de23a7abad bookmeta, bibfetch, bindings: what a caller needs to trust a cover 
bookmeta: Cover::guessed says the file left the choice open — a Kindle
book's first image record, or an EPUB image picked because its name says
cover — so a caller with a register to ask can prefer that. A cover the
file names is never a guess. Cover::extension matches bibfetch's.

bibfetch: Open Library answers 403 to more than a hundred covers by ISBN
in five minutes from one address. That is now Error::Limited, not
Unreachable: the other form of the ISBN is not asked, the cover id still
is, since those are not limited, and the CLI exits 75. Covers asked by
ISBN are kept three seconds apart across the process (Covers::pace), which
stays under the limit however long a run goes. A deadline too long to add
to the clock no longer panics; it is no deadline.

bindings: fetch_cover and search_title for Python, and catena.Limited,
which fetch_identifier raises too when a register refuses for being asked
too often.

corpus-core: exit 75, a service refused for now.
 2026-11-13 15:44:50 -05:00
funkpower 7aa1783a3e Merge remote-tracking branch 'origin/bibfetch-search' into fixes-040 2026-11-13 15:32:45 -05:00
funkpower 4d21cc5927 Merge remote-tracking branch 'origin/bookmeta-epub-cover' into fixes-040 2026-11-13 15:32:45 -05:00
funkpower 6510973850 Merge remote-tracking branch 'origin/booksniff-more-formats' into fixes-040 2026-11-13 15:32:45 -05:00
funkpower 897b2210b6 Merge remote-tracking branch 'origin/main' into bookmeta-epub-cover 
# Conflicts:
#	crates/bookmeta/src/mobi.rs
 2026-11-13 15:29:53 -05:00
funkpower 03c19c7293 Merge remote-tracking branch 'origin/main' into bibfetch-search 2026-11-13 15:29:05 -05:00
funkpower ce8332d6d8 bibfetch: the books a register has under a title 
A book that arrives as a file often has no identifier to ask about, only
a title and perhaps a name. `bibfetch search --title T [--author A]
[--limit N]` asks Open Library's index which books it has under those,
and Google Books only when Open Library has nothing, and prints one
record per candidate in the order the register gave them. From Rust,
`bibfetch::search(&agent, title, author, limit, attempts)`.

Each candidate is one edition. The index answers with works and, asked
for `editions.*`, names the edition of each that best matches; the ISBN,
publisher, language, year and cover come from that edition, the title,
authors, subjects and length from the work. The work's own ISBN list
mixes editions: the first War and Peace work lists four, while the
edition matched is a 1917 printing with none.

`id` is the first ISBN whose check digit agrees, in thirteen digits,
thirteen-digit entries taken first. A candidate with none is named by
the register's key: `olid` (OL…M for the edition, OL…W for the work when
no edition was named) or `googlebooks` (a volume id). Google's `OTHER`
shelfmarks are never read as ISBNs.

`cover_id` keeps the meaning it has in a lookup record: the edition's
own first cover, taken from `editions.cover_i` and only when the record
names that edition. The work's `cover_i` and `cover_edition_key` are not
asked for. A work with no edition named, and every Google Books
candidate, has none.

Open Library having nothing stands when Google Books is down; both down
is exit 69. Exit 0 a candidate, 1 none, 65 an empty title.

The retrying GET that lookups use is now `ask::get`, shared by both, and
no longer sleeps after its last attempt.
 2026-11-13 15:28:08 -05:00
funkpower 57ce9137e6 bookmeta: read an EPUB's cover, by the four routes a package can name one 
The cover is the first image reached by EPUB 3's cover-image property,
EPUB 2's <meta name="cover"> (by id, or by address where a converter wrote
one), the guide's cover page and the first <img> or SVG <image> on it, and
last any manifest image whose id or address says cover. A route that names
something unusable leaves a note and the next is asked.

Addresses are resolved against the document they are written in, with the
fragment off and percent-escapes undone; one that climbs out of the archive
or carries a scheme is not followed. Every member is read through a bound,
checked against the size the archive declares and enforced on the read
itself, so an understated entry stops at the cap too.

The media type is read off the bytes, now WebP as well, and a manifest that
disagrees is a note; bytes that are not an image are no cover. MAX_COVER
(16 MiB) is the cap for both formats, so a Kindle cover record larger than
that is left out as well.
 2026-11-13 15:26:49 -05:00
funkpower b4000b9b67 Merge pull request 'bibfetch: Open Library cover fetch by ISBN' (#2) from bibfetch-openlibrary-cover into main 2026-11-13 15:26:01 -05:00
funkpower 5c4f770d76 booksniff: RTF, Microsoft Reader, FictionBook and comic archives 
Four formats a library of ebooks holds and this had no name for, so each
came back as text, as HTML or as unknown.

RTF is its opening group, after any byte-order mark and blank space.
Microsoft Reader is its signature at the front. A FictionBook is XML
whose root element is FictionBook, found by stepping over the
declaration, comments and doctype in the first four kilobytes. That walk
is over bytes rather than text, because a great many FictionBooks are
windows-1251 and would otherwise be called binary; other XML is still
HTML, as before.

A zip whose only real member is one FictionBook is zip-wrapped-fb2, the
same way a zipped PDF is zip-wrapped-pdf: handed to a converter as
FictionBook it is still a zip. A zip whose every real member is a JPEG,
PNG, GIF or WebP is cbz, once directories, ComicInfo.xml and desktop
clutter are passed over. One PDF or one readme among the pages and it is
not a comic. Only the names in the archive's directory are read, as for
every other zip.

The kind vocabulary grows by five: lit, fb2, rtf, cbz and
zip-wrapped-fb2.
 2026-11-13 15:25:33 -05:00
funkpower 23969269e8 Merge remote-tracking branch 'origin/main' into bibfetch-openlibrary-cover 2026-11-13 15:24:43 -05:00
funkpower 452102639a Merge pull request 'bookmeta: MOBI/AZW/AZW3 metadata and cover from the MOBI header and EXTH records' (#1) from bookmeta-mobi-exth into main 2026-11-13 15:24:34 -05:00
funkpower a105b1c9aa bookmeta: with no cover named, take only the record at first_image 
When a MOBI or AZW3 file names no cover (no EXTH 201 or 202), the cover is
the record at the first image index if it is a JPEG, PNG or GIF, and
nothing otherwise, as calibre does (metadata/mobi.py:524-532). The walk to
the next image is gone: a later record may be an inline illustration, and
kindlegen puts a font or RESC record first.

EXTH 121 is no longer read, which also stops a stale KF8 boundary below
first_image from hiding the cover.
 2026-11-13 15:22:04 -05:00
funkpower 1abed4632d bibfetch: a cover id is the edition's own, and a fetch has a deadline 
Review id 3 on catena#2.

P0: Record.cover_id came from the search index's cover_i, the cover
chosen for the whole work, and the index is asked first, so cover()
fell back to another edition's image (Penguin War and Peace,
9780140447934, got Standard Ebooks' Maude cover 12621906). cover_i is
no longer asked for or read; cover_id is the first positive id in the
edition record's covers, or None. An ISBN request that fails, as
opposed to a 404, now ends the fetch with that failure instead of
going on to the cover id.

Redirects go only over HTTPS on the standard port to
covers.openlibrary.org, archive.org or a host under archive.org, and
never to an address by number; the covers store the fetch began at is
still allowed.

The 1 KB size floor is gone. With default=false a missing cover is a
404, and the stand-in without it is a 43-byte GIF the media type check
refuses, so the floor could only drop real Small covers.

Covers.deadline (DEADLINE, two minutes) bounds a whole fetch: every
address, attempt, redirect, body and wait. It starts once the fetch
holds the process lock. The timeout test no longer times a span that
includes the lock wait.
 2026-11-13 15:04:34 -05:00
funkpower 130324bbe0 bookmeta: file EXTH 113 by shape, and fall back to the first image record for a cover 
EXTH 113 is an ASIN only when it is ten letters and digits; a UUID goes
under uuid, as the EPUB reader files it, and anything else under other.
With no EXTH 201 or 202, the first record from the first image index that
is a JPEG, PNG or GIF is the cover, stopping at the KF8 boundary (EXTH 121).
 2026-11-13 14:51:43 -05:00
funkpower 19e5f3ddc3 bibfetch: a book's cover from Open Library, by its ISBN 
`bibfetch::cover(&agent, isbn, cover_id, size)` asks Open Library's covers
store by the ISBN's thirteen-digit form, then its ten-digit form, then by a
cover id a lookup record named, and returns the image or `Ok(None)`. The
answer is checked before it is trusted: JPEG or PNG by content type and by
magic bytes, between 1 KB and 10 MB, at most three redirects, none of them
off HTTPS. One request at a time per process; a 429 or 503 is waited out
for its Retry-After, up to a minute.

`idnorm::isbn` converts between the two forms, and only for an ISBN whose
check digit agrees. Lookup records carry `cover_id` from the edition record
and the search index. The CLI gains `bibfetch cover <isbn> -o <file>`.
 2026-11-13 14:36:32 -05:00
funkpower b7d98e9d1a bookmeta: read MOBI, AZW and AZW3 from the MOBI header and its EXTH records 
A Kindle book's title, authors, publisher, description, ISBN, ASIN,
subjects, date and language come from the EXTH block, with the full name,
the Palm database name and the header's locale as fallbacks. The cover
image (EXTH 201, or the thumbnail) is returned as Metadata::cover with its
media type; it stays out of the JSON record. Every offset is bounds-checked:
a damaged header gives an empty record with a note, a broken EXTH block
keeps what came before the break, and an encrypted book is still read and
noted as drm.

Metadata gains description, which the EPUB reader now fills from
dc:description as well.

Fixtures are a story and cover written for the tests (CC0), converted with
calibre to MOBI 6 and AZW3, plus no-EXTH, DRM-flagged and truncated copies.
 2026-11-13 14:17:44 -05:00
funkpower 6f32f52293 0.3.1, properly — the version, and the test that names the first register 
The commit that reordered the chain did not carry the version bump. The
bump was chained after check.sh with &&, the commit after a semicolon, so
a failing check skipped the bump and let the commit through. The tag then
pointed at a tree still calling itself 0.3.0, which the release workflow
asserts against and would have refused.

check.sh was failing because a test names which register is asked first
about an ISBN, which is exactly what changed. It now names the index, for
the reason the reorder happened.
 2026-09-22 10:03:50 -04:00
funkpower e480f07c32 0.3.1 — ask the index first, which answers in one request 
The chain asked OpenLibrary's edition record first and its search index
second. The edition record is the slow one: measured against this
library it takes five to twenty-eight seconds, and it names its authors
by key, so a second request was needed anyway. The index, asked for its
fields by name, returns the title, the authors, the publisher, the
language, a page count and the subjects together, in about a quarter of
a second.

Reordering them turns three requests per book into one, and a pass over
five thousand books from fifty-three hours into well under one. The
edition record is still there for a subtitle or a publisher the index
did not give.
 2026-09-22 09:36:46 -04:00
funkpower d551adc14c 0.3.0 — ask every register and keep all of the answer 
The chain asked three registers and returned whichever answered first
with a title and an author, discarding the rest. The registers are
complementary rather than redundant: OpenLibrary's edition record
carries the publisher, the pagination and the language and names its
authors by key, while its search index carries the authors and not the
publisher. One book, two half-answers, and the whole of it only if they
are put together.

They are merged now, first answer winning each field, so the order of
the chain is the order of trust. The record says which registers spoke.

It also carries what was being thrown away: subjects, language and a
page count, from whichever register has them. The index will not return
those unless asked for by name, which is how a language it held went
unread. Subjects are capped at twelve — the index lists every heading
every edition of a work has ever been given, which on a well-known book
is hundreds.

An edition record keeps its subtitle in a separate field, so a title
came back without it.
 2026-09-22 08:39:27 -04:00
funkpower 90fb977a10 colophon: six more things the front matter of a book is not 
Read out of the review queue a real run produced, where each of these was
sitting as a proposed title waiting for a person to say no:

  1. General Remarks            a numbered heading
  2   James W. McAllister       a numbered heading
  5222 What Is This             a folio that ran into the line
  827 North Fourth Street       an address
  Page xxxvi / Chapter I        a marker
  title                         a form label, printed literally
  Sidebottom, Harry. Ancient    a bibliography entry
  To Mackenzie,                 a dedication

The numbers are the delicate part. A leading number that could be a year
is left alone, because 1984 Revisited is a book and 5222 What Is This is
a folio, and nothing else distinguishes them. A dedication has to end in
a comma, so To Kill a Mockingbird survives.

The copyright notices in the same pile are already fixed: those
proposals were written before 0.2.2 and are stale rather than wrong.
 2026-09-21 15:47:24 -04:00
funkpower 05b712d084 0.2.2 — a subtitle begins with a capital 
title_case counted small words from the start of the whole string, so
every subtitle that opened with one was set in lower case: 'Aesthetic in
Kant: a Critique', 'Born Fighting: the Scots-Irish'. A colon starts a
title of its own and the first word of one is capitalised however small
it is.

Found by book-pipeline, which had been proposing a title page's capitals
verbatim into a catalogue and needed this to stop.
 2026-09-21 11:30:26 -04:00
funkpower 60943420b4 0.2.1 — the holder goes between the symbol and the year 
'(c) Alfredo Ferrarin 2004' was read as a title. The notice pattern
wanted the year to follow the symbol directly, and a copyright line puts
the holder's name in between, which is the ordinary way to set one.
 2026-09-21 11:19:54 -04:00
funkpower 3b1fbb73ce 0.2.0 
The Python surface gained enough this week that three different builds
were installed on two machines all calling themselves 0.1.0: moria had
read_colophon through merge_colophon, mirkwood had a partial from the
middle of the afternoon, and PyPI had neither. A version that does not
move is a version nobody can depend on, which is how book-pipeline ended
up running a wheel scp'd into place by hand.

New since 0.1.0: colophon --shelf and catena.shelve, which file a book's
printed Library of Congress headings onto a closed list; catena
.read_colophon; catena.fetch_identifier, the registers in process; and
catena.colophon_prompt / merge_colophon, the question about what the
pages did not settle and the reply folded back in.
 2026-09-21 11:10:13 -04:00
funkpower 8f048efd69 bindings: let go of the interpreter while reading a book 
read_colophon is several milliseconds of computation over a few hundred
kilobytes, and it held the GIL for all of them. A caller reading a
library through asyncio.to_thread therefore got no overlap whatsoever:
six books in flight, one book's worth of work at a time.

Measured on this library, that was most of why a concurrent pass still
paced at four and a half hours.
 2026-09-21 10:57:38 -04:00
funkpower 0416359c65 bindings: colophon's own question, and the reply folded back in 
colophon has composed the question about what its pages did not settle
since it was written, and merged a reply back into the record it read.
Both were reachable only from a terminal, so a pipeline wanting the same
fallback would have had to write its own prompt — a second one, drifting
from the first, asking for a shape merge does not accept.

merge fills only the unsettled fields, so a model cannot overwrite what a
book printed about itself. That is the property that makes asking one
safe, and it belongs to colophon rather than to any caller.
 2026-09-21 10:49:51 -04:00
funkpower f4497b3f0a colophon: the front matter of a book nobody typeset carefully 
Read against five thousand real books, the title page is often not the
first thing in the file, and what is there instead has shape. Six kinds
of it were being read as titles:

  vi   CONTENTS                                  a folioed heading
  Copyright (c) 2013 by Barry Davies, BEM        a notice
  Translated from the French by Willard R. Trask a credit
  Vol. 11. HENRY JAMES, JR.                      a collected edition's list
  [PART ONE:                                     a part heading
  Piper Verlag GmbH, Munich<span style=...       a converter's leavings

Rejecting one line only moves the misread down one, because a page that
converted badly converted badly throughout, so the whole shape goes:
markdown links, stylesheet declarations, HTML entities, epub filenames,
and the sentences a copyright page sets.

The notice pattern wants the symbol or the year, never the word alone.
The first draft rejected 'Copyright and Its Discontents', which its own
test caught.

On sixty books this drops five claimed titles, all of them wrong, and
fixes six that were reading one homily's heading for every other
homily's title.
 2026-09-21 10:47:10 -04:00
funkpower d76cee8abe fmt 
check.sh runs cargo fmt --check and three of my commits did not.
 2026-09-21 10:26:15 -04:00
funkpower f4d936e620 recipes, examples: shelving a library by what its books printed 
The suite could read a cataloguing block and could ask a register, and
nothing showed the two together — which is the only way either is worth
acting on. A title page alone gives up a running head when the scan
starts late; an ISBN alone returns a confidently wrong book when the
catalogue has been carrying the wrong one for years.

The example's title comparison folds to letters and digits rather than
comparing strings. A register and a title page differ over case, over a
curly apostrophe against a straight one, and over whether the subtitle
counts as part of the title. An example that called those disagreements
would teach the wrong lesson, and did until it was fixed.

It runs on this library and finds both of the cases worth finding: a file
called Document5 by -ducati996 that both sources agree is Russell's
Problems of Philosophy, and a Pelikan whose stored ISBN belongs to a book
about sculpture.
 2026-09-21 10:21:52 -04:00
funkpower 18cc217c0d bindings: what a book says about itself, and what the register says back 
book-pipeline reaches for colophon and bibfetch on every book it repairs.
Shelling out would mean a process per book per tool across five thousand
books, and parsing back the JSON it is about to parse anyway.

bibfetch's fetch loop lived in main.rs, so binding it meant either
lifting it or writing a second one. A second implementation of 'which
register answers about this kind of identifier, and in what order' would
drift from the first, and the copy nobody runs from a terminal would
drift silently. It is now bibfetch::ask, and the command line calls it
too — verified by running the CLI before and after and diffing.

fetch_identifier releases the GIL around the request. It is the only
function here that opens a socket, and holding the interpreter for the
length of an HTTP call would stop every other thread in it.

bibfetch is optional and only the python feature turns it on: its HTTP
client does not build for wasm32, and an unconditional dependency would
have taken the JavaScript package down with it.
 2026-09-21 10:20:24 -04:00
funkpower 6199fce251 colophon: file a book's printed subjects onto a shelf someone has 
A cataloguing block states subjects in Library of Congress form, which
is precise, deep and no use for browsing. Real blocks from this library
read 'Political science – Early works to 1800' and 'English
literature—Middle English, 1100–1500—History and criticism'. Nobody has
a shelf called that.

--shelf takes the closed list the library is actually browsed by and
files the printed headings onto it, longest term first so a period of
thought does not collapse into its parent. Matching is on whole words
after folding punctuation away, because a cataloguing block separates
with em dashes, en dashes, full stops and plain hyphens, often three in
one heading, and OCR turns any of those into any other.

What matches nothing comes back under unshelved rather than being
dropped: a subject silently discarded is indistinguishable from a book
that printed none, and the difference is the whole value of reading the
block.
 2026-09-21 10:14:52 -04:00
funkpower 4ea28c03b6 bibfetch: ask every register that answers about a book, in turn 
An ISBN is not a DOI. The body that assigns a DOI is the only one that has
it, so one register is the whole answer. A book's identifier is assigned by
its publisher and recorded by whoever gets round to it, and the record that
comes back is as often as not missing the thing the caller wanted: asked
about 9780199205592, OpenLibrary's edition record names *The First World
War* and the press that printed it, and nobody at all as its author, because
an edition record names its authors by key.

So an ISBN now has a chain rather than a register. The edition record first,
then OpenLibrary's own index — which answers with `author_name` filled in
and costs nothing — and Google Books last, because it answers anonymous
callers only until a daily quota runs out, after which it replies 429 to
everyone behind the same address, as it is doing here today.

The first complete answer wins, an incomplete one is kept in case nothing
better comes, and a register being down is a failure only when every
register is. `source` on the record says which one answered.

colophon comes with it: the imprint reader was taking `Oxford is a
registered trade mark of Oxford University Press` for a publisher and
`Oxford N`, off `Oxford New York`, for a place. A legal notice mentioning a
press is not the press, and a city is the city and not the first letter of
the next one.
 2026-09-21 09:28:27 -04:00
funkpower ba8ec81735 colophon: the record a book prints in its own front matter 
A catalogue's worst records are the ones whose file says nothing true about
itself: a PDF titled `Untitled` by an author called `Oxford`, which is a
publisher, from a filename that was never anything but a filename. What
those books do have is pages. The title page carries the title as the
publisher set it; the copyright page carries the imprint, the year, the
ISBN, and in most English books since the 1970s the cataloguing block, which
is a catalogue record a librarian already made, subject headings and all.

So this reads the pages. Every field names the page it came from, and a book
that printed nothing comes back empty with its fields in `unsettled` rather
than filled with something plausible — an extraction that starts at chapter
one has no front matter in it, and saying so is the answer.

Two pages look like a title page and are not, and both are how a catalogue
comes to name the wrong book. A series advertisement lists other titles in
the shape a title page uses, over as many as four pages with only the first
announcing itself. A translation's cataloguing block prints the original
title in brackets above the real one. Both are found and refused.

Three things the pages give that no file does: the subject headings a
librarian assigned, an ISBN that exists only in print — which turns a book
with no identifier into one `bibfetch` can look up — and a title in the case
the publisher meant, because a half title prints unbroken what the title
page breaks across two lines and sets in capitals.

`--prompt` and `--merge` are citefind's: a question about exactly the fields
the pages left unsettled, composed here and answered wherever the caller
keeps a model, with the advertising taken out of what the model is shown.
A field a model answered is marked `model`, so a record always says which of
it was read and which was inferred.

Measured against 600 books of the home corpus: the cataloguing block
answered for 21% of them and agreed with the catalogue 89% of the time, most
of the residue being the catalogue's error rather than this one's. Title
pages answered for another 41%. The remaining 38% print nothing readable,
and are what the prompt is for.
 2026-09-21 09:28:26 -04:00
funkpower d55486977d release: staging cannot create a package that has never existed 
The registry answers POST /-/stage/package with a 404 for a name it has not
seen, so the first release of a new name cannot be staged at all — no token
scope changes that. The step now says so and lets the release stand rather
than failing after PyPI has already published.

The first version goes out by hand with 2FA; every one after it stages from
here.
 2026-09-21 08:45:50 -04:00
56 changed files with 9130 additions and 162 deletions
Expand all files Collapse all files

No files matched your search

.forgejo/workflows/release.yml
+10
View File
@@ -147,5 +147,15 @@ jobs:
echo "::notice::catena-corpus@$version is already published"
exit 0
fi
# Staging offers a new version of a package that exists. It cannot
# bring one into being: the registry answers `POST /-/stage/package`
# with a 404 for a name it has never seen. So the first release of a
# new name is published by hand, with 2FA, and every release after
# it goes through here. Not a failure — PyPI is already out.
if ! npm view catena-corpus version >/dev/null 2>&1; then
echo "::warning::catena-corpus does not exist on npm yet, and staging cannot create it."
echo "::warning::Publish the first version by hand: npm publish crates/bindings/npm/dist --access public"
exit 0
fi
cd crates/bindings/npm/dist && npm stage publish --access public
echo "::notice::staged catena-corpus@$version; approve it with 2FA to publish"
Cargo.lock
Generated
+41 -25
View File
@@ -101,7 +101,7 @@ checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
[[package]]
name = "avplan"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -126,7 +126,7 @@ checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5"
[[package]]
name = "bibclean"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -143,7 +143,7 @@ dependencies = [
[[package]]
name = "bibfetch"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -196,7 +196,7 @@ dependencies = [
[[package]]
name = "bookmeta"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -210,7 +210,7 @@ dependencies = [
[[package]]
name = "booksniff"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -250,12 +250,14 @@ checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04"
[[package]]
name = "catena"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"bibclean",
"bibfetch",
"booksniff",
"chunkmd",
"citefind",
"colophon",
"corpus-core",
"cuekit",
"cuepack",
@@ -275,7 +277,7 @@ dependencies = [
[[package]]
name = "catena-cli"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"clap",
"clap_mangen",
@@ -310,7 +312,7 @@ checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
[[package]]
name = "chunkmd"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -332,7 +334,7 @@ dependencies = [
[[package]]
name = "citefind"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"chunkmd",
@@ -395,6 +397,20 @@ dependencies = [
"roff",
]
[[package]]
name = "colophon"
version = "0.3.1"
dependencies = [
"bibclean",
"catena-cli",
"clap",
"corpus-core",
"regex",
"schemars",
"serde",
"serde_json",
]
[[package]]
name = "colorchoice"
version = "1.0.5"
@@ -432,7 +448,7 @@ dependencies = [
[[package]]
name = "corpus-core"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"schemars",
"serde",
@@ -476,7 +492,7 @@ dependencies = [
[[package]]
name = "cuekit"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"corpus-core",
"regex",
@@ -488,7 +504,7 @@ dependencies = [
[[package]]
name = "cuenorm"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -498,7 +514,7 @@ dependencies = [
[[package]]
name = "cuepack"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -509,7 +525,7 @@ dependencies = [
[[package]]
name = "cueseg"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -596,7 +612,7 @@ dependencies = [
[[package]]
name = "emb"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -860,7 +876,7 @@ dependencies = [
[[package]]
name = "idnorm"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"bibclean",
"catena-cli",
@@ -926,7 +942,7 @@ dependencies = [
[[package]]
name = "langid"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -1006,7 +1022,7 @@ dependencies = [
[[package]]
name = "mdscrub"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -1018,7 +1034,7 @@ dependencies = [
[[package]]
name = "mdsplit"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -1097,7 +1113,7 @@ checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
[[package]]
name = "pdfgutter"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -1110,7 +1126,7 @@ dependencies = [
[[package]]
name = "pdftriage"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -1354,7 +1370,7 @@ checksum = "323c417e1d9665a65b263ec744ba09030cfb277e9daa0b018a4ab62e57bc8189"
[[package]]
name = "rrf"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -1501,7 +1517,7 @@ dependencies = [
[[package]]
name = "seriesof"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
@@ -1618,7 +1634,7 @@ checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca"
[[package]]
name = "textkit"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"regex",
"tiktoken-rs",
@@ -1708,7 +1724,7 @@ checksum = "fd3ca314f692efd6c868f8408f53fe444634a845f96c028b97d35f6a1f79f0ee"
[[package]]
name = "tocfind"
version = "0.1.0"
version = "0.3.1"
dependencies = [
"catena-cli",
"clap",
Cargo.toml
+27 -26
View File
@@ -3,38 +3,39 @@ resolver = "3"
members = ["crates/*"]
[workspace.package]
version = "0.1.0"
version = "0.3.1"
edition = "2024"
rust-version = "1.85"
license = "MPL-2.0"
repository = "https://git.pumpin.online/funkpower/catena"
[workspace.dependencies]
catena-cli = { path = "crates/catena-cli", version = "0.1.0" }
corpus-core = { path = "crates/corpus-core", version = "0.1.0" }
cuekit = { path = "crates/cuekit", version = "0.1.0" }
textkit = { path = "crates/textkit", version = "0.1.0" }
chunkmd = { path = "crates/chunkmd", version = "0.1.0" }
mdscrub = { path = "crates/mdscrub", version = "0.1.0" }
mdsplit = { path = "crates/mdsplit", version = "0.1.0" }
pdftriage = { path = "crates/pdftriage", version = "0.1.0" }
cuenorm = { path = "crates/cuenorm", version = "0.1.0" }
cueseg = { path = "crates/cueseg", version = "0.1.0" }
rrf = { path = "crates/rrf", version = "0.1.0" }
cuepack = { path = "crates/cuepack", version = "0.1.0" }
bibclean = { path = "crates/bibclean", version = "0.1.0" }
avplan = { path = "crates/avplan", version = "0.1.0" }
pdfgutter = { path = "crates/pdfgutter", version = "0.1.0" }
catena = { path = "crates/bindings", version = "0.1.0" }
seriesof = { path = "crates/seriesof", version = "0.1.0" }
idnorm = { path = "crates/idnorm", version = "0.1.0" }
bibfetch = { path = "crates/bibfetch", version = "0.1.0" }
bookmeta = { path = "crates/bookmeta", version = "0.1.0" }
langid = { path = "crates/langid", version = "0.1.0" }
tocfind = { path = "crates/tocfind", version = "0.1.0" }
booksniff = { path = "crates/booksniff", version = "0.1.0" }
citefind = { path = "crates/citefind", version = "0.1.0" }
emb = { path = "crates/emb", version = "0.1.0" }
catena-cli = { path = "crates/catena-cli", version = "0.3.1" }
corpus-core = { path = "crates/corpus-core", version = "0.3.1" }
cuekit = { path = "crates/cuekit", version = "0.3.1" }
textkit = { path = "crates/textkit", version = "0.3.1" }
chunkmd = { path = "crates/chunkmd", version = "0.3.1" }
mdscrub = { path = "crates/mdscrub", version = "0.3.1" }
mdsplit = { path = "crates/mdsplit", version = "0.3.1" }
pdftriage = { path = "crates/pdftriage", version = "0.3.1" }
cuenorm = { path = "crates/cuenorm", version = "0.3.1" }
cueseg = { path = "crates/cueseg", version = "0.3.1" }
rrf = { path = "crates/rrf", version = "0.3.1" }
cuepack = { path = "crates/cuepack", version = "0.3.1" }
bibclean = { path = "crates/bibclean", version = "0.3.1" }
avplan = { path = "crates/avplan", version = "0.3.1" }
pdfgutter = { path = "crates/pdfgutter", version = "0.3.1" }
catena = { path = "crates/bindings", version = "0.3.1" }
seriesof = { path = "crates/seriesof", version = "0.3.1" }
idnorm = { path = "crates/idnorm", version = "0.3.1" }
bibfetch = { path = "crates/bibfetch", version = "0.3.1" }
bookmeta = { path = "crates/bookmeta", version = "0.3.1" }
langid = { path = "crates/langid", version = "0.3.1" }
tocfind = { path = "crates/tocfind", version = "0.3.1" }
booksniff = { path = "crates/booksniff", version = "0.3.1" }
citefind = { path = "crates/citefind", version = "0.3.1" }
colophon = { path = "crates/colophon", version = "0.3.1" }
emb = { path = "crates/emb", version = "0.3.1" }
clap = { version = "4", features = ["derive"] }
clap_mangen = "0.3"
README.md
+2 -1
View File
@@ -25,6 +25,7 @@ Each tool installs by name, so you can take one without taking the rest.
|---|---|
| [`booksniff`](crates/booksniff/README.md) | What a file actually is, whatever its name claims |
| [`bookmeta`](crates/bookmeta/README.md) | The metadata a file already carries |
| [`colophon`](crates/colophon/README.md) | The record a book prints in its own front matter |
| [`tocfind`](crates/tocfind/README.md) | A book's own table of contents |
| [`langid`](crates/langid/README.md) | What language a document is in |
| [`pdfgutter`](crates/pdfgutter/README.md) | Books scanned two pages to a sheet, split back into pages |
@@ -33,7 +34,7 @@ Each tool installs by name, so you can take one without taking the rest.
| [`vocabmerge`](crates/bibclean/README.md#vocabmerge--the-other-command) | The near-duplicate headings in a catalogue's vocabulary |
| [`seriesof`](crates/seriesof/README.md) | The series a book belongs to, and which volume this is |
| [`idnorm`](crates/idnorm/README.md) | DOIs, ISSNs, ISBNs and arXiv ids, read and checked |
| [`bibfetch`](crates/bibfetch/README.md) | What a register says about an identifier |
| [`bibfetch`](crates/bibfetch/README.md) | What a register says about an identifier, and which books it has under a title |
| [`citefind`](crates/citefind/README.md) | The references in a bibliography or a page of notes |
| [`mdscrub`](crates/mdscrub/README.md) | Converted markdown in, clean markdown out |
| [`mdsplit`](crates/mdsplit/README.md) | One markdown file into the works inside it |
crates/bibclean/src/fix.rs
+45 -1
View File
@@ -67,7 +67,19 @@ pub fn title_case(title: &str) -> String {
.to_lowercase()
.trim_end_matches(['.', ',', ':', ';', '!', '?', ')', ']', '"', '\''])
.to_string();
if SMALL_WORDS.binary_search(&core.as_str()).is_ok() && at > 0 && at < last {
// A small word stays small in the middle of a title, but not
// where a title actually starts. A colon starts a subtitle, and
// "Aesthetic in Kant: a Critique" is the giveaway that the rule
// was only ever counting from the beginning of the whole string.
let opens_a_subtitle = at > 0
&& words[at - 1]
.trim_end_matches(['"', '\'', ')', ']'])
.ends_with([':', ';', '?', '!', '.']);
if SMALL_WORDS.binary_search(&core.as_str()).is_ok()
&& at > 0
&& at < last
&& !opens_a_subtitle
{
return word.to_lowercase();
}
let lowered = word.to_lowercase();
@@ -218,3 +230,35 @@ mod tests {
);
}
}
#[cfg(test)]
mod subtitle_case_tests {
use super::title_case;
#[test]
fn a_subtitle_starts_with_a_capital() {
// The rule counted small words from the start of the string, so every
// subtitle beginning "A", "The" or "Of" was set lower case.
assert_eq!(
title_case("AESTHETIC IN KANT: A CRITIQUE"),
"Aesthetic in Kant: A Critique"
);
assert_eq!(
title_case("BORN FIGHTING: THE SCOTS-IRISH"),
"Born Fighting: The Scots-Irish"
);
assert_eq!(
title_case("A HISTORY: AN INTRODUCTION TO IT"),
"A History: An Introduction to It"
);
}
#[test]
fn a_small_word_in_the_middle_is_still_small() {
assert_eq!(
title_case("THE RISE AND FALL OF THE THIRD REICH"),
"The Rise and Fall of the Third Reich"
);
assert_eq!(title_case("WAR OF THE WORLDS"), "War of the Worlds");
}
}
crates/bibclean/src/lib.rs
+6 -1
View File
@@ -260,9 +260,14 @@ mod tests {
#[test]
fn an_underscored_title_is_cleaned_before_it_is_cased() {
let cleaned = clean(&book("THE FALL_ AND THE RISE_txt", "Arendt, Hannah"));
// "And" is capitalised because the colon starts a subtitle, and the
// first word of one is capitalised however small it is. This asserted
// "and" until the rule was found to be counting small words from the
// start of the whole string, which set every subtitle opening "A",
// "The" or "An" in lower case.
assert_eq!(
cleaned.book.title.as_deref(),
Some("The Fall: and the Rise")
Some("The Fall: And the Rise")
);
assert_eq!(cleaned.changes.len(), 2);
}
crates/bibfetch/README.md
+141 -3
View File
@@ -20,6 +20,8 @@ So this asks, and that is the only thing it does.
- **Filling in a reference** you have only the identifier for.
- **Before citing something**, to check the year and the authors are what you
think they are.
- **Cataloguing a book that came with no identifier**, to find the editions a
register has under its title, and an ISBN to file it by.
## Install
@@ -84,7 +86,20 @@ is a failure — nobody replied, and the identifier may be perfectly good.
|---|---|---|
| `doi` | Crossref | It assigns most scholarly DOIs |
| `arxiv` | the arXiv | Its own register of preprints |
| `isbn` | OpenLibrary | Books, without a key |
| `isbn` | OpenLibrary's edition record, then its index, then Google Books | Books, without a key |
One register is enough for a DOI or a preprint: the body that assigns the
identifier is the only one that has it. A book is different. An ISBN is
assigned by a publisher and recorded by whoever gets round to it, so an
answer that names the book but not who wrote it is common — OpenLibrary's
edition record names its authors by key rather than by name, and its index
names them. So the registers are asked in turn and the first complete answer
wins. An incomplete one is kept in case nothing better comes, and a register
being down is a failure only when every register is.
Google Books is asked last because it answers anonymous callers only until a
daily quota runs out, after which it replies `429` to everyone sharing the
address.
An **ISSN** has no register that answers without a key, so it is not looked up
on its own. A journal's ISSN comes back on the works published in it instead.
@@ -94,6 +109,127 @@ register that is rate-limiting, short enough that one which is simply down
does not hold up a pipeline. An answer this cannot read is not retried,
because it will not read any better next time.
## Covers
```sh
bibfetch cover 978-0-306-40615-7 -o cover.jpg
```
Fetches a book's cover from Open Library's covers store by its ISBN and
writes the image. The thirteen-digit form is asked first and the ten-digit
form second, because a book is often filed under only one of them; an ISBN
whose check digit is wrong is asked about as written, since converting it
would give another book's number. `--cover-id` names an Open Library cover id
to ask by last, which is what the `cover_id` field of a lookup record holds:
the first cover in Open Library's record of that edition, never the cover its
index picked for the whole work, which is often another edition's. The cover
id is asked only when every form of the ISBN asked was answered with no cover; if
asking by ISBN failed, that failure is the result.
What comes back is checked before it is written: a JPEG or PNG by its bytes
as well as by what the server says, no larger than ten megabytes, reached in at
most three redirects, each over HTTPS to `covers.openlibrary.org`,
`archive.org` or a host under `archive.org`, and never to an address by number.
One request is made at a time, a `429` or `503` is waited out for as long as
its `Retry-After` says, up to a minute, and the whole fetch, every attempt and
redirect included, stops after two minutes. The two minutes start once it is
the fetch's turn, so a fetch queued behind others in the same process can wait
that long again first.
Open Library allows a hundred covers by ISBN every five minutes from one
address, and refuses the rest with `403`. Covers asked by ISBN are kept three
seconds apart across the whole process (`Covers::pace`), which stays under that
limit however long a run goes. A refusal all the same is not mistaken for an
outage: the other form of the ISBN is not asked, the cover id is, because
Open Library does not limit those, and with no cover found that way the fetch
fails with `Error::Limited`, which means ask again later.
| Option | Effect |
|---|---|
| `-o`, `--output <FILE>` | Where to write the image |
| `--size <SIZE>` | `small`, `medium` or `large`. Default `large` |
| `--cover-id <ID>` | An Open Library cover id to ask by when the ISBN has none |
Exit `0` the cover was written, `1` Open Library has none, `65` the argument
was not an ISBN or the answer was not an image, `69` Open Library could not be
reached, `74` the file could not be written, `75` Open Library refused for
being asked too often and will answer later.
From Rust, `bibfetch::cover(&agent, isbn, cover_id, size)` returns
`Ok(None)` when there is no cover, and a [`Cover`](src/cover.rs) with the
bytes, the media type and the address asked when there is.
## Searching by title
```
$ bibfetch search --title 'The First World War' --author Howard --limit 2
{"v":1,"id":"9780192853622","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War","authors":["Michael Eliot Howard"],"year":2002,"container":"Oxford University Press","subjects":["World War, 1914-1918"],"language":"eng","pages":154,"cover_id":1123124}
{"v":1,"id":"9780199205592","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War: A Very Short Introduction","authors":["Michael Howard"],"year":2007,"container":"Oxford University Press","language":"eng","cover_id":14606475}
```
A book that arrives as a file often has no identifier to ask about, only a
title and perhaps a name. This asks which books a register has under those,
and prints one record per candidate: Open Library's index first, and Google
Books only when Open Library has nothing.
Each candidate is one edition. Open Library's index answers with works, and
names the edition of each that best matches the search; what belongs to an
edition — its ISBN, publisher, language, year and cover — is that edition's,
and what belongs to the work — the title, the authors, the subjects and the
length — is the work's. Google Books answers with volumes, which are editions
already.
| Field | When the candidate has an ISBN | When it has none |
|---|---|---|
| `id` | The first ISBN whose check digit agrees, in thirteen digits; one written in thirteen is taken over one written in ten | The register's own key: an Open Library id such as `OL37863121M` for the edition, or `OL267171W` for the work when no edition was named; a Google Books volume id |
| `kind` | `isbn` | `olid` for Open Library, `googlebooks` for Google Books |
An ISBN whose check digit is wrong is passed over rather than corrected, and
Google's shelfmarks, typed `OTHER`, are not read as ISBNs even when their
digits would pass for one. A record of kind `isbn` can be handed straight back
to `bibfetch` for everything the registers say about that edition; `olid` and
`googlebooks` are names no lookup here answers about.
`cover_id` is the same thing it is in a lookup record: the first cover in
Open Library's record of the edition the record names, by that edition's ISBN
or its key, so `bibfetch cover --cover-id` fetches that edition's cover and no
other. The cover the index picked for the whole work is never asked for,
because it is often another edition's. A work the index names no edition of
gives the first ISBN it lists that reads, which may be any edition's, and no
cover id; Google Books knows nothing of Open Library's covers and gives none.
The candidates are what the register said, in the order it said them. This
makes no judgement about which one is the book in hand, and keeps one that is
plainly another book, because a title is not an identifier: *The Hamlet* comes
back for *Hamlet*. Deciding whether a candidate is the same work is a
comparison of titles, and [`bibclean`](../bibclean/README.md)'s `same_work`
is one.
```sh
bibfetch search --title 'Anathem' --author Stephenson \
| jq -r 'select(.kind == "isbn") | .id' | head -1
```
Open Library answering that it has nothing is an answer, and it stands when
Google Books cannot be reached; a register being down is a failure only when
both are. Each is asked the same way a lookup asks, retried with the same
waits. Open Library is sent the title and the author as written; Google Books
is sent each word as its own `intitle:` or `inauthor:` term, so nothing in a
title is read as search syntax.
| Option | Effect |
|---|---|
| `--title <TITLE>` | The title, or as much of it as you have |
| `--author <AUTHOR>` | An author's name, or part of one |
| `--limit <COUNT>` | The most candidates to print, `1` to `40`. Default `10` |
Exit `0` at least one candidate was printed, `1` no register has the title,
`65` the title was empty, `69` no register could be reached, `74` the output
could not be written.
From Rust, `bibfetch::search(&agent, title, author, limit, attempts)` returns
the candidates, empty when no register has the title.
## Reaching the network
This and [`emb`](../emb/README.md) are the only two tools here that do. Both
@@ -101,10 +237,12 @@ say so in `--help`, and for both it is the whole of what they do rather than
something that happens on the way to something else — which is the condition
under which [the contract](../../docs/contract.md) allows it at all.
Nothing is cached, nothing is stored, and nothing is sent but the identifier
Nothing is cached, nothing is stored but a cover you asked to have written,
and nothing is sent but the identifier, or the title and author searched for,
and the address you pass to `--polite`.
## See also
- [`idnorm`](../idnorm/README.md) — reading the identifier in the first place.
- [`bibclean`](../bibclean/README.md) — the rest of a catalogue record.
- [`bibclean`](../bibclean/README.md) — the rest of a catalogue record, and
whether a candidate is the same work.
crates/bibfetch/src/ask.rs
+153
View File
@@ -0,0 +1,153 @@
//! Asking the registers, which is the part that touches the network.
//!
//! Lifted out of the command line so that the Python bindings ask in exactly
//! the same order, with the same retries and the same backoff. Two
//! implementations of "which register answers about this identifier" would
//! drift, and the one nobody runs from a terminal would drift silently.
use std::time::Duration;
use crate::{Error, Record, Source};
/// How to ask: who to say you are, how long to wait, how often to retry.
#[derive(Debug, Clone)]
pub struct Ask {
/// An address the registers ask for, and answer faster when they get.
pub polite: Option<String>,
/// How long to wait on any one register.
pub timeout: Duration,
/// How many times to try a register that is unreachable.
pub attempts: u32,
}
impl Default for Ask {
fn default() -> Self {
Self {
polite: None,
timeout: crate::TIMEOUT,
attempts: crate::ATTEMPTS,
}
}
}
impl Ask {
/// An agent configured to ask this way.
pub fn agent(&self) -> ureq::Agent {
ureq::Agent::config_builder()
.timeout_global(Some(self.timeout))
.user_agent(user_agent(self.polite.as_deref()))
.build()
.into()
}
}
/// Ask every register that answers about this kind of identifier, in turn,
/// and stop at the first complete answer.
///
/// An incomplete answer is kept in case nothing better comes: a title with no
/// authors is still more than nothing, and a register being down is only a
/// failure when every register is. `Ok(None)` means no register here answers
/// about that kind at all.
pub fn lookup(
agent: &ureq::Agent,
id: &str,
kind: &str,
attempts: u32,
) -> Result<Option<Record>, Error> {
let chain = Source::chain_for_kind(kind);
if chain.is_empty() {
return Ok(None);
}
let mut best: Option<Record> = None;
let mut last_error = None;
for source in chain {
match ask(agent, *source, id, kind, attempts) {
Ok(record) => {
match &mut best {
// Merged rather than replaced. The registers answer about
// different parts of one book: OpenLibrary's edition record
// carries the publisher, the pagination and the language
// and names its authors by key, while its search index
// carries the authors and not the publisher. Keeping
// whichever arrived first throws the other half away.
Some(kept) => kept.fill_from(record),
None => best = Some(record),
}
if best.as_ref().is_some_and(Record::complete) {
break;
}
}
Err(why) => last_error = Some(why),
}
}
match best {
Some(record) => Ok(Some(record)),
None => {
Err(last_error.unwrap_or_else(|| Error::Unreachable("no register answered".into())))
}
}
}
/// Ask a register, retrying a failure that might not be one.
fn ask(
agent: &ureq::Agent,
source: Source,
id: &str,
kind: &str,
attempts: u32,
) -> Result<crate::Record, Error> {
match get(agent, &source.url(id), attempts)? {
Some(body) => source.read(id, kind, &body),
// A register that does not have it says so with a status, and that is
// an answer rather than a failure.
None => Ok(crate::missing(id, kind, source)),
}
}
/// The body at an address, or `None` when the register says it has no such
/// thing; a failure is asked again.
///
/// Waits a second before the second attempt and two before the third: long
/// enough for a register that is rate-limiting, short enough that one which
/// is simply down does not hold up a pipeline. Reading what came back is left
/// to the caller, because an answer this cannot read will not read any better
/// next time.
pub(crate) fn get(agent: &ureq::Agent, url: &str, attempts: u32) -> Result<Option<String>, Error> {
let attempts = attempts.max(1);
let mut last = Error::Unreachable("no attempt was made".into());
for attempt in 0..attempts {
match once(agent, url) {
Ok(answer) => return Ok(answer),
Err(e) => last = e,
}
if attempt + 1 < attempts {
std::thread::sleep(Duration::from_secs(1 << attempt.min(5)));
}
}
Err(last)
}
fn once(agent: &ureq::Agent, url: &str) -> Result<Option<String>, Error> {
let mut response = match agent.get(url).call() {
Ok(response) => response,
Err(ureq::Error::StatusCode(404)) => return Ok(None),
Err(e) => return Err(Error::Unreachable(e.to_string())),
};
response
.body_mut()
.read_to_string()
.map(Some)
.map_err(|e| Error::Unreachable(e.to_string()))
}
/// Who is asking.
///
/// The registers ask for an address and answer faster when they get one; a
/// tool that does not say who it is gets the slow pool.
fn user_agent(polite: Option<&str>) -> String {
let version = env!("CARGO_PKG_VERSION");
match polite {
Some(address) => format!("bibfetch/{version} (mailto:{address})"),
None => format!("bibfetch/{version}"),
}
}
crates/bibfetch/src/cover.rs
+1240
View File
File diff suppressed because it is too large. Load diff
crates/bibfetch/src/lib.rs
+177
View File
@@ -12,7 +12,14 @@
//! them: Crossref for a DOI, the arXiv for a preprint, OpenLibrary for a
//! book. There is no register for an ISSN that answers without a key, so an
//! ISSN is looked up as the journal of a work rather than on its own.
//!
//! A book with no identifier at all can still be looked for by its title,
//! with [`search()`]: Open Library's index, then Google Books, each candidate
//! a record of one edition, in the order the register gave them.
pub mod ask;
pub mod cover;
pub mod search;
pub mod source;
use std::time::Duration;
@@ -20,6 +27,8 @@ use std::time::Duration;
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
pub use cover::{Cover, CoverSize, cover};
pub use search::search;
pub use source::Source;
/// How long to wait on a register before giving up.
@@ -58,6 +67,28 @@ pub struct Record {
/// The abstract, where the register carries one.
#[serde(skip_serializing_if = "Option::is_none")]
pub summary: Option<String>,
/// The subject headings the register files it under, as it words them.
///
/// A register states these for far more books than print a cataloguing
/// block, which makes them the second source a shelf otherwise lacks.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub subjects: Vec<String>,
/// The language it is written in, as the register gives it: a two- or
/// three-letter code, or a name. Not normalised here — what a register
/// said is the record, and deciding that "eng" and "English" are one
/// thing belongs to whoever is filing them.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub language: Option<String>,
/// How many pages the register says it has.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub pages: Option<u32>,
/// The first cover id in Open Library's record of this edition, for
/// [`cover()`] to ask by when the ISBN has none. Taken from the edition
/// record only, never from the index's cover for the work, which is
/// often another edition's; `None` when the edition names no cover or
/// the lookup stopped before asking for the edition record.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub cover_id: Option<u64>,
}
fn version() -> u32 {
@@ -79,6 +110,11 @@ pub enum Error {
Shape(String),
/// Nothing here knows where to ask about this kind.
NoRegister(String),
/// What was asked about is not an identifier of the kind it should be.
Invalid(String),
/// The register refused because it has been asked too often from this
/// address. It is not down, and asking again later will work.
Limited(String),
}
impl std::fmt::Display for Error {
@@ -89,6 +125,8 @@ impl std::fmt::Display for Error {
Error::NoRegister(kind) => {
write!(f, "no register here answers about {kind} identifiers")
}
Error::Invalid(why) => write!(f, "{why}"),
Error::Limited(why) => write!(f, "{why}"),
}
}
}
@@ -111,6 +149,74 @@ pub fn missing(id: &str, kind: &str, source: Source) -> Record {
year: None,
container: None,
summary: None,
subjects: Vec::new(),
language: None,
pages: None,
cover_id: None,
}
}
impl Record {
/// Whether this record says everything a catalogue wants.
///
/// Used to decide when to stop asking. Title and authors are the minimum
/// that makes a record usable; the rest is worth another register only
/// while one is left to ask.
pub fn complete(&self) -> bool {
self.found
&& self.title.is_some()
&& !self.authors.is_empty()
&& self.container.is_some()
&& self.language.is_some()
}
/// Take from another record whatever this one does not have.
///
/// The registers are complementary rather than redundant, and asking them
/// in turn and keeping one answer throws away the difference. OpenLibrary's
/// edition record gives the publisher, the pagination and the language but
/// names its authors by key; its search index gives the authors but not the
/// publisher. One book, two half-answers, and the whole of it only if they
/// are put together.
///
/// Nothing already answered is overwritten, so the order of the chain is
/// the order of trust.
pub fn fill_from(&mut self, other: Record) {
if !other.found {
return;
}
self.found = true;
if self.title.is_none() {
self.title = other.title;
}
if self.authors.is_empty() {
self.authors = other.authors;
}
if self.year.is_none() {
self.year = other.year;
}
if self.container.is_none() {
self.container = other.container;
}
if self.summary.is_none() {
self.summary = other.summary;
}
if self.subjects.is_empty() {
self.subjects = other.subjects;
}
if self.language.is_none() {
self.language = other.language;
}
if self.pages.is_none() {
self.pages = other.pages;
}
if self.cover_id.is_none() {
self.cover_id = other.cover_id;
}
// Which register answered becomes which registers did.
if !self.source.split('+').any(|s| s == other.source) {
self.source = format!("{}+{}", self.source, other.source);
}
}
}
@@ -134,3 +240,74 @@ mod tests {
assert!(json.contains("\"found\":false"), "{json}");
}
}
#[cfg(test)]
mod merge_tests {
use super::*;
fn rec(source: &str) -> Record {
let mut r = missing("9780199205592", "isbn", Source::OpenLibrary);
r.source = source.to_string();
r.found = true;
r
}
#[test]
fn two_half_answers_make_one_record() {
// Exactly what the two OpenLibrary endpoints give for one book: the
// edition record has the publisher and the language and names its
// authors by key; the index has the authors and not the publisher.
let mut edition = rec("openlibrary");
edition.title = Some("The First World War: A Very Short Introduction".into());
edition.container = Some("Oxford University Press".into());
edition.language = Some("eng".into());
let mut index = rec("openlibrary");
index.title = Some("The First World War".into());
index.authors = vec!["Michael Howard".into()];
index.year = Some(2007);
edition.fill_from(index);
assert_eq!(edition.authors, ["Michael Howard"]);
assert_eq!(edition.year, Some(2007));
// The fuller title arrived first and is not displaced by the shorter.
assert_eq!(
edition.title.as_deref(),
Some("The First World War: A Very Short Introduction")
);
}
#[test]
fn a_register_that_has_nothing_contributes_nothing() {
let mut kept = rec("openlibrary");
kept.title = Some("A Book".into());
let absent = missing("9780199205592", "isbn", Source::GoogleBooks);
kept.fill_from(absent);
assert_eq!(kept.title.as_deref(), Some("A Book"));
assert_eq!(kept.source, "openlibrary");
}
#[test]
fn the_record_names_every_register_that_answered() {
let mut a = rec("openlibrary");
a.title = Some("A Book".into());
let mut b = rec("googlebooks");
b.found = true;
b.pages = Some(134);
a.fill_from(b);
assert_eq!(a.source, "openlibrary+googlebooks");
assert_eq!(a.pages, Some(134));
}
#[test]
fn complete_means_a_catalogue_has_what_it_needs() {
let mut r = rec("openlibrary");
assert!(!r.complete());
r.title = Some("A Book".into());
r.authors = vec!["Someone".into()];
assert!(!r.complete(), "still missing the publisher and language");
r.container = Some("A Press".into());
r.language = Some("eng".into());
assert!(r.complete());
}
}
crates/bibfetch/src/main.rs
+157 -70
View File
@@ -4,7 +4,6 @@ use std::path::PathBuf;
use std::process::ExitCode;
use std::time::Duration;
use bibfetch::{Error, Source};
use clap::Parser;
use corpus_core::exit;
@@ -30,33 +29,117 @@ Exit codes: 0 every identifier was found, 1 at least one was not,
Example:
idnorm refs.txt | bibfetch | jq -r 'select(.found) | .title'
idnorm --kind doi refs.txt | bibfetch --polite you@example.org"
idnorm --kind doi refs.txt | bibfetch --polite you@example.org
bibfetch cover 978-0-306-40615-7 -o cover.jpg
bibfetch search --title 'The First World War' --author Howard"
)]
struct Args {
#[command(subcommand)]
command: Option<Command>,
/// Identifier records to look up, as `idnorm` writes them. Reads stdin
/// when absent.
path: Option<PathBuf>,
/// An address to put in the request, which the registers ask for and
/// answer faster when they get.
#[arg(long, value_name = "EMAIL")]
#[arg(long, global = true, value_name = "EMAIL")]
polite: Option<String>,
/// How long to wait on a register, in seconds.
#[arg(long, default_value_t = bibfetch::TIMEOUT.as_secs(), value_name = "SECONDS")]
#[arg(long, global = true, default_value_t = bibfetch::TIMEOUT.as_secs(), value_name = "SECONDS")]
timeout: u64,
/// How many times to ask before believing a failure.
#[arg(long, default_value_t = bibfetch::ATTEMPTS, value_name = "COUNT")]
#[arg(long, global = true, default_value_t = bibfetch::ATTEMPTS, value_name = "COUNT")]
attempts: u32,
}
#[derive(clap::Subcommand)]
enum Command {
/// Fetch a book's cover from Open Library by its ISBN.
///
/// Exit codes: 0 the cover was written, 1 Open Library has none,
/// 65 not an ISBN or not an image, 69 Open Library could not be reached,
/// 74 the file could not be written.
Cover {
/// The ISBN, ten or thirteen digits, hyphens or not.
isbn: String,
/// Where to write the image.
#[arg(short, long, value_name = "FILE")]
output: PathBuf,
/// Which size to ask for.
#[arg(long, value_enum, default_value_t = Size::Large)]
size: Size,
/// An Open Library cover id to ask by when the ISBN has no cover.
#[arg(long, value_name = "ID")]
cover_id: Option<u64>,
},
/// Find the books a register has under a title, and an author.
///
/// Prints one record per candidate edition, in the order the register
/// gave them: Open Library's index, or Google Books when that has
/// nothing. Which one is the book in hand is left to you.
///
/// Exit codes: 0 at least one candidate, 1 none, 65 no title,
/// 69 no register could be reached, 74 the output could not be written.
Search {
/// The title, or as much of it as you have.
#[arg(long)]
title: String,
/// An author's name, or part of one.
#[arg(long)]
author: Option<String>,
/// The most candidates to print.
#[arg(long, default_value_t = 10, value_name = "COUNT",
value_parser = clap::value_parser!(u16).range(1..=bibfetch::search::MAX_LIMIT as i64))]
limit: u16,
},
}
#[derive(Clone, Copy, clap::ValueEnum)]
enum Size {
Small,
Medium,
Large,
}
fn main() -> ExitCode {
let args: Args = match catena_cli::parse() {
Ok(args) => args,
Err(code) => return code,
};
let agent = bibfetch::ask::Ask {
polite: args.polite.clone(),
timeout: Duration::from_secs(args.timeout),
attempts: args.attempts,
}
.agent();
match &args.command {
Some(Command::Cover {
isbn,
output,
size,
cover_id,
}) => return cover(&agent, args.attempts, isbn, output, *size, *cover_id),
Some(Command::Search {
title,
author,
limit,
}) => {
return search(
&agent,
args.attempts,
title,
author.as_deref(),
usize::from(*limit),
);
}
None => {}
}
let text = match catena_cli::read_text(args.path.as_deref()) {
Ok(text) => text,
Err((code, why)) => {
@@ -65,12 +148,6 @@ fn main() -> ExitCode {
}
};
let agent: ureq::Agent = ureq::Agent::config_builder()
.timeout_global(Some(Duration::from_secs(args.timeout)))
.user_agent(user_agent(args.polite.as_deref()))
.build()
.into();
let mut records = Vec::new();
let mut missing = 0usize;
@@ -86,14 +163,12 @@ fn main() -> ExitCode {
}
};
let Some(source) = Source::for_kind(&kind) else {
eprintln!("bibfetch: {id}: no register here answers about {kind} identifiers");
missing += 1;
continue;
};
match ask(&agent, source, &id, &kind, args.attempts) {
Ok(record) => {
match bibfetch::ask::lookup(&agent, &id, &kind, args.attempts) {
Ok(None) => {
eprintln!("bibfetch: {id}: no register here answers about {kind} identifiers");
missing += 1;
}
Ok(Some(record)) => {
if !record.found {
missing += 1;
}
@@ -116,62 +191,74 @@ fn main() -> ExitCode {
})
}
/// Ask a register, retrying a failure that might not be one.
///
/// Waits a second before the second attempt and two before the third: long
/// enough for a register that is rate-limiting, short enough that one which
/// is simply down does not hold up a pipeline.
fn ask(
/// Fetch one cover and write it where asked.
fn cover(
agent: &ureq::Agent,
source: Source,
id: &str,
kind: &str,
attempts: u32,
) -> Result<bibfetch::Record, Error> {
let mut last = Error::Unreachable("no attempt was made".into());
for attempt in 0..attempts.max(1) {
match fetch(agent, source, id, kind) {
Ok(record) => return Ok(record),
// A shape this cannot read will not read any better next time.
Err(e @ Error::Shape(_)) => return Err(e),
Err(e) => last = e,
}
std::thread::sleep(Duration::from_secs(1 << attempt));
}
Err(last)
}
fn fetch(
agent: &ureq::Agent,
source: Source,
id: &str,
kind: &str,
) -> Result<bibfetch::Record, Error> {
let response = agent.get(source.url(id)).call();
let mut response = match response {
Ok(response) => response,
// A register that does not have it says so with a status, and that is
// an answer rather than a failure.
Err(ureq::Error::StatusCode(404)) => return Ok(bibfetch::missing(id, kind, source)),
Err(e) => return Err(Error::Unreachable(e.to_string())),
isbn: &str,
output: &std::path::Path,
size: Size,
cover_id: Option<u64>,
) -> ExitCode {
let covers = bibfetch::cover::Covers {
attempts,
..Default::default()
};
let body = response
.body_mut()
.read_to_string()
.map_err(|e| Error::Unreachable(e.to_string()))?;
source.read(id, kind, &body)
let size = match size {
Size::Small => bibfetch::CoverSize::Small,
Size::Medium => bibfetch::CoverSize::Medium,
Size::Large => bibfetch::CoverSize::Large,
};
match covers.fetch(agent, isbn, cover_id, size) {
Ok(Some(found)) => match std::fs::write(output, &found.bytes) {
Ok(()) => ExitCode::from(exit::OK),
Err(why) => {
eprintln!("bibfetch: {}: {why}", output.display());
ExitCode::from(exit::IO)
}
},
Ok(None) => {
eprintln!("bibfetch: {isbn}: Open Library has no cover");
ExitCode::from(exit::NO)
}
Err(why) => {
eprintln!("bibfetch: {isbn}: {why}");
ExitCode::from(match why {
bibfetch::Error::Unreachable(_) => exit::UNAVAILABLE,
bibfetch::Error::Limited(_) => exit::TEMPFAIL,
_ => exit::DATA,
})
}
}
}
/// Who is asking.
///
/// The registers ask for an address and answer faster when they get one; a
/// tool that does not say who it is gets the slow pool.
fn user_agent(polite: Option<&str>) -> String {
let version = env!("CARGO_PKG_VERSION");
match polite {
Some(address) => format!("bibfetch/{version} (mailto:{address})"),
None => format!("bibfetch/{version}"),
/// Search by title and print the candidates.
fn search(
agent: &ureq::Agent,
attempts: u32,
title: &str,
author: Option<&str>,
limit: usize,
) -> ExitCode {
match bibfetch::search(agent, title, author, limit, attempts) {
Ok(found) => {
let wrote = catena_cli::stream("bibfetch", &found);
if format!("{wrote:?}") != format!("{:?}", ExitCode::from(exit::OK)) {
return wrote;
}
if found.is_empty() {
eprintln!("bibfetch: no register has {title:?}");
return ExitCode::from(exit::NO);
}
ExitCode::from(exit::OK)
}
Err(why) => {
eprintln!("bibfetch: {why}");
ExitCode::from(match why {
bibfetch::Error::Invalid(_) => exit::DATA,
_ => exit::UNAVAILABLE,
})
}
}
}
crates/bibfetch/src/search.rs
+780
View File
@@ -0,0 +1,780 @@
//! Books by their title, for when there is no identifier to ask about.
//!
//! A book that arrives as a file often carries no ISBN at all, only a title
//! and perhaps a name. A register can still be asked which books it has under
//! those, and what it answers is a list of candidates: each an edition the
//! register holds, with whatever identifier, publisher and cover it gives.
//!
//! Which candidate is the book in hand is not decided here. The candidates
//! are what the register said, in the order it said them; deciding whether
//! one is the same work is a comparison of titles, and that belongs to
//! whoever holds the book.
use serde::Deserialize;
use crate::source::{self, GoogleIdentifier, GoogleReply, SUBJECT_LIMIT, Source};
use crate::{Error, Record};
/// The most candidates one search asks for. Google Books answers with no more
/// than forty at a time, and a title that needs more than forty candidates to
/// find the right one needs a better title.
pub const MAX_LIMIT: usize = 40;
/// What Open Library's index is asked to say about each work, and about the
/// one edition of it that best matches the search.
///
/// Not the work's own `cover_i` or `cover_edition_key`: the cover the index
/// chose for a work is often another edition's. The edition's `cover_i` is
/// the first cover in that edition's record, which is what a lookup by ISBN
/// gives as `cover_id`.
const OPEN_LIBRARY_FIELDS: &str = "key,title,subtitle,author_name,first_publish_year,publisher,\
isbn,language,subject,number_of_pages_median,editions,editions.key,editions.isbn,\
editions.publisher,editions.language,editions.publish_date,editions.cover_i";
/// Ask which books a register has under a title, and by an author if one is
/// named.
///
/// Open Library's index first, and Google Books only when that answered with
/// nothing. A register being down is a failure only when every register
/// asked is: Open Library answering that it has nothing is an answer, and it
/// stands when Google Books cannot be reached. Asks nothing when `limit` is
/// zero; a `limit` over [`MAX_LIMIT`] is taken as that.
pub fn search(
agent: &ureq::Agent,
title: &str,
author: Option<&str>,
limit: usize,
attempts: u32,
) -> Result<Vec<Record>, Error> {
search_with(title, author, limit, |url| {
crate::ask::get(agent, url, attempts)
})
}
/// [`search`], with how to fetch an address passed in.
fn search_with(
title: &str,
author: Option<&str>,
limit: usize,
mut get: impl FnMut(&str) -> Result<Option<String>, Error>,
) -> Result<Vec<Record>, Error> {
let title = title.trim();
if title.is_empty() {
return Err(Error::Invalid("no title to search for".into()));
}
let author = author.map(str::trim).filter(|author| !author.is_empty());
let limit = limit.min(MAX_LIMIT);
if limit == 0 {
return Ok(Vec::new());
}
let mut fetch = |url: String| match get(&url)? {
Some(body) => Ok(body),
// A search that matches nothing answers with an empty list. A 404 is
// the address being wrong, not the book being absent.
None => Err(Error::Unreachable(format!("{url}: answered 404"))),
};
let open_library =
fetch(open_library_url(title, author, limit)).and_then(|body| open_library(&body, limit));
if open_library.as_ref().is_ok_and(|found| !found.is_empty()) {
return open_library;
}
let google = match google_url(title, author, limit) {
Some(url) => fetch(url).and_then(|body| google_books(&body, limit)),
// Nothing in the title Google could be asked by.
None => Ok(Vec::new()),
};
match (open_library, google) {
(_, Ok(found)) if !found.is_empty() => Ok(found),
(Ok(none), _) => Ok(none),
(Err(why), _) => Err(why),
}
}
/// Where to ask Open Library.
fn open_library_url(title: &str, author: Option<&str>, limit: usize) -> String {
let author = author
.map(|author| format!("&author={}", source::encode(author)))
.unwrap_or_default();
format!(
"https://openlibrary.org/search.json?title={}{author}&fields={OPEN_LIBRARY_FIELDS}&limit={limit}",
source::encode(title),
)
}
/// Where to ask Google Books, or `None` when there is nothing to ask by.
///
/// Every word is its own `intitle:` or `inauthor:` term, so no character of
/// the title can open a phrase, negate a term or name another field: a word is
/// the letters, digits and apostrophes in it, and anything else separates
/// words.
fn google_url(title: &str, author: Option<&str>, limit: usize) -> Option<String> {
let terms: Vec<String> = words(title)
.map(|word| format!("intitle:{word}"))
.chain(
author
.into_iter()
.flat_map(words)
.map(|word| format!("inauthor:{word}")),
)
.collect();
if !terms.iter().any(|term| term.starts_with("intitle:")) {
return None;
}
Some(format!(
"https://www.googleapis.com/books/v1/volumes?q={}&maxResults={limit}",
source::encode(&terms.join(" "))
))
}
fn words(text: &str) -> impl Iterator<Item = &str> {
text.split(|c: char| !(c.is_alphanumeric() || c == '\''))
.map(|word| word.trim_matches('\''))
.filter(|word| !word.is_empty())
}
/// The first ISBN in a list whose check digit agrees, in its thirteen-digit
/// form. One written in thirteen digits is taken over one written in ten.
fn first_isbn<'a>(isbns: impl Iterator<Item = &'a str> + Clone) -> Option<String> {
isbns
.clone()
.filter(|isbn| idnorm::isbn::digits(isbn).len() == 13)
.find_map(idnorm::isbn::to_13)
.or_else(|| isbns.into_iter().find_map(idnorm::isbn::to_13))
}
#[derive(Deserialize)]
struct OpenLibraryReply {
#[serde(default)]
docs: Vec<Work>,
}
#[derive(Deserialize)]
struct Work {
#[serde(default)]
key: Option<String>,
#[serde(default)]
title: Option<String>,
#[serde(default)]
subtitle: Option<String>,
#[serde(default)]
author_name: Vec<String>,
#[serde(default)]
first_publish_year: Option<i64>,
#[serde(default)]
publisher: Vec<String>,
#[serde(default)]
isbn: Vec<String>,
#[serde(default)]
language: Vec<String>,
#[serde(default)]
subject: Vec<String>,
#[serde(default)]
number_of_pages_median: Option<u32>,
#[serde(default)]
editions: Option<Editions>,
}
#[derive(Deserialize)]
struct Editions {
#[serde(default)]
docs: Vec<Edition>,
}
#[derive(Deserialize)]
struct Edition {
#[serde(default)]
key: Option<String>,
#[serde(default)]
isbn: Vec<String>,
#[serde(default)]
publisher: Vec<String>,
#[serde(default)]
language: Vec<String>,
#[serde(default)]
publish_date: Vec<String>,
#[serde(default)]
cover_i: Option<i64>,
}
/// The last segment of an Open Library key: `OL3585423M` out of
/// `/books/OL3585423M`. The letter at the end says what it names, `M` an
/// edition and `W` a work.
fn olid(key: &str) -> Option<String> {
key.rsplit('/')
.next()
.filter(|id| !id.is_empty())
.map(str::to_string)
}
/// Read Open Library's answer into one candidate per work, in its order.
///
/// What belongs to an edition — its ISBN, publisher, language, year and
/// cover — is taken from the edition the index chose as matching best; what
/// belongs to the work — its title, authors, subjects and length — from the
/// work. A work the index names no edition of gives its first ISBN that reads,
/// which may be any edition's, and no cover id.
fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
let reply: OpenLibraryReply =
serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?;
Ok(reply
.docs
.into_iter()
.filter_map(candidate)
.take(limit)
.collect())
}
fn candidate(work: Work) -> Option<Record> {
let edition = work.editions.and_then(|e| e.docs.into_iter().next());
// The cover is kept only when the record names the edition it is the
// cover of, by that edition's ISBN or its key.
let (id, kind, edition_named) = match &edition {
Some(edition) => match first_isbn(edition.isbn.iter().map(String::as_str)) {
Some(isbn) => (isbn, "isbn", true),
None => match edition.key.as_deref().and_then(olid) {
Some(key) => (key, "olid", true),
None => (work.key.as_deref().and_then(olid)?, "olid", false),
},
},
None => match first_isbn(work.isbn.iter().map(String::as_str)) {
Some(isbn) => (isbn, "isbn", false),
// A work with no key cannot be named, so it is not a candidate.
None => (work.key.as_deref().and_then(olid)?, "olid", false),
},
};
let mut record = source::blank(&id, kind, Source::OpenLibrarySearch);
record.title = match (work.title, work.subtitle) {
(Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")),
(title, _) => title,
};
record.authors = work.author_name;
record.subjects = work.subject;
record.subjects.truncate(SUBJECT_LIMIT);
record.pages = work.number_of_pages_median;
let (publisher, language, year) = match edition {
Some(edition) => {
if edition_named {
record.cover_id = edition
.cover_i
.and_then(|id| u64::try_from(id).ok())
.filter(|id| *id > 0);
}
(
edition.publisher.into_iter().next(),
edition.language.into_iter().next(),
edition
.publish_date
.iter()
.find_map(|date| source::year_in(date)),
)
}
None => (None, None, None),
};
record.container = publisher.or_else(|| work.publisher.into_iter().next());
record.language = language.or_else(|| work.language.into_iter().next());
record.year = year.or(work.first_publish_year);
Some(record)
}
/// Read Google Books' answer into one candidate per volume, in its order.
fn google_books(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
let reply: GoogleReply = serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?;
Ok(reply
.items
.into_iter()
.filter_map(|item| {
// Only what Google types as an ISBN: its `OTHER` identifiers are
// library shelfmarks, some of which are ten digits long.
let isbns = item
.volume_info
.identifiers
.iter()
.filter(|GoogleIdentifier { kind, .. }| kind == "ISBN_13" || kind == "ISBN_10")
.map(|identifier| identifier.identifier.as_str());
let (id, kind) = match first_isbn(isbns) {
Some(isbn) => (isbn, "isbn"),
None => (item.id.filter(|id| !id.is_empty())?, "googlebooks"),
};
Some(source::google_volume(&id, kind, item.volume_info))
})
.take(limit)
.collect())
}
#[cfg(test)]
mod tests {
use super::*;
/// Open Library's answer to `title=the first world war&author=howard`,
/// recorded 2026-11-13 with the fields above and `limit=2`, the ISBN and
/// subject lists shortened.
const HOWARD: &str = r#"{"numFound": 9, "start": 0, "numFoundExact": true, "num_found": 9,
"documentation_url": "https://openlibrary.org/dev/docs/api/search", "q": "", "offset": null,
"docs": [
{"author_name": ["Michael Eliot Howard"], "first_publish_year": 2002,
"isbn": ["9780192853622", "0192853627"], "key": "/works/OL1382726W",
"language": ["eng"], "number_of_pages_median": 154,
"publisher": ["Oxford University Press"], "title": "The First World War",
"subject": ["World War, 1914-1918"],
"editions": {"numFound": 1, "start": 0, "numFoundExact": true, "docs": [
{"key": "/books/OL3585423M", "cover_i": 1123124, "language": ["eng"],
"publisher": ["Oxford University Press"], "publish_date": ["2002"],
"isbn": ["0192853627", "9780192853622"]}]}},
{"author_name": ["Michael Howard"], "first_publish_year": 2007,
"isbn": ["0199205590", "9780199205592"], "key": "/works/OL4282399W",
"language": ["eng"], "publisher": ["Oxford University Press"],
"subtitle": "A Very Short Introduction", "title": "The First World War",
"editions": {"numFound": 1, "start": 0, "numFoundExact": true, "docs": [
{"key": "/books/OL7403004M", "cover_i": 14606475, "language": ["eng"],
"publisher": ["Oxford University Press"], "publish_date": ["20 May 2007"],
"isbn": ["9780199205592", "0199205590"]}]},
"subject": []}]}"#;
/// Open Library's answer to `title=war and peace&author=tolstoy`, recorded
/// 2026-11-13, first work only. The edition the index matched best is a
/// 1917 printing that has no ISBN, though other editions of the work do.
const TOLSTOY: &str = r#"{"numFound": 146, "docs": [
{"author_name": ["Leo Tolstoy"], "first_publish_year": 1867,
"isbn": ["9781524881993", "1645178889"], "key": "/works/OL267171W",
"title": "War and Peace",
"editions": {"numFound": 487, "docs": [
{"key": "/books/OL37863121M", "cover_i": 12736340, "language": ["eng"],
"publisher": ["Thomas Y. Crowell"], "publish_date": ["1917"]}]}}]}"#;
/// Open Library's answer when nothing has the title.
const NOTHING: &str = r#"{"numFound": 0, "start": 0, "numFoundExact": true, "num_found": 0,
"documentation_url": "https://openlibrary.org/dev/docs/api/search", "q": "",
"offset": null, "docs": []}"#;
/// Google Books' answer for a volume, in the shape its API documents.
const GOOGLE: &str = r#"{"kind": "books#volumes", "totalItems": 2, "items": [
{"kind": "books#volume", "id": "zyTCAlFPjgYC", "volumeInfo": {
"title": "The Google Story", "authors": ["David A. Vise", "Mark Malseed"],
"publisher": "Random House Digital, Inc.", "publishedDate": "2005-11-15",
"industryIdentifiers": [
{"type": "ISBN_10", "identifier": "055380457X"},
{"type": "ISBN_13", "identifier": "9780553804577"}],
"pageCount": 207, "categories": ["Browsers (Computer programs)"],
"language": "en"}},
{"kind": "books#volume", "id": "a1b2C3d4", "volumeInfo": {
"title": "The Google Story", "subtitle": "A Dissertation",
"publishedDate": "2009",
"industryIdentifiers": [
{"type": "OTHER", "identifier": "UOM:0306406152"}]}}]}"#;
/// Fetch from a table of recorded bodies, keeping every address asked.
fn recorded<'a>(
asked: &'a mut Vec<String>,
bodies: &'a [(&'a str, Result<&'a str, Error>)],
) -> impl FnMut(&str) -> Result<Option<String>, Error> + 'a {
move |url| {
asked.push(url.to_string());
let (_, body) = bodies
.iter()
.find(|(host, _)| url.contains(host))
.unwrap_or_else(|| panic!("asked an address nothing was recorded for: {url}"));
body.clone().map(|body| Some(body.to_string()))
}
}
fn down() -> Error {
Error::Unreachable("connection refused".into())
}
#[test]
fn each_work_open_library_has_is_a_candidate_named_by_its_isbn() {
let mut asked = Vec::new();
let bodies = [("openlibrary.org", Ok(HOWARD))];
let found = search_with(
"The First World War",
Some("Howard"),
5,
recorded(&mut asked, &bodies),
)
.expect("candidates");
assert_eq!(
asked.len(),
1,
"Google is not asked when Open Library answers"
);
assert_eq!(found.len(), 2);
let first = &found[0];
assert!(first.found);
assert_eq!(first.v, corpus_core::V);
assert_eq!(first.source, "openlibrary");
// Written ten-digit first by the edition, given in thirteen.
assert_eq!(
(first.id.as_str(), first.kind.as_str()),
("9780192853622", "isbn")
);
assert_eq!(first.title.as_deref(), Some("The First World War"));
assert_eq!(first.authors, ["Michael Eliot Howard"]);
assert_eq!(first.year, Some(2002));
assert_eq!(first.container.as_deref(), Some("Oxford University Press"));
assert_eq!(first.language.as_deref(), Some("eng"));
assert_eq!(first.subjects, ["World War, 1914-1918"]);
assert_eq!(first.pages, Some(154));
assert_eq!(first.cover_id, Some(1123124));
let second = &found[1];
assert_eq!(second.id, "9780199205592");
assert_eq!(
second.title.as_deref(),
Some("The First World War: A Very Short Introduction")
);
// The edition's year, not the work's first.
assert_eq!(second.year, Some(2007));
// The first cover in OL7403004M's own record; the work's cover is
// 1137779, which is not.
assert_eq!(second.cover_id, Some(14606475));
}
#[test]
fn the_candidates_stay_in_the_order_the_register_gave_them() {
let mut asked = Vec::new();
let bodies = [("openlibrary.org", Ok(HOWARD))];
let found =
search_with("first world war", None, 5, recorded(&mut asked, &bodies)).expect("ok");
let ids: Vec<&str> = found.iter().map(|r| r.id.as_str()).collect();
assert_eq!(ids, ["9780192853622", "9780199205592"]);
let found =
search_with("first world war", None, 1, recorded(&mut asked, &bodies)).expect("ok");
assert_eq!(found.len(), 1);
assert_eq!(found[0].id, "9780192853622");
}
#[test]
fn an_edition_with_no_isbn_is_named_by_its_key_and_keeps_its_cover() {
let mut asked = Vec::new();
let bodies = [("openlibrary.org", Ok(TOLSTOY))];
let found = search_with(
"War and Peace",
Some("Tolstoy"),
5,
recorded(&mut asked, &bodies),
)
.expect("candidates");
let only = &found[0];
// Not 9781524881993: that is another edition's, and the publisher,
// year and cover here are this one's.
assert_eq!(
(only.id.as_str(), only.kind.as_str()),
("OL37863121M", "olid")
);
assert_eq!(only.container.as_deref(), Some("Thomas Y. Crowell"));
assert_eq!(only.year, Some(1917));
assert_eq!(only.cover_id, Some(12736340));
}
#[test]
fn a_work_with_no_edition_named_gives_its_isbn_but_no_cover() {
let body = r#"{"docs": [{"key": "/works/OL1W", "title": "A Book",
"isbn": ["0-306-40615-2"], "publisher": ["Plenum"], "first_publish_year": 1981}]}"#;
let found = open_library(body, 5).expect("candidates");
assert_eq!(
(found[0].id.as_str(), found[0].kind.as_str()),
("9780306406157", "isbn")
);
assert_eq!(found[0].container.as_deref(), Some("Plenum"));
assert_eq!(found[0].year, Some(1981));
assert_eq!(found[0].cover_id, None);
}
#[test]
fn a_work_with_no_isbn_and_no_edition_is_named_by_the_works_key() {
let body = r#"{"docs": [{"key": "/works/OL267171W", "title": "War and Peace"}]}"#;
let found = open_library(body, 5).expect("candidates");
assert_eq!(
(found[0].id.as_str(), found[0].kind.as_str()),
("OL267171W", "olid")
);
assert_eq!(found[0].cover_id, None);
}
#[test]
fn isbns_whose_check_digit_is_wrong_are_passed_over() {
// 9780306406158 and 0306406153 are 9780306406157 mistyped.
let body = r#"{"docs": [
{"key": "/works/OL1W", "title": "A", "editions": {"docs": [
{"key": "/books/OL1M", "isbn": ["9780306406158", "0306406153"], "cover_i": 7}]}},
{"key": "/works/OL2W", "title": "B", "editions": {"docs": [
{"key": "/books/OL2M", "isbn": ["9780306406158", "0306406152"]}]}}]}"#;
let found = open_library(body, 5).expect("candidates");
assert_eq!(
(found[0].id.as_str(), found[0].kind.as_str()),
("OL1M", "olid")
);
assert_eq!(found[0].cover_id, Some(7));
// The ten-digit one reads, and is given in thirteen.
assert_eq!(
(found[1].id.as_str(), found[1].kind.as_str()),
("9780306406157", "isbn")
);
}
#[test]
fn thirteen_digits_are_taken_over_ten() {
// Two different ISBNs on one edition: the thirteen-digit one wins
// although the ten-digit one is listed first.
let isbns = ["0306406152", "9780199205592"];
assert_eq!(
first_isbn(isbns.into_iter()).as_deref(),
Some("9780199205592")
);
assert_eq!(first_isbn(["nonsense"].into_iter()), None);
}
#[test]
fn nothing_from_open_library_is_asked_of_google_books() {
let mut asked = Vec::new();
let bodies = [
("openlibrary.org", Ok(NOTHING)),
("googleapis.com", Ok(GOOGLE)),
];
let found = search_with(
"The Google Story",
Some("Vise"),
5,
recorded(&mut asked, &bodies),
)
.expect("candidates");
assert_eq!(asked.len(), 2);
assert!(asked[1].starts_with("https://www.googleapis.com/books/v1/volumes?q="));
assert_eq!(found.len(), 2);
let first = &found[0];
assert_eq!(first.source, "googlebooks");
assert_eq!(
(first.id.as_str(), first.kind.as_str()),
("9780553804577", "isbn")
);
assert_eq!(first.authors, ["David A. Vise", "Mark Malseed"]);
assert_eq!(first.year, Some(2005));
assert_eq!(
first.container.as_deref(),
Some("Random House Digital, Inc.")
);
assert_eq!(first.pages, Some(207));
// Google Books knows nothing of Open Library's covers.
assert_eq!(first.cover_id, None);
// A shelfmark is not an ISBN, though its digits would pass for one.
let second = &found[1];
assert_eq!(
(second.id.as_str(), second.kind.as_str()),
("a1b2C3d4", "googlebooks")
);
assert_eq!(
second.title.as_deref(),
Some("The Google Story: A Dissertation")
);
}
#[test]
fn nothing_from_either_register_is_no_candidates() {
let mut asked = Vec::new();
let bodies = [
("openlibrary.org", Ok(NOTHING)),
(
"googleapis.com",
Ok(r#"{"kind": "books#volumes", "totalItems": 0}"#),
),
];
let found =
search_with("Nothing", None, 5, recorded(&mut asked, &bodies)).expect("an answer");
assert!(found.is_empty());
}
#[test]
fn open_library_having_nothing_stands_when_google_books_is_down() {
let mut asked = Vec::new();
let bodies = [
("openlibrary.org", Ok(NOTHING)),
("googleapis.com", Err(down())),
];
let found =
search_with("Nothing", None, 5, recorded(&mut asked, &bodies)).expect("an answer");
assert!(found.is_empty());
}
#[test]
fn open_library_down_is_asked_of_google_books() {
let mut asked = Vec::new();
let bodies = [
("openlibrary.org", Err(down())),
("googleapis.com", Ok(GOOGLE)),
];
let found =
search_with("The Google Story", None, 5, recorded(&mut asked, &bodies)).expect("ok");
assert_eq!(found[0].source, "googlebooks");
}
#[test]
fn every_register_down_is_a_failure() {
let mut asked = Vec::new();
let bodies = [
("openlibrary.org", Err(down())),
("googleapis.com", Err(down())),
];
let failed = search_with("A Book", None, 5, recorded(&mut asked, &bodies));
assert!(matches!(failed, Err(Error::Unreachable(_))), "{failed:?}");
}
#[test]
fn an_answer_that_cannot_be_read_says_so() {
assert!(matches!(open_library("<html>", 5), Err(Error::Shape(_))));
assert!(matches!(google_books("<html>", 5), Err(Error::Shape(_))));
}
#[test]
fn the_author_is_left_out_when_there_is_none() {
for author in [None, Some(""), Some(" ")] {
let url = open_library_url(
"Anathem",
author.map(str::trim).filter(|a| !a.is_empty()),
5,
);
assert!(!url.contains("author="), "{url}");
}
let mut asked = Vec::new();
let bodies = [
("openlibrary.org", Ok(NOTHING)),
("googleapis.com", Ok(GOOGLE)),
];
search_with("Anathem", Some(" "), 5, recorded(&mut asked, &bodies)).expect("ok");
assert!(!asked[0].contains("author="), "{}", asked[0]);
assert!(!asked[1].contains("inauthor"), "{}", asked[1]);
assert!(asked[1].contains("q=intitle%3AAnathem&"), "{}", asked[1]);
}
#[test]
fn the_author_narrows_the_search_when_there_is_one() {
let url = open_library_url("Anathem", Some("Neal Stephenson"), 5);
assert!(
url.contains("?title=Anathem&author=Neal%20Stephenson&fields="),
"{url}"
);
let url = google_url("Anathem", Some("Neal Stephenson"), 5).expect("a url");
assert!(
url.contains("q=intitle%3AAnathem%20inauthor%3ANeal%20inauthor%3AStephenson&"),
"{url}"
);
}
#[test]
fn a_title_cannot_change_the_shape_of_the_address() {
let hostile = r#"Fish & Chips: "A History" #2?=x/y+z"#;
for url in [
open_library_url(hostile, Some("O'Brien & Sons"), 5),
google_url(hostile, Some("O'Brien & Sons"), 5).expect("a url"),
] {
assert_eq!(url.matches('?').count(), 1, "{url}");
assert_eq!(url.matches('#').count(), 0, "{url}");
assert_eq!(url.matches('"').count(), 0, "{url}");
assert_eq!(url.matches('+').count(), 0, "{url}");
let query = url.split_once('?').expect("a query").1;
let names: Vec<&str> = query
.split('&')
.map(|pair| pair.split_once('=').expect("name=value").0)
.collect();
assert!(
names == ["title", "author", "fields", "limit"] || names == ["q", "maxResults"],
"{url}"
);
}
// Open Library is asked the title as written.
let url = open_library_url(hostile, None, 5);
assert!(
url.contains("title=Fish%20%26%20Chips%3A%20%22A%20History%22%20%232%3F%3Dx%2Fy%2Bz&"),
"{url}"
);
// Google is asked word by word, so nothing in a title is syntax.
let url = google_url(hostile, Some("O'Brien & Sons"), 5).expect("a url");
let q = "intitle:Fish intitle:Chips intitle:A intitle:History intitle:2 intitle:x \
intitle:y intitle:z inauthor:O'Brien inauthor:Sons";
assert!(url.contains(&format!("q={}&", source::encode(q))), "{url}");
}
#[test]
fn a_title_outside_ascii_is_sent_as_utf8() {
let url = open_library_url("Преступление и наказание", Some("Dostoevský"), 5);
assert!(
url.contains("title=%D0%9F%D1%80%D0%B5%D1%81%D1%82%D1%83%D0%BF"),
"{url}"
);
assert!(url.contains("author=Dostoevsk%C3%BD&"), "{url}");
let url = google_url("Преступление и наказание", None, 5).expect("a url");
// Letters in any script are words.
assert!(url.contains("intitle%3A%D0%B8%20"), "{url}");
}
#[test]
fn a_title_with_no_words_is_not_asked_of_google_books() {
assert_eq!(google_url("?!", Some("Someone"), 5), None);
let mut asked = Vec::new();
let bodies = [("openlibrary.org", Ok(NOTHING))];
let found = search_with("?!", Some("Someone"), 5, recorded(&mut asked, &bodies))
.expect("an answer");
assert!(found.is_empty());
assert_eq!(asked.len(), 1);
}
#[test]
fn no_title_is_refused_without_asking() {
let mut asked = Vec::new();
let failed = search_with(" ", Some("Someone"), 5, recorded(&mut asked, &[]));
assert!(matches!(failed, Err(Error::Invalid(_))), "{failed:?}");
assert!(asked.is_empty());
}
#[test]
fn the_limit_is_asked_for_and_held_to() {
let mut asked = Vec::new();
let bodies = [
("openlibrary.org", Ok(NOTHING)),
("googleapis.com", Ok(GOOGLE)),
];
let found =
search_with("The Google Story", None, 1, recorded(&mut asked, &bodies)).expect("ok");
assert_eq!(found.len(), 1);
assert!(asked[0].ends_with("&limit=1"), "{}", asked[0]);
assert!(asked[1].ends_with("&maxResults=1"), "{}", asked[1]);
let mut asked = Vec::new();
search_with("The Google Story", None, 500, recorded(&mut asked, &bodies)).expect("ok");
assert!(asked[0].ends_with("&limit=40"), "{}", asked[0]);
assert!(asked[1].ends_with("&maxResults=40"), "{}", asked[1]);
let mut asked = Vec::new();
let found =
search_with("The Google Story", None, 0, recorded(&mut asked, &bodies)).expect("ok");
assert!(found.is_empty());
assert!(asked.is_empty());
}
#[test]
fn the_index_is_not_asked_for_the_works_cover() {
let url = open_library_url("A", None, 5);
let fields = url
.split('&')
.find_map(|pair| pair.strip_prefix("fields="))
.expect("fields");
let fields: Vec<&str> = fields.split(',').collect();
assert!(fields.contains(&"editions.cover_i"));
assert!(!fields.contains(&"cover_i"));
assert!(!fields.contains(&"cover_edition_key"));
}
#[test]
fn a_404_from_a_search_is_a_failure_and_not_an_absence() {
let failed = search_with("A Book", None, 5, |_| Ok(None));
assert!(matches!(failed, Err(Error::Unreachable(_))), "{failed:?}");
}
}
crates/bibfetch/src/source.rs
+282 -9
View File
@@ -17,6 +17,13 @@ pub enum Source {
Arxiv,
/// OpenLibrary, for books by ISBN.
OpenLibrary,
/// OpenLibrary's index, rather than its edition record. Asked when the
/// edition record has the book but not who wrote it, which is often: an
/// edition names its authors by key, and the index names them.
OpenLibrarySearch,
/// Google Books, for books by ISBN. Asked last, because it answers
/// anonymous callers only until a daily quota runs out.
GoogleBooks,
}
impl Source {
@@ -26,16 +33,42 @@ impl Source {
Source::Crossref => "crossref",
Source::Arxiv => "arxiv",
Source::OpenLibrary => "openlibrary",
Source::OpenLibrarySearch => "openlibrary",
Source::GoogleBooks => "googlebooks",
}
}
/// Which register answers about a kind of identifier.
pub fn for_kind(kind: &str) -> Option<Self> {
Self::chain_for_kind(kind).first().copied()
}
/// Every register that answers about a kind, in the order to ask them.
///
/// One register is enough for a DOI or a preprint: the body that assigns
/// the identifier is the only one that has it. A book is different. An
/// ISBN is assigned by a publisher and recorded by whoever gets round to
/// it, so coverage is patchy in different places for different books,
/// and an answer missing the authors is common enough to be worth asking
/// a second register about.
pub fn chain_for_kind(kind: &str) -> &'static [Self] {
match kind {
"doi" => Some(Source::Crossref),
"arxiv" => Some(Source::Arxiv),
"isbn" => Some(Source::OpenLibrary),
_ => None,
"doi" => &[Source::Crossref],
"arxiv" => &[Source::Arxiv],
// The index first. Asked for the fields by name it answers with
// the title, the authors, the publisher, the language, a page
// count and the subjects in one request, which is the whole of
// what a catalogue wants; the edition record is then needed only
// for a subtitle or a publisher the index did not give. Asking it
// second cost three requests per book where one usually does, and
// at three books in flight that is the difference between two
// days and two hours.
"isbn" => &[
Source::OpenLibrarySearch,
Source::OpenLibrary,
Source::GoogleBooks,
],
_ => &[],
}
}
@@ -50,6 +83,21 @@ impl Source {
Source::OpenLibrary => {
format!("https://openlibrary.org/isbn/{}.json", encode(id))
}
// `fields` or the index answers with a handful of defaults and
// keeps the rest, which is how a language it holds went unread.
// Not `cover_i`: that is the cover chosen for the whole work, and
// on a work with hundreds of editions it is usually another
// edition's, in another translation or language.
Source::OpenLibrarySearch => format!(
"https://openlibrary.org/search.json?q=isbn:{}\
&fields=title,subtitle,author_name,first_publish_year,publisher,language,subject,\
number_of_pages_median&limit=1",
encode(id)
),
Source::GoogleBooks => format!(
"https://www.googleapis.com/books/v1/volumes?q=isbn:{}",
encode(id)
),
}
}
@@ -59,6 +107,8 @@ impl Source {
Source::Crossref => crossref(id, kind, body),
Source::Arxiv => arxiv(id, kind, body),
Source::OpenLibrary => open_library(id, kind, body),
Source::OpenLibrarySearch => open_library_search(id, kind, body),
Source::GoogleBooks => google_books(id, kind, body),
}
}
}
@@ -67,7 +117,7 @@ impl Source {
///
/// A DOI suffix may hold a slash, a question mark or a hash, and every one of
/// those means something else in a URL.
fn encode(text: &str) -> String {
pub(crate) fn encode(text: &str) -> String {
let mut out = String::with_capacity(text.len());
for byte in text.bytes() {
match byte {
@@ -81,7 +131,7 @@ fn encode(text: &str) -> String {
}
/// A record with the fields nothing found fills in.
fn blank(id: &str, kind: &str, source: Source) -> Record {
pub(crate) fn blank(id: &str, kind: &str, source: Source) -> Record {
let mut record = crate::missing(id, kind, source);
record.found = true;
record
@@ -186,9 +236,27 @@ struct OpenLibraryBook {
#[serde(default)]
title: Option<String>,
#[serde(default)]
subtitle: Option<String>,
#[serde(default)]
publish_date: Option<String>,
#[serde(default)]
publishers: Vec<String>,
#[serde(default)]
subjects: Vec<String>,
/// `[{"key": "/languages/eng"}]`, which is how OpenLibrary names one.
#[serde(default)]
languages: Vec<OpenLibraryKey>,
#[serde(default)]
number_of_pages: Option<u32>,
/// Cover ids, best first. A deleted cover is left in the list as `-1`.
#[serde(default)]
covers: Vec<i64>,
}
#[derive(serde::Deserialize)]
struct OpenLibraryKey {
#[serde(default)]
key: String,
}
fn open_library(id: &str, kind: &str, body: &str) -> Result<Record, Error> {
@@ -196,15 +264,167 @@ fn open_library(id: &str, kind: &str, body: &str) -> Result<Record, Error> {
serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?;
let mut record = blank(id, kind, Source::OpenLibrary);
record.title = book.title;
// An edition record keeps the subtitle in its own field, and a catalogue
// wants the whole title. Google Books is already joined the same way.
record.title = match (book.title, book.subtitle) {
(Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")),
(title, _) => title,
};
record.container = book.publishers.into_iter().next();
// The date is free text — "1999", "March 1999", "1999-03-01".
record.year = book.publish_date.as_deref().and_then(year_in);
record.subjects = book.subjects;
// "/languages/eng" names the language by the last segment of a key.
record.language = book
.languages
.into_iter()
.find_map(|l| l.key.rsplit('/').next().map(str::to_string))
.filter(|l| !l.is_empty());
record.pages = book.number_of_pages;
// The edition's own cover, and the only place a cover id is taken from.
record.cover_id = book
.covers
.into_iter()
.find_map(|id| u64::try_from(id).ok().filter(|id| *id > 0));
Ok(record)
}
#[derive(Deserialize)]
struct SearchReply {
#[serde(default)]
docs: Vec<SearchDoc>,
}
#[derive(Deserialize)]
struct SearchDoc {
#[serde(default)]
title: Option<String>,
#[serde(default)]
subtitle: Option<String>,
#[serde(default)]
author_name: Vec<String>,
#[serde(default)]
first_publish_year: Option<i64>,
#[serde(default)]
publisher: Vec<String>,
#[serde(default)]
language: Vec<String>,
#[serde(default)]
subject: Vec<String>,
#[serde(rename = "number_of_pages_median", default)]
pages: Option<u32>,
}
/// How many of a work's subject headings are worth carrying.
///
/// The index lists every heading every edition has ever been given, which on a
/// well-known book is hundreds and on Hamlet is thousands. A shelf is filed
/// from a handful, and the first are the broadest.
pub(crate) const SUBJECT_LIMIT: usize = 12;
/// A search that matched nothing answers with no documents rather than a
/// status, so an empty reply is the absence and not a failure.
fn open_library_search(id: &str, kind: &str, body: &str) -> Result<Record, Error> {
let reply: SearchReply = serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?;
let Some(doc) = reply.docs.into_iter().next() else {
return Ok(crate::missing(id, kind, Source::OpenLibrarySearch));
};
let mut record = blank(id, kind, Source::OpenLibrarySearch);
record.title = match (doc.title, doc.subtitle) {
(Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")),
(title, _) => title,
};
record.authors = doc.author_name;
record.year = doc.first_publish_year;
record.container = doc.publisher.into_iter().next();
record.language = doc.language.into_iter().next();
// The index lists every subject any edition of the work carries, which
// runs to hundreds on a well-known book. A shelf needs a handful.
record.subjects = doc.subject;
record.subjects.truncate(SUBJECT_LIMIT);
record.pages = doc.pages;
Ok(record)
}
#[derive(Deserialize)]
pub(crate) struct GoogleReply {
#[serde(default)]
pub(crate) items: Vec<GoogleItem>,
}
#[derive(Deserialize)]
pub(crate) struct GoogleItem {
/// Google's own key for the volume.
#[serde(default)]
pub(crate) id: Option<String>,
#[serde(rename = "volumeInfo", default)]
pub(crate) volume_info: GoogleVolume,
}
#[derive(Deserialize, Default)]
pub(crate) struct GoogleVolume {
#[serde(default)]
title: Option<String>,
#[serde(default)]
subtitle: Option<String>,
#[serde(default)]
authors: Vec<String>,
#[serde(rename = "publishedDate", default)]
published_date: Option<String>,
#[serde(default)]
publisher: Option<String>,
#[serde(default)]
description: Option<String>,
#[serde(default)]
categories: Vec<String>,
#[serde(rename = "language", default)]
language: Option<String>,
#[serde(rename = "pageCount", default)]
page_count: Option<u32>,
/// `[{"type": "ISBN_13", "identifier": "…"}]`, alongside library
/// shelfmarks typed `OTHER`.
#[serde(rename = "industryIdentifiers", default)]
pub(crate) identifiers: Vec<GoogleIdentifier>,
}
#[derive(Deserialize)]
pub(crate) struct GoogleIdentifier {
#[serde(rename = "type", default)]
pub(crate) kind: String,
#[serde(default)]
pub(crate) identifier: String,
}
/// A search that matched nothing answers with no items rather than a status,
/// so an empty reply is the absence and not a failure.
fn google_books(id: &str, kind: &str, body: &str) -> Result<Record, Error> {
let reply: GoogleReply = serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?;
let Some(item) = reply.items.into_iter().next() else {
return Ok(crate::missing(id, kind, Source::GoogleBooks));
};
Ok(google_volume(id, kind, item.volume_info))
}
/// One Google Books volume as a record.
pub(crate) fn google_volume(id: &str, kind: &str, volume: GoogleVolume) -> Record {
let mut record = blank(id, kind, Source::GoogleBooks);
record.title = match (volume.title, volume.subtitle) {
(Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")),
(title, _) => title,
};
record.authors = volume.authors;
record.year = volume.published_date.as_deref().and_then(year_in);
record.container = volume.publisher;
record.summary = volume.description.map(|text| collapse(&text));
record.subjects = volume.categories;
record.language = volume.language;
record.pages = volume.page_count;
record
}
/// The first four-digit year in a string.
fn year_in(text: &str) -> Option<i64> {
pub(crate) fn year_in(text: &str) -> Option<i64> {
let bytes: Vec<char> = text.chars().collect();
for window in bytes.windows(4) {
if window.iter().all(char::is_ascii_digit) {
@@ -255,7 +475,10 @@ mod tests {
fn each_kind_goes_to_the_body_that_assigns_it() {
assert_eq!(Source::for_kind("doi"), Some(Source::Crossref));
assert_eq!(Source::for_kind("arxiv"), Some(Source::Arxiv));
assert_eq!(Source::for_kind("isbn"), Some(Source::OpenLibrary));
// The index, not the edition record: it answers with the whole of
// what a catalogue wants in one request, where the edition record
// takes up to twenty-eight seconds and names its authors by key.
assert_eq!(Source::for_kind("isbn"), Some(Source::OpenLibrarySearch));
// No register answers about an ISSN without a key.
assert_eq!(Source::for_kind("issn"), None);
}
@@ -326,6 +549,56 @@ mod tests {
}
}
#[test]
fn the_cover_id_a_record_names_is_kept_for_the_cover_lookup() {
// A deleted cover stays in an edition's list as -1, ahead of the
// ones still there.
let record = open_library("1", "isbn", r#"{"title":"A","covers":[-1,1162784,99]}"#)
.expect("a record");
assert_eq!(record.cover_id, Some(1162784));
let record = open_library("1", "isbn", r#"{"title":"A"}"#).expect("a record");
assert_eq!(record.cover_id, None);
}
#[test]
fn the_cover_id_is_the_editions_and_not_the_works() {
// ISBN 9780140447934 today: the index (asked first) names the work's
// cover; the edition record has none. Merged as `ask::lookup` merges.
let mut kept = open_library_search(
"9780140447934",
"isbn",
r#"{"docs":[{"title":"War and Peace","cover_i":12621906}]}"#,
)
.expect("a record");
let edition = open_library("9780140447934", "isbn", r#"{"title":"War and Peace"}"#)
.expect("a record");
kept.fill_from(edition);
// Cover 12621906 is OL34941467M, Standard Ebooks' Maude translation.
assert_eq!(kept.cover_id, None);
}
#[test]
fn an_edition_with_no_covers_gives_no_cover_id() {
let index = r#"{"docs":[{"title":"War and Peace","cover_i":12621906}]}"#;
// `lookup` often stops at the index, before the edition is asked.
let alone = open_library_search("9780140447934", "isbn", index).expect("a record");
assert_eq!(alone.cover_id, None);
// An edition whose only cover was deleted, and one with an empty list.
for edition in [
r#"{"title":"War and Peace","covers":[-1]}"#,
r#"{"title":"War and Peace","covers":[]}"#,
] {
let mut kept = open_library_search("9780140447934", "isbn", index).expect("a record");
kept.fill_from(open_library("9780140447934", "isbn", edition).expect("a record"));
assert_eq!(kept.cover_id, None, "{edition}");
}
}
#[test]
fn the_index_is_not_asked_for_the_works_cover() {
assert!(!Source::OpenLibrarySearch.url("1").contains("cover_i"));
}
#[test]
fn an_answer_that_is_not_what_it_should_be_says_so() {
assert!(matches!(
crates/bindings/Cargo.toml
+7 -1
View File
@@ -30,6 +30,7 @@ crate-type = ["cdylib", "rlib"]
[dependencies]
booksniff.workspace = true
bibclean.workspace = true
colophon.workspace = true
chunkmd.workspace = true
citefind.workspace = true
corpus-core.workspace = true
@@ -47,6 +48,11 @@ textkit.workspace = true
pyo3 = { workspace = true, optional = true }
# Only the Python build gets this. bibfetch is the one tool here that opens a
# socket, and its HTTP client does not build for wasm32 — so a dependency that
# is unconditional would take the JavaScript package down with it.
bibfetch = { workspace = true, optional = true }
# The JavaScript build. Only ever compiled for a wasm32 target; on any other
# the module is compiled out and these sit unused.
wasm-bindgen = { workspace = true, optional = true }
@@ -56,7 +62,7 @@ serde-wasm-bindgen = { workspace = true, optional = true }
# Only maturin turns this on. pyo3's extension-module feature leaves the
# Python symbols to be resolved by the interpreter at load time, so an
# ordinary `cargo build` would fail to link if it were on by default.
python = ["dep:pyo3"]
python = ["dep:pyo3", "dep:bibfetch"]
# The C ABI. Off by default so an ordinary build does not also export a
# hundred symbols nobody asked for.
capi = []
crates/bindings/python/catena/__init__.py
+16
View File
@@ -16,10 +16,14 @@ from ._catena import (
chunk_markdown,
citation_prompt,
clean_record,
colophon_prompt,
clean_underscores,
compose_subjects,
count_tokens,
cue_windows,
Limited,
fetch_cover,
fetch_identifier,
find_citations,
fuse_hits,
identify,
@@ -29,16 +33,20 @@ from ._catena import (
looks_like_footnote_citations,
looks_like_person,
merge_citation_reply,
merge_colophon,
needs_subject,
pack_cues,
parse_identifier,
parse_subtitles,
read_colophon,
record_problems,
record_severity,
same_work,
scrub,
search_title,
segment_cues,
series_of,
shelve,
split_collection,
suggest_author,
title_case,
@@ -48,14 +56,18 @@ from ._catena import (
__all__ = [
"__version__",
"Limited",
"canonical_subject",
"chunk_markdown",
"citation_prompt",
"clean_record",
"colophon_prompt",
"clean_underscores",
"compose_subjects",
"count_tokens",
"cue_windows",
"fetch_cover",
"fetch_identifier",
"find_citations",
"fuse_hits",
"identify",
@@ -65,16 +77,20 @@ __all__ = [
"looks_like_footnote_citations",
"looks_like_person",
"merge_citation_reply",
"merge_colophon",
"needs_subject",
"pack_cues",
"parse_identifier",
"parse_subtitles",
"read_colophon",
"record_problems",
"record_severity",
"same_work",
"scrub",
"search_title",
"segment_cues",
"series_of",
"shelve",
"split_collection",
"suggest_author",
"title_case",
crates/bindings/python/catena/__init__.pyi
+80
View File
@@ -75,3 +75,83 @@ def valid_isbn(text: str) -> bool: ...
def isbn_from_text(text: str) -> str | None: ...
def isbn_from_filename(name: str) -> str | None: ...
def same_work(ours: str, theirs: str) -> bool: ...
def read_colophon(
text: str, pages: int = ..., shelf: str | None = ...
) -> dict[str, Any]:
"""What a book prints about itself in its opening pages.
`shelf` is a closed list of headings to file the printed subjects onto,
one per line, optionally `Heading = term, term`.
"""
...
def shelve(shelf: str, subjects: list[str]) -> tuple[list[str], list[str]]:
"""File subject headings onto a shelf. Returns (shelved, unshelved)."""
...
def fetch_identifier(
id: str,
kind: str = ...,
polite: str | None = ...,
timeout: int = ...,
attempts: int = ...,
) -> dict[str, Any] | None:
"""Ask the register that assigns an identifier what it knows about one.
Blocks on the network. `None` when no register answers about that kind.
"""
...
class Limited(Exception):
"""A register refused because it has been asked too often; ask again later."""
def fetch_cover(
isbn: str,
cover_id: int | None = ...,
size: str = ...,
polite: str | None = ...,
timeout: int = ...,
deadline: int = ...,
) -> dict[str, Any] | None:
"""The cover Open Library has for a book, by ISBN, then by cover id.
A dict with `bytes`, `media_type`, `extension`, `source` and `url`, or
`None` when there is none. Raises `Limited` when Open Library refused for
being asked too often. Blocks on the network.
"""
...
def search_title(
title: str,
author: str | None = ...,
limit: int = ...,
polite: str | None = ...,
timeout: int = ...,
attempts: int = ...,
) -> list[dict[str, Any]]:
"""The editions the registers list for a title and author, in their order.
Records shaped like `fetch_identifier`'s. Use `same_work` to decide which
one is the book. Blocks on the network.
"""
...
def colophon_prompt(
record: dict[str, Any], text: str, pages: int = ...
) -> str | None:
"""The question to put to a model about what the pages did not settle.
None when the pages settled everything. Nothing is sent anywhere.
"""
...
def merge_colophon(
record: dict[str, Any], reply: str
) -> tuple[dict[str, Any], int]:
"""A model's reply folded in. Returns (record, fields_settled).
Only fills what the pages left unsettled; a model never overwrites what a
book printed about itself.
"""
...
crates/bindings/src/python.rs
+223 -1
View File
@@ -10,7 +10,7 @@
use pyo3::exceptions::PyValueError;
use pyo3::prelude::*;
use pyo3::types::{PyDict, PyList};
use pyo3::types::{PyBytes, PyDict, PyList};
use crate::{ChunkBudgets, CueBudgets};
@@ -431,5 +431,227 @@ fn _catena(module: &Bound<'_, PyModule>) -> PyResult<()> {
module.add_function(wrap_pyfunction!(merge_citation_reply, module)?)?;
module.add_function(wrap_pyfunction!(series_of, module)?)?;
module.add_function(wrap_pyfunction!(parse_identifier, module)?)?;
module.add_function(wrap_pyfunction!(read_colophon, module)?)?;
module.add_function(wrap_pyfunction!(shelve, module)?)?;
module.add_function(wrap_pyfunction!(fetch_identifier, module)?)?;
module.add_function(wrap_pyfunction!(fetch_cover, module)?)?;
module.add_function(wrap_pyfunction!(search_title, module)?)?;
module.add("Limited", module.py().get_type::<Limited>())?;
module.add_function(wrap_pyfunction!(colophon_prompt, module)?)?;
module.add_function(wrap_pyfunction!(merge_colophon, module)?)?;
Ok(())
}
// ───────────────────── what a book says about itself ──────────────────────
/// Read what a book prints about itself in its opening pages.
///
/// `shelf` is the closed list of headings to file the printed subjects onto,
/// in the format `colophon --shelf` takes: one heading per line, optionally
/// `Heading = term, term`. Without it the subjects come back as the book
/// printed them and nothing is filed.
#[pyfunction]
#[pyo3(signature = (text, pages = colophon::PAGES, shelf = None))]
fn read_colophon<'py>(
py: Python<'py>,
text: String,
pages: usize,
shelf: Option<String>,
) -> PyResult<Bound<'py, PyAny>> {
if pages == 0 {
return Err(PyValueError::new_err("pages = 0 would read nothing"));
}
// Reading a book's front matter is several milliseconds of pure
// computation on a few hundred kilobytes, and holding the interpreter for
// it makes `asyncio.to_thread` a lie: a caller reading a library several
// books at a time got no overlap at all until this let go.
let printed = py.detach(move || {
let shelf = shelf.as_deref().map(colophon::shelf::Shelf::parse);
colophon::read_onto(&text, pages, shelf.as_ref())
});
to_py(py, &printed)
}
/// File subject headings onto a shelf without reading a book.
///
/// For the headings a catalogue already holds: the same matching the printed
/// ones get, so a record's existing subjects and its book's agree about what
/// counts as a heading. Returns `(shelved, unshelved)`.
#[pyfunction]
fn shelve(shelf: &str, subjects: Vec<String>) -> (Vec<String>, Vec<String>) {
let filed = colophon::shelf::Shelf::parse(shelf).file(&subjects);
(filed.on, filed.unfiled)
}
// ──────────────────────────── the registers ───────────────────────────────
/// Ask the body that assigns an identifier what it knows about one.
///
/// Crossref for a DOI, the arXiv for a preprint, OpenLibrary and Google Books
/// for an ISBN, asked in turn until one gives a complete answer. This is the
/// only function here that touches the network, and it blocks: call it from a
/// thread, or with `asyncio.to_thread`.
///
/// Returns the record, or `None` when no register here answers about that kind
/// of identifier. Raises when every register that should answer was
/// unreachable, because that is a different thing from the book not existing.
#[pyfunction]
#[pyo3(signature = (id, kind = "isbn", polite = None, timeout = 20, attempts = 3))]
fn fetch_identifier<'py>(
py: Python<'py>,
id: &str,
kind: &str,
polite: Option<String>,
timeout: u64,
attempts: u32,
) -> PyResult<Option<Bound<'py, PyAny>>> {
let ask = bibfetch::ask::Ask {
polite,
timeout: std::time::Duration::from_secs(timeout.max(1)),
attempts: attempts.max(1),
};
let agent = ask.agent();
// The GIL is not needed while waiting on a socket, and holding it there
// would stop every other thread in the interpreter for the whole request.
let found = py
.detach(|| bibfetch::ask::lookup(&agent, id, kind, ask.attempts))
.map_err(register_error)?;
match found {
Some(record) => Ok(Some(to_py(py, &record)?)),
None => Ok(None),
}
}
pyo3::create_exception!(
_catena,
Limited,
pyo3::exceptions::PyException,
"A register refused because it has been asked too often; ask again later."
);
/// A register's error as the Python exception that says what to do about it.
fn register_error(error: bibfetch::Error) -> PyErr {
match error {
bibfetch::Error::Limited(why) => Limited::new_err(why),
other => PyValueError::new_err(other.to_string()),
}
}
/// The cover Open Library has for a book, by its ISBN, then by the cover id
/// a record named.
///
/// Returns a dict with `bytes`, `media_type` (`image/jpeg` or `image/png`),
/// `extension`, `source` and `url`, or `None` when there is no cover. Covers
/// asked by ISBN are kept three seconds apart across the process, and the
/// whole fetch stops after `deadline` seconds. Raises `catena.Limited` when
/// Open Library refused for being asked too often, and `ValueError` when the
/// ISBN is not one or the store could not be reached. Blocks on the network.
#[pyfunction]
#[pyo3(signature = (isbn, cover_id = None, size = "large", polite = None, timeout = 20, deadline = 120))]
fn fetch_cover<'py>(
py: Python<'py>,
isbn: &str,
cover_id: Option<u64>,
size: &str,
polite: Option<String>,
timeout: u64,
deadline: u64,
) -> PyResult<Option<Bound<'py, PyDict>>> {
let size = match size {
"small" => bibfetch::CoverSize::Small,
"medium" => bibfetch::CoverSize::Medium,
"large" => bibfetch::CoverSize::Large,
other => {
return Err(PyValueError::new_err(format!(
"size is small, medium or large, not {other:?}"
)));
}
};
let agent = bibfetch::ask::Ask {
polite,
timeout: std::time::Duration::from_secs(timeout.max(1)),
..Default::default()
}
.agent();
let covers = bibfetch::cover::Covers {
deadline: std::time::Duration::from_secs(deadline.max(1)),
..Default::default()
};
let found = py
.detach(|| covers.fetch(&agent, isbn, cover_id, size))
.map_err(register_error)?;
let Some(cover) = found else {
return Ok(None);
};
let out = PyDict::new(py);
out.set_item("bytes", PyBytes::new(py, &cover.bytes))?;
out.set_item("media_type", cover.media_type)?;
out.set_item("extension", cover.extension())?;
out.set_item("source", cover.source)?;
out.set_item("url", cover.url)?;
Ok(Some(out))
}
/// The editions the registers list for a title, and an author if one is known.
///
/// Open Library is asked first and Google Books only when it has nothing.
/// The candidates come back in the register's own order, as records shaped
/// like `fetch_identifier`'s; nothing here decides which is the book, which is
/// what `same_work` is for. Blocks on the network.
#[pyfunction]
#[pyo3(signature = (title, author = None, limit = 10, polite = None, timeout = 20, attempts = 3))]
fn search_title<'py>(
py: Python<'py>,
title: &str,
author: Option<&str>,
limit: usize,
polite: Option<String>,
timeout: u64,
attempts: u32,
) -> PyResult<Bound<'py, PyList>> {
let ask = bibfetch::ask::Ask {
polite,
timeout: std::time::Duration::from_secs(timeout.max(1)),
attempts: attempts.max(1),
};
let agent = ask.agent();
let found = py
.detach(|| bibfetch::search(&agent, title, author, limit, ask.attempts))
.map_err(register_error)?;
list_to_py(py, &found)
}
/// The question to put to a model about the fields a book's pages did not
/// settle, or `None` when they settled everything.
///
/// Nothing is sent anywhere: this composes the text and hands it back, so the
/// caller decides which model, on whose hardware, at what cost. The reply goes
/// to `merge_colophon`.
#[pyfunction]
#[pyo3(signature = (record, text, pages = colophon::PAGES))]
fn colophon_prompt(
record: &Bound<'_, PyAny>,
text: &str,
pages: usize,
) -> PyResult<Option<String>> {
let printed: colophon::Printed = from_py(record)?;
Ok(colophon::prompt::compose(&printed, text, pages))
}
/// A model's reply folded into the record the pages gave.
///
/// Only the fields the pages left unsettled are filled: a model cannot
/// overwrite something a book printed about itself, which is the whole reason
/// the pages are read first. Returns the completed record and how many fields
/// the reply actually settled.
#[pyfunction]
fn merge_colophon<'py>(
py: Python<'py>,
record: &Bound<'py, PyAny>,
reply: &str,
) -> PyResult<(Bound<'py, PyAny>, usize)> {
let mut printed: colophon::Printed = from_py(record)?;
let filled = colophon::prompt::merge(&mut printed, reply)
.map_err(|e| PyValueError::new_err(e.to_string()))?;
Ok((to_py(py, &printed)?, filled))
}
crates/bookmeta/README.md
+71 -1
View File
@@ -84,6 +84,7 @@ bookmeta library/*.epub \
|---|---|
| EPUB | The package document named by `META-INF/container.xml` — Dublin Core, plus the `<meta>` entries Calibre adds for a series |
| PDF | The trailer's information dictionary |
| MOBI, AZW, AZW3 | The MOBI header in the first record, and the EXTH records after it |
An EPUB whose container is malformed falls back to the first `.opf` in the
archive, because a converter that wrote a bad container often still wrote a
@@ -94,6 +95,75 @@ both become `isbn`, a UUID becomes `uuid`, anything else `other`. PDF dates
are reduced to the day where they parse and kept verbatim where they do not,
because a date nobody can read is still evidence.
A Kindle book's title is EXTH 503 where it is there, then the MOBI header's
full name, then the Palm database name (cut to 31 bytes, with a note). Authors
(100) and subjects (105) are every record of their kind, in order; publisher
(101), description (103), date (106) and language (524) the first. EXTH 104 is
an `isbn` when it is shaped like one and `other` when not. 113 is filed by
its shape: a `uuid` when it is one (calibre writes its book id there), an
`asin` when it is ten letters and digits, and `other` when it is neither. A
contributor (108) is not an author, the same as an EPUB's
`dc:contributor`, and is not reported. With no EXTH language, the language
comes from the header's locale. Strings are decoded as the header says,
UTF-8 or Windows-1252. A book whose text is encrypted still has its header
read, and carries the note `drm`.
Every offset in a Kindle book is checked before it is followed: headers that
are cut short or point outside the file give an empty record with a note, and
an EXTH block that breaks off keeps the records before the break.
## Covers
`Metadata::cover` is the cover image the file carries, with its media type
read off its first bytes: JPEG, PNG, GIF, WebP or BMP. It is for library
callers and is not part of the JSON record, which is a line of text.
A Kindle book's cover is EXTH 201, or the thumbnail, 202, when no cover is
named. With neither, it is the record at the header's first image index, as
calibre does, and only that record: when it is not a JPEG, PNG or GIF there is
no cover, because the records after it can be illustrations, and kindlegen
sometimes puts a font there first.
An EPUB's is the first of these that reaches an image:
1. The manifest item whose `properties` include `cover-image` (EPUB 3).
2. The item `<meta name="cover" content="…">` names (EPUB 2). Some
converters write the image's address there rather than its id, and that is
followed too.
3. The guide's `<reference type="cover">`, a page, and the `<img src>` or
SVG `<image href>` on it. Where the page has several, the one whose file
name, `alt`, `id` or `class` says `cover` is taken; where none does, the
first is, with a note that there were others. A guide that points straight
at an image is taken at its word.
4. Any image in the manifest whose id or file name says `cover`, one named
`cover` outright first and the rest in manifest order.
A name says `cover` when it has it as a word of its own, in any case. Words
break at anything not a letter or digit and where a lower case letter meets a
capital, so `CoverImage`, `Book_COVER` and `cover-thumb` say it, and
`discover-ch1` and `recovery-map` do not.
A route that names something unusable — not in the archive, not an image —
leaves a note, and the next route is asked. Addresses are resolved against the
document they are written in, with their fragment off and their
percent-escapes undone, and one that climbs out of the archive or names a
scheme (`http:`, `data:`) is not followed.
What the bytes say beats what the manifest says: a JPEG listed as
`image/png` comes back as `image/jpeg`, with a note. A cover larger than
`bookmeta::MAX_COVER` (16 MiB) is left out with a note, in either format, and
an EPUB's is never decompressed further than that, whatever size the archive
claims for it. Nor are more than eight members read looking for one,
however many candidates the manifest lists: the search stops there, with a
note.
`Cover::guessed` says the file left the choice open: a Kindle book's first
image record, or an EPUB image picked because its name says `cover`. Either
can be an illustration from inside the book, so a caller that can get a cover
another way, from a register by ISBN, should prefer that one. A cover the file
names is never a guess. `Cover::extension` gives the usual file extension for
its media type.
## Asking a model
Some files say nothing useful. `--prompt` writes the question and **sends
@@ -147,7 +217,7 @@ if let Some(prompt) = bookmeta::prompt::compose(&meta, Some(opening)) {
}
```
`bookmeta::epub` and `bookmeta::pdf` read one format each, and
`bookmeta::epub`, `bookmeta::pdf` and `bookmeta::mobi` read one format each, and
`Metadata::missing` says which fields a record still wants.
## See also
crates/bookmeta/src/epub.rs
+1007 -4
View File
File diff suppressed because it is too large. Load diff
crates/bookmeta/src/lib.rs
+81
View File
@@ -14,6 +14,7 @@
//! [`bibfetch`]: https://git.pumpin.online/funkpower/catena
pub mod epub;
pub mod mobi;
pub mod pdf;
pub mod prompt;
@@ -30,6 +31,8 @@ pub enum Source {
Opf,
/// A PDF's information dictionary.
PdfInfo,
/// A Kindle book's MOBI header and the EXTH records after it.
Mobi,
/// Nothing was found to read.
None,
}
@@ -40,6 +43,7 @@ impl Source {
match self {
Source::Opf => "opf",
Source::PdfInfo => "pdf_info",
Source::Mobi => "mobi",
Source::None => "none",
}
}
@@ -64,6 +68,9 @@ pub struct Metadata {
/// Who published it.
#[serde(skip_serializing_if = "Option::is_none")]
pub publisher: Option<String>,
/// The publisher's description of the book, as written: often HTML.
#[serde(skip_serializing_if = "Option::is_none")]
pub description: Option<String>,
/// The publication date, in whatever form the file wrote it.
#[serde(skip_serializing_if = "Option::is_none")]
pub date: Option<String>,
@@ -86,6 +93,76 @@ pub struct Metadata {
/// What was worth saying about the reading.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub notes: Vec<String>,
/// The cover image the file carries.
///
/// Not part of the record: a record is a line of text and this is an
/// image. A library caller takes it from here.
#[serde(skip)]
#[schemars(skip)]
pub cover: Option<Cover>,
}
/// A cover image, as the file stores it.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Cover {
/// What kind of image it is, read off its first bytes: `image/jpeg`,
/// `image/png`, `image/gif`, `image/webp` or `image/bmp`.
pub media_type: String,
/// The image itself, never more than [`MAX_COVER`] bytes.
pub bytes: Vec<u8>,
/// Whether the file left the choice open. A MOBI that names no cover, or
/// an EPUB whose only cover is an image with "cover" in its name, has one
/// picked by position or by name, and that pick can be an illustration
/// from inside the book. A caller with a better source should prefer it.
pub guessed: bool,
}
impl Cover {
/// The usual file extension for the image, without the dot.
pub fn extension(&self) -> &'static str {
match self.media_type.as_str() {
"image/png" => "png",
"image/gif" => "gif",
"image/webp" => "webp",
"image/bmp" => "bmp",
_ => "jpg",
}
}
}
/// How large a cover is worth keeping.
///
/// A real one is a few hundred kilobytes. An EPUB's cover has to be
/// decompressed to be read, and this is as far as that goes: a larger one is
/// left out with a note, in every format, so a hostile file cannot make a
/// reader allocate whatever it likes.
pub const MAX_COVER: usize = 16 * 1024 * 1024;
/// A size, as a note says it.
pub(crate) fn size(bytes: usize) -> String {
const MIB: usize = 1024 * 1024;
if bytes >= MIB && bytes % MIB == 0 {
format!("{} MiB", bytes / MIB)
} else {
format!("{bytes} bytes")
}
}
/// What kind of image these bytes are, by their magic.
pub(crate) fn image_type(data: &[u8]) -> Option<&'static str> {
if data.starts_with(&[0xff, 0xd8, 0xff]) {
Some("image/jpeg")
} else if data.starts_with(b"\x89PNG\r\n\x1a\n") {
Some("image/png")
} else if data.starts_with(b"GIF87a") || data.starts_with(b"GIF89a") {
Some("image/gif")
} else if data.starts_with(b"RIFF") && data.get(8..12) == Some(b"WEBP") {
Some("image/webp")
} else if data.starts_with(b"BM") && data.len() > 14 {
Some("image/bmp")
} else {
None
}
}
fn version() -> u32 {
@@ -143,6 +220,7 @@ impl Metadata {
self.title.is_none()
&& self.authors.is_empty()
&& self.publisher.is_none()
&& self.description.is_none()
&& self.date.is_none()
&& self.language.is_none()
&& self.identifiers.is_empty()
@@ -171,6 +249,9 @@ pub fn from_bytes(path: &str, bytes: &[u8]) -> Metadata {
if bytes.starts_with(b"%PDF") {
return pdf::read(path, bytes);
}
if mobi::is_mobi(bytes) {
return mobi::read(path, bytes);
}
let mut meta = Metadata::empty(path, Source::None);
meta.note("not a format this reads metadata out of");
meta
crates/bookmeta/src/mobi.rs
+870
View File
@@ -0,0 +1,870 @@
//! A Kindle book's MOBI header and the EXTH records after it.
//!
//! MOBI, AZW and AZW3 files are Palm databases. The first record opens with a
//! PalmDOC header and a MOBI header, which gives the book's full name and says
//! how its strings are encoded. When a flag in that header is set, an EXTH
//! block follows with everything else: authors, publisher, ISBN, subjects, and
//! which image record is the cover. AZW3 (KF8) keeps the same layout in its
//! first record, so one reader covers all three.
//!
//! Every offset in the file is checked before it is followed. A file whose
//! headers are cut short or written wrongly gives an empty record with a note;
//! an EXTH block that breaks off keeps the records before the break. Neither
//! panics or reads past the end.
use crate::epub::{digits_of, is_isbn, is_uuid};
use crate::{Cover, MAX_COVER, Metadata, Source, image_type};
/// Where a Palm database says what it holds.
const TYPE_AT: usize = 60;
/// What a Kindle book says there.
const BOOK_TYPE: &[u8] = b"BOOKMOBI";
/// The Palm database's own name: 32 bytes, NUL-padded.
const NAME_LEN: usize = 32;
/// Where the number of records is written.
const RECORD_COUNT_AT: usize = 76;
/// Where the record list starts: eight bytes per record, offset first.
const RECORD_LIST_AT: usize = 78;
/// Where the PalmDOC header says whether the text is encrypted.
const ENCRYPTION_AT: usize = 12;
/// Where the MOBI header begins inside the first record.
const MOBI_AT: usize = 16;
/// The MOBI header's own magic.
const MOBI_MAGIC: &[u8] = b"MOBI";
// Fields of the MOBI header, as offsets from its start. Each is only there
// when the header's declared length reaches past it.
const HEADER_LENGTH: usize = 4;
const TEXT_ENCODING: usize = 12;
const FULL_NAME_OFFSET: usize = 68;
const FULL_NAME_LENGTH: usize = 72;
const LOCALE: usize = 76;
const FIRST_IMAGE: usize = 92;
const EXTH_FLAGS: usize = 112;
/// The flag that says an EXTH block follows the MOBI header.
const HAS_EXTH: u32 = 0x40;
/// The EXTH block's magic.
const EXTH_MAGIC: &[u8] = b"EXTH";
/// The text encodings a MOBI header names.
const UTF_8: u32 = 65001;
const WINDOWS_1252: u32 = 1252;
/// What an index or offset says when it points at nothing.
const NONE: u32 = 0xffff_ffff;
// EXTH record types.
const AUTHOR: u32 = 100;
const PUBLISHER: u32 = 101;
const DESCRIPTION: u32 = 103;
const ISBN: u32 = 104;
const SUBJECT: u32 = 105;
const PUBLISHED: u32 = 106;
const ASIN: u32 = 113;
const COVER_OFFSET: u32 = 201;
const THUMB_OFFSET: u32 = 202;
const UPDATED_TITLE: u32 = 503;
const LANGUAGE: u32 = 524;
/// Whether these bytes are a Kindle book this module can read.
pub fn is_mobi(bytes: &[u8]) -> bool {
bytes.get(TYPE_AT..TYPE_AT + BOOK_TYPE.len()) == Some(BOOK_TYPE)
}
/// Read what a MOBI, AZW or AZW3 file says about itself.
pub fn read(path: &str, bytes: &[u8]) -> Metadata {
let mut meta = Metadata::empty(path, Source::Mobi);
if let Err(why) = fill(&mut meta, bytes) {
let mut empty = Metadata::empty(path, Source::None);
empty.note(why);
return empty;
}
if meta.is_empty() {
meta.note("a MOBI header with nothing filled in");
}
meta
}
/// The strings an EXTH block carries, before they are placed in a record.
#[derive(Default)]
struct Exth {
authors: Vec<String>,
publisher: Option<String>,
description: Option<String>,
isbn: Option<String>,
subjects: Vec<String>,
published: Option<String>,
asin: Option<String>,
cover: Option<u32>,
thumb: Option<u32>,
title: Option<String>,
language: Option<String>,
}
fn fill(meta: &mut Metadata, bytes: &[u8]) -> Result<(), String> {
if !is_mobi(bytes) {
return Err("not a Kindle book: the Palm database type is not BOOKMOBI".into());
}
let records = records(bytes)?;
let first = record(bytes, &records, 0).ok_or("the first record lies outside the file")?;
if first.get(MOBI_AT..MOBI_AT + MOBI_MAGIC.len()) != Some(MOBI_MAGIC) {
return Err("no MOBI header in the first record".into());
}
let header_length =
u32_at(first, MOBI_AT + HEADER_LENGTH).ok_or("the MOBI header is cut short")? as usize;
let header = Header {
record: first,
length: header_length,
};
let encoding = header.field(TEXT_ENCODING).unwrap_or(WINDOWS_1252);
if encoding != UTF_8 && encoding != WINDOWS_1252 {
meta.note(format!(
"text encoding {encoding}, which is neither UTF-8 nor Windows-1252; read as Windows-1252"
));
}
let decode = |raw: &[u8]| text(raw, encoding);
if u16_at(first, ENCRYPTION_AT).is_some_and(|kind| kind != 0) {
meta.note("drm");
}
let exth = if header
.field(EXTH_FLAGS)
.is_some_and(|flags| flags & HAS_EXTH != 0)
{
let at = MOBI_AT
.checked_add(header_length)
.ok_or("the MOBI header length is out of range")?;
exth(first, at, &decode, meta)
} else {
Exth::default()
};
let full_name = header
.field(FULL_NAME_OFFSET)
.zip(header.field(FULL_NAME_LENGTH))
.and_then(|(at, len)| slice(first, at as usize, len as usize))
.and_then(decode);
meta.title = match (exth.title, full_name) {
(Some(title), _) | (None, Some(title)) => Some(title),
(None, None) => {
let name = palm_name(bytes);
if name.is_some() {
meta.note("title from the Palm database name, which is cut to 31 bytes");
}
name
}
};
meta.authors = exth.authors;
meta.publisher = exth.publisher;
meta.description = exth.description;
meta.date = exth.published;
meta.subjects = exth.subjects;
meta.language = exth.language.or_else(|| {
header
.field(LOCALE)
.and_then(|locale| language_of(locale).map(str::to_string))
});
if let Some(isbn) = exth.isbn {
if is_isbn(&isbn) {
meta.identifiers.insert("isbn".into(), digits_of(&isbn));
} else {
meta.identifiers.insert("other".into(), isbn);
}
}
// EXTH 113 holds an ASIN when the book was made for Amazon, and calibre's
// own book UUID when it was converted without one.
if let Some(value) = exth.asin {
let lower = value.to_lowercase();
if is_uuid(&lower) {
meta.identifiers.insert("uuid".into(), lower);
} else if is_asin(&value) {
meta.identifiers.insert("asin".into(), value);
} else {
meta.identifiers.entry("other".into()).or_insert(value);
}
}
let first_image = header.field(FIRST_IMAGE);
if let Some(offset) = exth.cover.or(exth.thumb) {
meta.cover = cover(bytes, &records, first_image, offset, meta);
} else {
meta.cover = first_image_record(bytes, &records, first_image);
}
Ok(())
}
/// The MOBI header, for reading fields that may or may not be in it.
struct Header<'a> {
record: &'a [u8],
length: usize,
}
impl Header<'_> {
/// A field at `at` bytes into the header, when the header reaches it.
fn field(&self, at: usize) -> Option<u32> {
if at.checked_add(4)? > self.length {
return None;
}
u32_at(self.record, MOBI_AT.checked_add(at)?)
}
}
/// Where each record starts, as the record list says.
fn records(bytes: &[u8]) -> Result<Vec<usize>, String> {
let count = u16_at(bytes, RECORD_COUNT_AT).ok_or("the Palm database header is cut short")?;
if count == 0 {
return Err("a Palm database with no records".into());
}
let mut out = Vec::with_capacity(count as usize);
for at in 0..count as usize {
let entry = RECORD_LIST_AT + at * 8;
let Some(offset) = u32_at(bytes, entry) else {
return Err("the record list runs past the end of the file".into());
};
out.push(offset as usize);
}
Ok(out)
}
/// One record's bytes: from its offset to the next record's, or to the end.
fn record<'a>(bytes: &'a [u8], records: &[usize], index: usize) -> Option<&'a [u8]> {
let start = *records.get(index)?;
let end = records.get(index + 1).copied().unwrap_or(bytes.len());
bytes.get(start..end)
}
/// Read the EXTH block at `at` inside the first record.
///
/// A block that ends early keeps what was read before the break, with a note:
/// the records already read are as good as they would have been.
fn exth(
first: &[u8],
at: usize,
decode: &dyn Fn(&[u8]) -> Option<String>,
meta: &mut Metadata,
) -> Exth {
let mut out = Exth::default();
if at >= first.len() || first.get(at..at + EXTH_MAGIC.len()) != Some(EXTH_MAGIC) {
meta.note("the header says an EXTH block follows, but none does");
return out;
}
let (Some(length), Some(count)) = (u32_at(first, at + 4), u32_at(first, at + 8)) else {
meta.note("the EXTH block ends early");
return out;
};
let end = at.saturating_add(length as usize).min(first.len());
let mut cursor = at + 12;
for _ in 0..count {
let (Some(kind), Some(size)) = (u32_at(first, cursor), u32_at(first, cursor + 4)) else {
meta.note("the EXTH block ends early");
break;
};
let size = size as usize;
let Some(data) = size
.checked_sub(8)
.and_then(|len| slice(first, cursor + 8, len))
.filter(|_| cursor.checked_add(size).is_some_and(|stop| stop <= end))
else {
meta.note("the EXTH block ends early");
break;
};
cursor += size;
match kind {
AUTHOR => out.authors.extend(decode(data)),
SUBJECT => out.subjects.extend(decode(data)),
PUBLISHER => set(&mut out.publisher, decode(data)),
DESCRIPTION => set(&mut out.description, decode(data)),
ISBN => set(&mut out.isbn, decode(data)),
PUBLISHED => set(&mut out.published, decode(data)),
ASIN => set(&mut out.asin, decode(data)),
UPDATED_TITLE => set(&mut out.title, decode(data)),
LANGUAGE => set(&mut out.language, decode(data)),
COVER_OFFSET => set(&mut out.cover, number(data)),
THUMB_OFFSET => set(&mut out.thumb, number(data)),
_ => {}
}
}
out
}
/// Keep the first value a field is given: a later duplicate is usually a
/// converter writing the same thing twice, and the first is what it meant.
fn set<T>(slot: &mut Option<T>, value: Option<T>) {
if slot.is_none() {
*slot = value;
}
}
/// A four-byte number record that points somewhere.
fn number(data: &[u8]) -> Option<u32> {
let value = u32::from_be_bytes(data.try_into().ok()?);
(value != NONE).then_some(value)
}
/// The cover image, from the record the header and the EXTH offset name.
fn cover(
bytes: &[u8],
records: &[usize],
first_image: Option<u32>,
offset: u32,
meta: &mut Metadata,
) -> Option<Cover> {
let Some(first_image) = first_image.filter(|&index| index != NONE) else {
meta.note("a cover is named, but the header has no image records");
return None;
};
let Some(data) = first_image
.checked_add(offset)
.and_then(|index| record(bytes, records, index as usize))
else {
meta.note("the cover points past the last record");
return None;
};
let Some(media_type) = image_type(data) else {
meta.note("the cover record is not an image");
return None;
};
if data.len() > MAX_COVER {
meta.note(format!(
"the cover record is larger than {}",
crate::size(MAX_COVER)
));
return None;
}
Some(Cover {
media_type: media_type.to_string(),
bytes: data.to_vec(),
guessed: false,
})
}
/// The cover when the file names none: calibre takes the record at the first
/// image index, and only that one.
///
/// When that record is not a JPEG, PNG or GIF there is no cover. The search
/// does not walk on to the next image, because a later record may be an inline
/// illustration, and kindlegen puts a font or a RESC record first.
fn first_image_record(bytes: &[u8], records: &[usize], first_image: Option<u32>) -> Option<Cover> {
let first = first_image.filter(|&index| index != NONE)?;
let data = record(bytes, records, first as usize)?;
if data.len() > MAX_COVER {
return None;
}
let media_type = image_type(data).filter(|kind| *kind != "image/bmp")?;
Some(Cover {
media_type: media_type.to_string(),
bytes: data.to_vec(),
guessed: true,
})
}
/// An ASIN is ten letters and digits; an ISBN-10 is one too.
fn is_asin(value: &str) -> bool {
value.len() == 10 && value.chars().all(|c| c.is_ascii_alphanumeric())
}
/// The Palm database's name, the last place a title can come from.
///
/// Converters write the title there with spaces made underscores; it is
/// reported as written.
fn palm_name(bytes: &[u8]) -> Option<String> {
let raw = bytes.get(..NAME_LEN)?;
let raw = raw.split(|&byte| byte == 0).next().unwrap_or(raw);
text(raw, WINDOWS_1252)
}
/// A string in the book's declared encoding, trimmed; empty is none.
fn text(raw: &[u8], encoding: u32) -> Option<String> {
let decoded: String = match (encoding, std::str::from_utf8(raw)) {
(UTF_8, Ok(text)) => text.to_string(),
(UTF_8, Err(_)) => String::from_utf8_lossy(raw).into_owned(),
_ => raw.iter().map(|&byte| windows_1252(byte)).collect(),
};
let trimmed = decoded.trim_matches(|c: char| c == '\0' || c.is_whitespace());
(!trimmed.is_empty()).then(|| trimmed.to_string())
}
/// One Windows-1252 byte as a character.
fn windows_1252(byte: u8) -> char {
match byte {
0x80..=0x9f => HIGH_CONTROLS[(byte - 0x80) as usize],
_ => byte as char,
}
}
/// The characters Windows-1252 puts where Latin-1 has C1 controls. The five
/// positions it leaves undefined decode to themselves, as browsers do.
const HIGH_CONTROLS: [char; 32] = [
'\u{20ac}', '\u{81}', '\u{201a}', '\u{192}', '\u{201e}', '\u{2026}', '\u{2020}', '\u{2021}',
'\u{2c6}', '\u{2030}', '\u{160}', '\u{2039}', '\u{152}', '\u{8d}', '\u{17d}', '\u{8f}',
'\u{90}', '\u{2018}', '\u{2019}', '\u{201c}', '\u{201d}', '\u{2022}', '\u{2013}', '\u{2014}',
'\u{2dc}', '\u{2122}', '\u{161}', '\u{203a}', '\u{153}', '\u{9d}', '\u{17e}', '\u{178}',
];
/// The language a MOBI locale names, for books with no EXTH language.
///
/// The low byte is a Windows primary language id; the region above it is
/// dropped, since the file's own language field never carries one either.
fn language_of(locale: u32) -> Option<&'static str> {
Some(match locale & 0xff {
0x01 => "ar",
0x02 => "bg",
0x03 => "ca",
0x04 => "zh",
0x05 => "cs",
0x06 => "da",
0x07 => "de",
0x08 => "el",
0x09 => "en",
0x0a => "es",
0x0b => "fi",
0x0c => "fr",
0x0d => "he",
0x0e => "hu",
0x0f => "is",
0x10 => "it",
0x11 => "ja",
0x12 => "ko",
0x13 => "nl",
0x14 => "no",
0x15 => "pl",
0x16 => "pt",
0x18 => "ro",
0x19 => "ru",
0x1a => "hr",
0x1b => "sk",
0x1d => "sv",
0x1f => "tr",
0x22 => "uk",
0x29 => "fa",
0x2a => "vi",
0x36 => "af",
0x39 => "hi",
_ => return None,
})
}
fn slice(bytes: &[u8], at: usize, len: usize) -> Option<&[u8]> {
bytes.get(at..at.checked_add(len)?)
}
fn u16_at(bytes: &[u8], at: usize) -> Option<u16> {
Some(u16::from_be_bytes(slice(bytes, at, 2)?.try_into().ok()?))
}
fn u32_at(bytes: &[u8], at: usize) -> Option<u32> {
Some(u32::from_be_bytes(slice(bytes, at, 4)?.try_into().ok()?))
}
#[cfg(test)]
mod tests {
use super::*;
const JPEG: &[u8] = &[0xff, 0xd8, 0xff, 0xe0, 0, 16, b'J', b'F', b'I', b'F'];
/// A Kindle book built field by field, so each test can break one thing.
struct Book {
name: &'static [u8],
encoding: u32,
encryption: u16,
locale: u32,
full_name: Vec<u8>,
exth: Option<Vec<(u32, Vec<u8>)>>,
first_image: u32,
images: Vec<Vec<u8>>,
}
impl Default for Book {
fn default() -> Self {
Book {
name: b"A_Test_Book",
encoding: UTF_8,
encryption: 0,
locale: 0x09,
full_name: b"A Test Book".to_vec(),
exth: Some(Vec::new()),
first_image: 1,
images: Vec::new(),
}
}
}
impl Book {
fn bytes(&self) -> Vec<u8> {
const HEADER: usize = 232;
let mut first = vec![0u8; MOBI_AT + HEADER];
first[ENCRYPTION_AT..ENCRYPTION_AT + 2].copy_from_slice(&self.encryption.to_be_bytes());
first[MOBI_AT..MOBI_AT + 4].copy_from_slice(MOBI_MAGIC);
let put = |first: &mut Vec<u8>, at: usize, value: u32| {
first[MOBI_AT + at..MOBI_AT + at + 4].copy_from_slice(&value.to_be_bytes());
};
put(&mut first, HEADER_LENGTH, HEADER as u32);
put(&mut first, TEXT_ENCODING, self.encoding);
put(&mut first, LOCALE, self.locale);
put(&mut first, FIRST_IMAGE, self.first_image);
if let Some(records) = &self.exth {
put(&mut first, EXTH_FLAGS, HAS_EXTH);
let mut body = Vec::new();
for (kind, data) in records {
body.extend_from_slice(&kind.to_be_bytes());
body.extend_from_slice(&(data.len() as u32 + 8).to_be_bytes());
body.extend_from_slice(data);
}
first.extend_from_slice(EXTH_MAGIC);
first.extend_from_slice(&(body.len() as u32 + 12).to_be_bytes());
first.extend_from_slice(&(records.len() as u32).to_be_bytes());
first.extend_from_slice(&body);
}
let name_at = first.len();
first.extend_from_slice(&self.full_name);
put(&mut first, FULL_NAME_OFFSET, name_at as u32);
put(&mut first, FULL_NAME_LENGTH, self.full_name.len() as u32);
let mut records = vec![first];
records.extend(self.images.iter().cloned());
let mut out = vec![0u8; RECORD_LIST_AT + records.len() * 8];
out[..self.name.len()].copy_from_slice(self.name);
out[TYPE_AT..TYPE_AT + 8].copy_from_slice(BOOK_TYPE);
out[RECORD_COUNT_AT..RECORD_COUNT_AT + 2]
.copy_from_slice(&(records.len() as u16).to_be_bytes());
for (at, record) in records.iter().enumerate() {
let offset = out.len() as u32;
let entry = RECORD_LIST_AT + at * 8;
out[entry..entry + 4].copy_from_slice(&offset.to_be_bytes());
out.extend_from_slice(record);
}
out
}
}
fn exth(records: &[(u32, &[u8])]) -> Option<Vec<(u32, Vec<u8>)>> {
Some(records.iter().map(|(k, v)| (*k, v.to_vec())).collect())
}
#[test]
fn the_exth_records_fill_the_record() {
let book = Book {
exth: exth(&[
(UPDATED_TITLE, b"The Real Title"),
(AUTHOR, b"First Author"),
(AUTHOR, b"Second Author"),
(PUBLISHER, b"A Press"),
(DESCRIPTION, b"What it is about."),
(ISBN, b"978-0-306-40615-7"),
(SUBJECT, b"History"),
(SUBJECT, b"Ships"),
(PUBLISHED, b"1999-01-02"),
(ASIN, b"B000TEST00"),
(LANGUAGE, b"de"),
]),
..Book::default()
};
let meta = read("x.mobi", &book.bytes());
assert_eq!(meta.source, "mobi");
assert_eq!(meta.title.as_deref(), Some("The Real Title"));
assert_eq!(meta.authors, ["First Author", "Second Author"]);
assert_eq!(meta.publisher.as_deref(), Some("A Press"));
assert_eq!(meta.description.as_deref(), Some("What it is about."));
assert_eq!(meta.date.as_deref(), Some("1999-01-02"));
assert_eq!(meta.subjects, ["History", "Ships"]);
assert_eq!(meta.language.as_deref(), Some("de"));
assert_eq!(meta.identifiers["isbn"], "9780306406157");
assert_eq!(meta.identifiers["asin"], "B000TEST00");
assert!(meta.notes.is_empty(), "{:?}", meta.notes);
}
#[test]
fn the_title_falls_back_to_the_full_name_and_then_the_database_name() {
let meta = read("x.mobi", &Book::default().bytes());
assert_eq!(meta.title.as_deref(), Some("A Test Book"));
let book = Book {
full_name: Vec::new(),
..Book::default()
};
let meta = read("x.mobi", &book.bytes());
assert_eq!(meta.title.as_deref(), Some("A_Test_Book"));
assert!(meta.notes[0].contains("Palm database name"));
}
#[test]
fn the_language_falls_back_to_the_locale() {
let book = Book {
locale: 0x040c,
..Book::default()
};
assert_eq!(
read("x.mobi", &book.bytes()).language.as_deref(),
Some("fr")
);
let book = Book {
locale: 0,
..Book::default()
};
assert_eq!(read("x.mobi", &book.bytes()).language, None);
}
#[test]
fn windows_1252_strings_decode_as_windows_1252() {
let book = Book {
encoding: WINDOWS_1252,
exth: exth(&[
(AUTHOR, b"Ren\xe9 Ch\x96ar"),
(PUBLISHER, b"\x93Quoted\x94"),
]),
..Book::default()
};
let meta = read("x.mobi", &book.bytes());
assert_eq!(meta.authors, ["René Ch–ar"]);
assert_eq!(meta.publisher.as_deref(), Some("“Quoted”"));
}
#[test]
fn the_cover_is_the_image_record_the_offset_names() {
let book = Book {
exth: exth(&[(COVER_OFFSET, &1u32.to_be_bytes())]),
images: vec![b"GIF89a-thumb".to_vec(), JPEG.to_vec()],
..Book::default()
};
let cover = read("x.mobi", &book.bytes()).cover.expect("a cover");
assert_eq!(cover.media_type, "image/jpeg");
assert_eq!(cover.bytes, JPEG);
}
#[test]
fn the_thumbnail_stands_in_when_no_cover_is_named() {
let book = Book {
exth: exth(&[(THUMB_OFFSET, &0u32.to_be_bytes())]),
images: vec![b"\x89PNG\r\n\x1a\nrest".to_vec()],
..Book::default()
};
let cover = read("x.mobi", &book.bytes()).cover.expect("a cover");
assert_eq!(cover.media_type, "image/png");
}
#[test]
fn with_no_cover_named_only_the_record_at_first_image_can_be_the_cover() {
let book = Book {
images: vec![JPEG.to_vec(), b"GIF89a-later".to_vec()],
..Book::default()
};
let cover = read("x.mobi", &book.bytes()).cover.expect("a cover");
assert_eq!(cover.media_type, "image/jpeg");
assert_eq!(cover.bytes, JPEG);
// The record at first_image is not an image: no cover, even though a
// later record is one.
let book = Book {
images: vec![
b"not an image".to_vec(),
JPEG.to_vec(),
b"GIF89a-later".to_vec(),
],
..Book::default()
};
assert_eq!(read("x.mobi", &book.bytes()).cover, None);
let book = Book {
images: vec![b"not an image".to_vec()],
..Book::default()
};
assert_eq!(read("x.mobi", &book.bytes()).cover, None);
}
#[test]
fn a_font_at_first_image_is_not_skipped_past_to_an_inline_image() {
// No EXTH 201 or 202. The first resource record is a font, as in
// kindlegen output; calibre reads only that record and finds no cover.
let book = Book {
images: vec![b"FONT\0\0\0\x18".to_vec(), JPEG.to_vec()],
..Book::default()
};
assert_eq!(read("x.mobi", &book.bytes()).cover, None);
}
#[test]
fn a_stale_kf8_boundary_below_first_image_does_not_hide_the_cover() {
let book = Book {
exth: exth(&[(121, &1u32.to_be_bytes())]),
first_image: 2,
images: vec![b"text".to_vec(), JPEG.to_vec()],
..Book::default()
};
assert!(read("x.mobi", &book.bytes()).cover.is_some());
}
#[test]
fn the_first_image_search_stops_at_the_kf8_boundary() {
let book = Book {
exth: exth(&[(121, &2u32.to_be_bytes())]),
images: vec![b"text".to_vec(), JPEG.to_vec()],
..Book::default()
};
// Record 1 is not an image, and record 2 is past the boundary.
assert_eq!(read("x.mobi", &book.bytes()).cover, None);
}
#[test]
fn exth_113_is_an_asin_a_uuid_or_something_else() {
let id = |value: &[u8]| {
let book = Book {
exth: exth(&[(ASIN, value)]),
..Book::default()
};
read("x.mobi", &book.bytes()).identifiers
};
assert_eq!(id(b"B01A7RJGIW")["asin"], "B01A7RJGIW");
assert_eq!(id(b"0306406152")["asin"], "0306406152");
let uuid = id(b"69AE65CA-B644-499F-A78C-139771C7140A");
assert_eq!(uuid["uuid"], "69ae65ca-b644-499f-a78c-139771c7140a");
assert_eq!(uuid.get("asin"), None);
let other = id(b"2AW6TL2JJEYYGXCOV3PWN6ZOYUORXAVX");
assert_eq!(other["other"], "2AW6TL2JJEYYGXCOV3PWN6ZOYUORXAVX");
assert_eq!(other.get("asin"), None);
}
#[test]
fn a_cover_that_points_nowhere_is_a_note_and_not_a_cover() {
let book = Book {
exth: exth(&[(COVER_OFFSET, &7u32.to_be_bytes())]),
images: vec![JPEG.to_vec()],
..Book::default()
};
let meta = read("x.mobi", &book.bytes());
assert_eq!(meta.cover, None);
assert_eq!(meta.notes, ["the cover points past the last record"]);
let book = Book {
exth: exth(&[(COVER_OFFSET, &u32::MAX.to_be_bytes())]),
first_image: u32::MAX - 1,
images: vec![JPEG.to_vec()],
..Book::default()
};
assert_eq!(read("x.mobi", &book.bytes()).cover, None);
let book = Book {
exth: exth(&[(COVER_OFFSET, &0u32.to_be_bytes())]),
images: vec![b"not an image".to_vec()],
..Book::default()
};
let meta = read("x.mobi", &book.bytes());
assert_eq!(meta.cover, None);
assert_eq!(meta.notes, ["the cover record is not an image"]);
}
#[test]
fn an_encrypted_book_still_has_its_header_read() {
let book = Book {
encryption: 2,
exth: exth(&[(AUTHOR, b"Somebody")]),
..Book::default()
};
let meta = read("x.azw", &book.bytes());
assert_eq!(meta.authors, ["Somebody"]);
assert_eq!(meta.notes, ["drm"]);
}
#[test]
fn an_exth_record_that_overruns_its_block_stops_the_reading() {
let mut bytes = Book {
exth: exth(&[(AUTHOR, b"Kept"), (PUBLISHER, b"Lost")]),
..Book::default()
}
.bytes();
// The second record's length, made to reach past the block.
let at = find(&bytes, b"Lost") - 4;
bytes[at..at + 4].copy_from_slice(&0x7fff_ffffu32.to_be_bytes());
let meta = read("x.mobi", &bytes);
assert_eq!(meta.authors, ["Kept"]);
assert_eq!(meta.publisher, None);
assert!(
meta.notes
.contains(&"the EXTH block ends early".to_string())
);
}
#[test]
fn a_record_length_shorter_than_its_own_header_stops_the_reading() {
let mut bytes = Book {
exth: exth(&[(AUTHOR, b"Kept")]),
..Book::default()
}
.bytes();
let at = find(&bytes, b"Kept") - 4;
bytes[at..at + 4].copy_from_slice(&3u32.to_be_bytes());
let meta = read("x.mobi", &bytes);
assert!(meta.authors.is_empty());
assert!(
meta.notes
.contains(&"the EXTH block ends early".to_string())
);
}
#[test]
fn what_is_not_a_kindle_book_says_so() {
let meta = read("x.mobi", b"not a palm database at all");
assert_eq!(meta.source, "none");
assert!(meta.notes[0].starts_with("not a Kindle book"));
}
#[test]
fn a_first_record_with_no_mobi_header_is_refused() {
let mut bytes = Book::default().bytes();
let at = mobi_header(&bytes);
bytes[at] = b'X';
let meta = read("x.mobi", &bytes);
assert_eq!(meta.source, "none");
assert_eq!(meta.notes, ["no MOBI header in the first record"]);
}
#[test]
fn a_header_too_short_for_a_field_does_not_have_that_field() {
let mut bytes = Book::default().bytes();
let at = mobi_header(&bytes) + HEADER_LENGTH;
// Long enough for the encoding, too short for the full name.
bytes[at..at + 4].copy_from_slice(&24u32.to_be_bytes());
let meta = read("x.mobi", &bytes);
assert_eq!(meta.title.as_deref(), Some("A_Test_Book"));
assert_eq!(meta.language, None);
}
#[test]
fn every_truncation_returns_without_panicking() {
let bytes = Book {
exth: exth(&[
(UPDATED_TITLE, b"Title"),
(AUTHOR, b"Author"),
(COVER_OFFSET, &0u32.to_be_bytes()),
]),
images: vec![JPEG.to_vec()],
..Book::default()
}
.bytes();
for len in 0..=bytes.len() {
let _ = read("x.mobi", &bytes[..len]);
}
}
/// Where the MOBI header starts: after the record list, so the `MOBI`
/// at the end of the database type is not it.
fn mobi_header(bytes: &[u8]) -> usize {
RECORD_LIST_AT + find(&bytes[RECORD_LIST_AT..], MOBI_MAGIC)
}
fn find(haystack: &[u8], needle: &[u8]) -> usize {
haystack
.windows(needle.len())
.position(|window| window == needle)
.expect("present")
}
}
crates/bookmeta/tests/fixtures/README.md
Vendored
+28
View File
@@ -0,0 +1,28 @@
# Fixtures
## MOBI and AZW3
`lantern.html` is a two-paragraph story written for these tests, and
`lantern-cover.jpg` a 120×180 cover drawn for them with Pillow. Both are
dedicated to the public domain (CC0 1.0), and so is everything made from them.
`lantern.mobi` (MOBI 6) and `lantern.azw3` (KF8) were made from them with
calibre 9.14:
```sh
common=(--title "A Lantern in the Orchard" --authors "Ada Quill & Tomás Reed"
--publisher "Orchard Press" --isbn 9780306406157 --tags "Fiction,Villages"
--language en --pubdate 2001-04-05 --comments "A short story written for testing."
--cover lantern-cover.jpg)
ebook-convert lantern.html lantern.mobi --mobi-file-type old "${common[@]}"
ebook-convert lantern.html lantern.azw3 "${common[@]}"
```
The other three are `lantern.mobi` with one thing broken. `r0` is the first
record's offset, the big-endian 32-bit number at byte 78 (136 here).
| File | Change |
|---|---|
| `lantern-no-exth.mobi` | The EXTH flag (`0x40`) cleared in the MOBI header's flags, at `r0 + 16 + 112` |
| `lantern-drm.mobi` | The PalmDOC encryption type, at `r0 + 12`, set to 2 |
| `lantern-truncated.mobi` | The first 1024 bytes only |
crates/bookmeta/tests/fixtures/lantern-cover.jpg
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.7 KiB

crates/bookmeta/tests/fixtures/lantern-drm.mobi
Vendored
BIN
View File
Binary file not shown.
crates/bookmeta/tests/fixtures/lantern-no-exth.mobi
Vendored
BIN
View File
Binary file not shown.
crates/bookmeta/tests/fixtures/lantern-truncated.mobi
Vendored
BIN
View File
Binary file not shown.
crates/bookmeta/tests/fixtures/lantern.azw3
Vendored
BIN
View File
Binary file not shown.
crates/bookmeta/tests/fixtures/lantern.html
Vendored
+5
View File
@@ -0,0 +1,5 @@
<html><head><meta charset="utf-8"><title>x</title></head><body>
<h1>A Lantern in the Orchard</h1>
<p>The lantern hung from the lowest branch of the pear tree, and every evening someone lit it. Nobody in the village could say who.</p>
<p>Café owners swore it was the miller; the miller swore it was the wind.</p>
</body></html>
crates/bookmeta/tests/fixtures/lantern.mobi
Vendored
BIN
View File
Binary file not shown.
crates/bookmeta/tests/mobi.rs
+146
View File
@@ -0,0 +1,146 @@
//! MOBI and AZW3 files as a converter writes them, and as they break.
use std::path::{Path, PathBuf};
fn fixture(name: &str) -> PathBuf {
Path::new(env!("CARGO_MANIFEST_DIR"))
.join("tests/fixtures")
.join(name)
}
fn read(name: &str) -> bookmeta::Metadata {
bookmeta::read(&fixture(name))
}
/// What the two converted fixtures were given, and so must say.
fn says_what_it_was_given(meta: &bookmeta::Metadata) {
assert_eq!(meta.source, "mobi");
assert_eq!(meta.title.as_deref(), Some("A Lantern in the Orchard"));
assert_eq!(meta.authors, ["Ada Quill", "Tomás Reed"]);
assert_eq!(meta.publisher.as_deref(), Some("Orchard Press"));
assert_eq!(
meta.description.as_deref(),
Some("A short story written for testing.")
);
assert_eq!(meta.date.as_deref(), Some("2001-04-05T04:00:00+00:00"));
assert_eq!(meta.language.as_deref(), Some("en"));
assert_eq!(meta.subjects, ["Fiction", "Villages"]);
assert_eq!(meta.identifiers["isbn"], "9780306406157");
assert_eq!(meta.identifiers.get("asin"), None);
assert_eq!(meta.identifiers["uuid"].len(), 36);
let cover = meta.cover.as_ref().expect("a cover");
assert_eq!(cover.media_type, "image/jpeg");
assert!(!cover.guessed, "EXTH 201 names the cover");
let original = std::fs::read(fixture("lantern-cover.jpg")).expect("the cover fixture");
assert_eq!(cover.bytes, original);
}
#[test]
fn a_mobi_6_file_says_what_it_was_given() {
let meta = read("lantern.mobi");
says_what_it_was_given(&meta);
assert!(meta.notes.is_empty(), "{:?}", meta.notes);
}
#[test]
fn an_azw3_file_says_what_it_was_given() {
let meta = read("lantern.azw3");
says_what_it_was_given(&meta);
assert!(meta.notes.is_empty(), "{:?}", meta.notes);
}
#[test]
fn exth_113_uuid_is_not_an_asin() {
for name in ["lantern.mobi", "lantern.azw3"] {
let meta = read(name);
assert_eq!(meta.identifiers.get("asin"), None, "{name}");
assert!(meta.identifiers.contains_key("uuid"), "{name}");
}
}
/// `lantern.mobi` with its cover and thumbnail offsets (EXTH 201 and 202)
/// retyped to an unused record type, so the file names no cover.
fn lantern_naming_no_cover() -> Vec<u8> {
let mut bytes = std::fs::read(fixture("lantern.mobi")).expect("the fixture");
let exth = bytes.windows(4).position(|w| w == b"EXTH").expect("EXTH");
let count = u32::from_be_bytes(bytes[exth + 8..exth + 12].try_into().unwrap());
let mut cursor = exth + 12;
let mut retyped = 0;
for _ in 0..count {
let kind = u32::from_be_bytes(bytes[cursor..cursor + 4].try_into().unwrap());
let size = u32::from_be_bytes(bytes[cursor + 4..cursor + 8].try_into().unwrap());
if kind == 201 || kind == 202 {
bytes[cursor..cursor + 4].copy_from_slice(&999u32.to_be_bytes());
retyped += 1;
}
cursor += size as usize;
}
assert!(retyped > 0, "the fixture names a cover");
bytes
}
#[test]
fn a_file_naming_no_cover_gets_its_first_image_record() {
let meta = bookmeta::from_bytes("x.mobi", &lantern_naming_no_cover());
let cover = meta.cover.expect("the first image record");
assert_eq!(cover.media_type, "image/jpeg");
assert!(cover.guessed, "a cover taken by position is a guess");
let original = std::fs::read(fixture("lantern-cover.jpg")).expect("the cover fixture");
assert_eq!(cover.bytes, original);
}
#[test]
fn a_file_with_no_exth_still_has_its_full_name_and_locale() {
let meta = read("lantern-no-exth.mobi");
assert_eq!(meta.source, "mobi");
assert_eq!(meta.title.as_deref(), Some("A Lantern in the Orchard"));
assert_eq!(meta.language.as_deref(), Some("en"));
assert!(meta.authors.is_empty());
assert!(meta.identifiers.is_empty());
// No EXTH names no cover, so the first image record stands in for one.
let cover = meta.cover.expect("the first image record");
assert_eq!(cover.media_type, "image/jpeg");
}
#[test]
fn a_truncated_file_is_an_empty_record_with_a_reason() {
let meta = read("lantern-truncated.mobi");
assert_eq!(meta.source, "none");
assert!(meta.is_empty());
assert_eq!(meta.notes, ["the first record lies outside the file"]);
}
#[test]
fn a_drm_flagged_file_has_its_metadata_and_says_drm() {
let meta = read("lantern-drm.mobi");
says_what_it_was_given(&meta);
assert_eq!(meta.notes, ["drm"]);
}
#[test]
fn every_truncation_of_the_mobi_fixture_returns() {
let bytes = std::fs::read(fixture("lantern.mobi")).expect("the fixture");
let first_record_end = u32::from_be_bytes(bytes[86..90].try_into().unwrap()) as usize;
for len in 0..=bytes.len() {
let meta = bookmeta::from_bytes("cut.mobi", &bytes[..len]);
if len < first_record_end {
assert!(meta.is_empty(), "cut at {len}: {meta:?}");
}
}
}
#[test]
fn every_single_byte_corruption_of_the_header_returns() {
let bytes = std::fs::read(fixture("lantern.mobi")).expect("the fixture");
// The Palm header, the record list, and the whole of the first record:
// every offset the reader follows is in there.
let first_record_end = u32::from_be_bytes(bytes[86..90].try_into().unwrap()) as usize;
for at in 0..first_record_end {
for value in [0x00, 0x7f, 0xff] {
let mut broken = bytes.clone();
broken[at] = value;
let _ = bookmeta::from_bytes("broken.mobi", &broken);
}
}
}
crates/booksniff/README.md
+21 -2
View File
@@ -26,8 +26,8 @@ file too small to be a book, a name that says "sample".
cover images that ended up in a folder of PDFs.
A library accumulates files that lie: a DjVu named `.pdf`, a Word document
named `.epub`, an EPUB saved as `.zip`, a single PDF that somebody zipped and
renamed back to `.pdf`.
named `.epub`, an RTF saved as `.doc`, an EPUB saved as `.zip`, a single PDF
that somebody zipped and renamed back to `.pdf`.
## Install
@@ -89,7 +89,12 @@ one bad file does not stop a sweep.
| `ole2` | The compound-file header every binary Word document starts with |
| `epub` | A zip holding `mimetype` and `META-INF/container.xml` |
| `zip-wrapped-pdf` | A zip whose only real member is one PDF |
| `zip-wrapped-fb2` | A zip whose only real member is one FictionBook — the `.fb2.zip` most of them are passed around as |
| `cbz` | A zip whose every real member is a page image: JPEG, PNG, GIF or WebP |
| `azw3`, `mobi` | A Palm database tagged `BOOKMOBI`, told apart by the format version inside the first record |
| `lit` | `ITOLITLS`, Microsoft Reader's signature |
| `rtf` | `{\rtf` at the front, after any byte-order mark and blank space |
| `fb2` | XML whose root element is `FictionBook`, found by stepping over the declaration, comments and doctype in the first four kilobytes |
| `html` | Markup at the front, after any byte-order mark |
| `txt` | Text that is none of the above |
| `unknown` | Anything else, with a note saying what was seen |
@@ -98,6 +103,20 @@ An archive holding several PDFs is a collection, not a wrapped book: choosing
among them would be a guess, and naming one chapter as the whole work is worse
than refusing.
A zipped FictionBook is reported as wrapped rather than as `fb2`, the same way
a zipped PDF is: a converter handed it as FictionBook is handed a zip, and the
kind is there to stop exactly that.
A comic is the pages and nothing else. Directories, the `ComicInfo.xml` a
comic reader writes into the archive, and the desktop clutter every archive
picks up are passed over; anything more — one PDF among the pages, a readme —
and the zip is not a comic, and says what it holds instead. Only names are
read, out of the archive's directory, so no member is ever unpacked.
FictionBook is read as bytes rather than as text. A great many are written in
windows-1251, which spells its markup in ASCII but is not UTF-8, and would
otherwise be called binary.
## As a library
```rust
crates/booksniff/src/fb2.rs
+162
View File
@@ -0,0 +1,162 @@
//! Telling a FictionBook from any other XML.
//!
//! A FictionBook is an XML document whose root element is `FictionBook`.
//! The root comes after the prolog — a declaration, comments, perhaps a
//! doctype — so it is found by stepping over those rather than by matching
//! at a fixed offset.
//!
//! The walk is over bytes, not text. A great many FictionBooks are written in
//! windows-1251, which is not UTF-8 but spells its markup in ASCII all the
//! same, and a reader that insisted on UTF-8 would call every one of them
//! binary.
use crate::BOM;
/// What the root element of a FictionBook is called.
const ROOT: &[u8] = b"FictionBook";
/// Whether the front of a file is the start of a FictionBook.
///
/// `head` is however much of the front the caller read. A root element that
/// does not begin inside it is not found, so the answer is no rather than a
/// guess.
pub fn is_fictionbook(head: &[u8]) -> bool {
root_element(head).is_some_and(|name| local_name(name) == ROOT)
}
/// The name of the first element, after the prolog.
fn root_element(head: &[u8]) -> Option<&[u8]> {
let mut rest = head.strip_prefix(BOM).unwrap_or(head);
loop {
rest = rest.trim_ascii_start();
if let Some(after) = rest.strip_prefix(b"<?") {
rest = after_marker(after, b"?>")?;
} else if let Some(after) = rest.strip_prefix(b"<!--") {
rest = after_marker(after, b"-->")?;
} else if let Some(after) = rest.strip_prefix(b"<!") {
rest = after_declaration(after)?;
} else {
// Anything but an element here is text before the root, which
// XML does not allow.
let after = rest.strip_prefix(b"<")?;
// A name that runs to the end of what was read may be the front
// of a longer one, so it is not taken.
let end = after.iter().position(|byte| !is_name_byte(*byte))?;
let ends_properly =
matches!(after[end], b'>' | b'/') || after[end].is_ascii_whitespace();
return (end > 0 && ends_properly).then_some(&after[..end]);
}
}
}
/// What follows the first `marker`, if it is there at all.
fn after_marker<'a>(rest: &'a [u8], marker: &[u8]) -> Option<&'a [u8]> {
rest.windows(marker.len())
.position(|window| window == marker)
.map(|at| &rest[at + marker.len()..])
}
/// What follows a doctype, whose internal subset may hold `>` of its own.
fn after_declaration(rest: &[u8]) -> Option<&[u8]> {
let mut depth = 0usize;
let mut quote = None;
for (at, &byte) in rest.iter().enumerate() {
match quote {
Some(open) if byte == open => quote = None,
Some(_) => {}
None => match byte {
b'"' | b'\'' => quote = Some(byte),
b'[' => depth += 1,
b']' => depth = depth.saturating_sub(1),
b'>' if depth == 0 => return Some(&rest[at + 1..]),
_ => {}
},
}
}
None
}
/// The name without its namespace prefix.
fn local_name(name: &[u8]) -> &[u8] {
match name.iter().rposition(|byte| *byte == b':') {
Some(at) => &name[at + 1..],
None => name,
}
}
fn is_name_byte(byte: u8) -> bool {
byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'-' | b'.' | b':') || byte >= 0x80
}
#[cfg(test)]
mod tests {
use super::*;
const BOOK: &[u8] = b"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n\
<FictionBook xmlns=\"http://www.gribuser.ru/xml/fictionbook/2.0\">\n\
<description><title-info><book-title>The Mill</book-title></title-info></description>";
#[test]
fn a_fictionbook_is_recognised() {
assert!(is_fictionbook(BOOK));
assert!(is_fictionbook(b"<FictionBook>"));
}
#[test]
fn the_prolog_is_stepped_over() {
let head = b"\xef\xbb\xbf <?xml version=\"1.0\"?>\r\n\
<!-- converted by hand -->\n\
<!DOCTYPE FictionBook [ <!ENTITY mill \"<b>the mill</b>\"> ]>\n\
<?xml-stylesheet href=\"fb2.css\"?>\n\
<FictionBook\n xmlns=\"http://www.gribuser.ru/xml/fictionbook/2.0\">";
assert!(is_fictionbook(head));
}
#[test]
fn a_prefixed_root_is_still_a_fictionbook() {
assert!(is_fictionbook(
b"<fb:FictionBook xmlns:fb=\"http://www.gribuser.ru/xml/fictionbook/2.0\">"
));
}
#[test]
fn markup_need_not_be_utf8() {
// windows-1251 for "Мельница" in the title.
let head = b"<?xml version=\"1.0\" encoding=\"windows-1251\"?>\
<FictionBook><description><book-title>\xcc\xe5\xeb\xfc\xed\xe8\xf6\xe0</book-title>";
assert!(is_fictionbook(head));
}
#[test]
fn other_xml_is_not_a_fictionbook() {
assert!(!is_fictionbook(
b"<?xml version=\"1.0\"?>\n<rss version=\"2.0\">"
));
assert!(!is_fictionbook(b"<?xml version=\"1.0\"?><FictionBookish>"));
assert!(!is_fictionbook(b"<html><FictionBook>"));
assert!(!is_fictionbook(b"text before <FictionBook>"));
assert!(!is_fictionbook(b""));
}
#[test]
fn a_root_the_window_cut_short_is_not_taken() {
assert!(!is_fictionbook(b"<?xml version=\"1.0\"?><FictionBook"));
assert!(!is_fictionbook(b"<?xml version=\"1.0\"?><Fiction"));
assert!(!is_fictionbook(b"<?xml version=\"1.0\""));
assert!(!is_fictionbook(b"<!-- never closed <FictionBook>"));
}
#[test]
fn a_truncated_or_mangled_prolog_does_not_panic() {
let head = b"\xef\xbb\xbf<?xml?><!--c--><!DOCTYPE x [ <!ENTITY a '>'> ] \"]\">\
<FictionBook>";
assert!(is_fictionbook(head));
for cut in 0..head.len() {
let _ = is_fictionbook(&head[..cut]);
}
for byte in 0..=255u8 {
let _ = is_fictionbook(&[b'<', byte]);
let _ = is_fictionbook(&[b'<', b'!', byte, b'>']);
}
}
}
crates/booksniff/src/lib.rs
+254 -13
View File
@@ -9,6 +9,7 @@
//! The containers are unambiguous, so they are read rather than guessed at.
pub mod bytes;
pub mod fb2;
pub mod mobi;
pub mod zip;
@@ -22,17 +23,32 @@ const DJVU: &[u8] = b"AT&TFORM";
const OLE2: &[u8] = &[0xd0, 0xcf, 0x11, 0xe0, 0xa1, 0xb1, 0x1a, 0xe1];
/// A PDF announces its version in its first bytes.
const PDF: &[u8] = b"%PDF-";
/// Microsoft Reader's signature, ahead of everything else in the file.
const LIT: &[u8] = b"ITOLITLS";
/// The group every RTF document is, opened with the control word that names
/// the format.
const RTF: &[u8] = br"{\rtf";
/// The UTF-8 byte-order mark.
const BOM: &[u8] = b"\xef\xbb\xbf";
/// Archive members that carry no content: resource forks and desktop clutter.
const ARCHIVE_NOISE: &[&str] = &["__macosx/", ".ds_store", "thumbs.db"];
/// What a comic book archive carries besides its pages: the catalogue entry
/// comic readers write into it.
const COMIC_NOISE: &[&str] = &["comicinfo.xml"];
/// The extensions of the images a comic book archive is made of.
const PAGE_IMAGES: &[&str] = &[".jpg", ".jpeg", ".png", ".gif", ".webp"];
/// Substrings in a name that mark a file as a sample rather than a book.
const SAMPLE_MARKERS: &[&str] = &["sample", "readme", "read me"];
/// Below this, a file is too small to be a book whatever it claims to be.
const SMALLEST_BOOK: u64 = 4096;
/// How much of the front is read to tell text from markup.
/// How much of the front is read to tell RTF, FictionBook, markup and text
/// apart.
const TEXT_WINDOW: usize = 4096;
/// Identify a file.
@@ -49,12 +65,17 @@ pub fn sniff(source: &impl Bytes, name: &str) -> Doc {
Kind::Pdf
} else if source.magic_at(0, OLE2) {
Kind::Ole2
} else if source.magic_at(0, LIT) {
Kind::Lit
} else if source.magic_at(0, zip::LOCAL_HEADER) {
archive(source, &mut doc)
} else if let Some(kind) = mobi::kind(source) {
kind
} else {
text(source, &mut doc)
match source.slice(0, TEXT_WINDOW) {
Ok(head) => front(&head, &mut doc),
Err(_) => Kind::Unknown,
}
};
if doc.kind == Kind::Unknown {
@@ -80,7 +101,8 @@ pub fn sniff(source: &impl Bytes, name: &str) -> Doc {
doc
}
/// What is inside a zip: an EPUB, a wrapped PDF, or a collection of pages.
/// What is inside a zip: an EPUB, a wrapped book, a comic, or a collection of
/// pages.
fn archive(source: &impl Bytes, doc: &mut Doc) -> Kind {
let Some(names) = zip::members(source) else {
doc.note("a zip whose directory could not be read");
@@ -99,6 +121,13 @@ fn archive(source: &impl Bytes, doc: &mut Doc) -> Kind {
doc.note(format!("a zip holding one PDF, {pdf:?}"));
return Kind::ZipWrappedPdf;
}
if let Some(fb2) = sole_member(&names, ".fb2") {
doc.note(format!("a zip holding one FictionBook, {fb2:?}"));
return Kind::ZipWrappedFb2;
}
if is_comic(&names) {
return Kind::Cbz;
}
if lower
.iter()
.any(|name| name.ends_with(".htm") || name.ends_with(".html") || name.ends_with(".xhtml"))
@@ -119,25 +148,67 @@ fn archive(source: &impl Bytes, doc: &mut Doc) -> Kind {
/// a collection, and choosing among them would be a guess — better to say it
/// is not a book than to name one chapter as the whole.
pub fn sole_pdf(names: &[String]) -> Option<&str> {
let real: Vec<&String> = names
sole_member(names, ".pdf")
}
/// The one member an archive exists to carry, if it is the only real member
/// and its name ends in `extension`.
fn sole_member<'a>(names: &'a [String], extension: &str) -> Option<&'a str> {
match real_members(names).as_slice() {
[only] if only.to_lowercase().ends_with(extension) => Some(only.as_str()),
_ => None,
}
}
/// Whether an archive is the pages of a comic and nothing else.
///
/// Every real member has to be an image. One PDF or one text file among the
/// pages makes it something else, and saying what would be a guess.
fn is_comic(names: &[String]) -> bool {
let pages: Vec<String> = real_members(names)
.into_iter()
.map(|name| name.to_lowercase())
.filter(|name| {
let file = name.rsplit('/').next().unwrap_or(name);
!COMIC_NOISE.contains(&file)
})
.collect();
!pages.is_empty()
&& pages
.iter()
.all(|name| PAGE_IMAGES.iter().any(|image| name.ends_with(image)))
}
/// The members that carry content: no directories, no desktop clutter.
fn real_members(names: &[String]) -> Vec<&String> {
names
.iter()
.filter(|name| !name.ends_with('/'))
.filter(|name| {
let lower = name.to_lowercase();
!ARCHIVE_NOISE.iter().any(|noise| lower.contains(noise))
})
.collect();
match real.as_slice() {
[only] if only.to_lowercase().ends_with(".pdf") => Some(only.as_str()),
_ => None,
.collect()
}
/// What the front of a file says when it is not a container: RTF, a
/// FictionBook, markup, text, or nothing this knows.
fn front(head: &[u8], doc: &mut Doc) -> Kind {
// Both are recognised on bytes rather than text. RTF is seven-bit by
// design but not always in practice, and a FictionBook is as often
// windows-1251 as UTF-8.
let lead = head.strip_prefix(BOM).unwrap_or(head).trim_ascii_start();
if lead.starts_with(RTF) {
return Kind::Rtf;
}
if fb2::is_fictionbook(head) {
return Kind::Fb2;
}
text(head, doc)
}
/// Text, markup, or nothing this knows.
fn text(source: &impl Bytes, doc: &mut Doc) -> Kind {
let Ok(head) = source.slice(0, TEXT_WINDOW) else {
return Kind::Unknown;
};
fn text(head: &[u8], doc: &mut Doc) -> Kind {
if head.is_empty() {
doc.note("the file is empty");
return Kind::Unknown;
@@ -149,7 +220,7 @@ fn text(source: &impl Bytes, doc: &mut Doc) -> Kind {
}
// The window may have cut a character in half, which is not the same
// thing as the file not being text.
let head = match std::str::from_utf8(&head) {
let head = match std::str::from_utf8(head) {
Ok(text) => text,
Err(e) if e.error_len().is_none() => {
std::str::from_utf8(&head[..e.valid_up_to()]).expect("a valid prefix")
@@ -178,10 +249,15 @@ fn names_the_same_thing(extension: &str, kind: Kind) -> bool {
Kind::Djvu => matches!(extension, "djvu" | "djv"),
Kind::Azw3 => matches!(extension, "azw3" | "azw" | "kfx"),
Kind::Mobi => matches!(extension, "mobi" | "prc" | "azw"),
Kind::Lit => extension == "lit",
Kind::Fb2 => extension == "fb2",
Kind::Rtf => extension == "rtf",
Kind::Txt => matches!(extension, "txt" | "text" | "md" | "markdown"),
Kind::Html => matches!(extension, "html" | "htm" | "xhtml" | "xml"),
Kind::Ole2 => matches!(extension, "doc" | "xls" | "ppt"),
Kind::ZipWrappedPdf => extension == "zip",
Kind::ZipWrappedFb2 => matches!(extension, "zip" | "fbz"),
Kind::Cbz => extension == "cbz",
// Nothing was recognised, so there is nothing to disagree with.
Kind::Unknown => true,
}
@@ -194,10 +270,15 @@ fn spelling(kind: Kind) -> &'static str {
Kind::Djvu => "DjVu",
Kind::Azw3 => "AZW3",
Kind::Mobi => "MOBI",
Kind::Lit => "Microsoft Reader",
Kind::Fb2 => "FictionBook",
Kind::Rtf => "RTF",
Kind::Txt => "plain text",
Kind::Html => "HTML",
Kind::Ole2 => "an OLE2 document",
Kind::ZipWrappedPdf => "a zip holding one PDF",
Kind::ZipWrappedFb2 => "a zip holding one FictionBook",
Kind::Cbz => "a comic book archive",
Kind::Unknown => "nothing recognisable",
}
}
@@ -332,4 +413,164 @@ mod tests {
head.truncate(TEXT_WINDOW + 1);
assert_eq!(sniff(&head.as_slice(), "a.txt").kind, Kind::Txt);
}
fn archive_of(names: &[&str], name: &str) -> Doc {
sniff(&zip::tests::zip(names).as_slice(), name)
}
#[test]
fn rtf_is_recognised_by_its_opening_group() {
assert_eq!(kind_of(br"{\rtf1\ansi\deff0 {\fonttbl", "a.rtf"), Kind::Rtf);
assert_eq!(
kind_of(b"\xef\xbb\xbf\r\n {\\rtf1\\ansi", "a.rtf"),
Kind::Rtf
);
// Eight-bit text in the body does not stop it being RTF.
assert_eq!(kind_of(b"{\\rtf1 caf\xe9 \x00", "a.rtf"), Kind::Rtf);
}
#[test]
fn rtf_cut_short_of_its_control_word_is_not_rtf() {
assert_eq!(sniff(&br"{\rt".as_slice(), "a.rtf").kind, Kind::Txt);
assert_eq!(kind_of(br"{\rtx1", "a.rtf"), Kind::Txt);
assert_eq!(kind_of(br"text {\rtf1", "a.rtf"), Kind::Txt);
}
#[test]
fn rtf_named_as_a_word_document_is_remarked_on() {
let doc = sniff(&padded(br"{\rtf1\ansi").as_slice(), "letter.doc");
assert_eq!(doc.kind, Kind::Rtf);
assert!(
doc.notes
.iter()
.any(|note| note == "named .doc, but the bytes say RTF"),
"{:?}",
doc.notes
);
}
#[test]
fn microsoft_reader_is_recognised_by_its_signature() {
let doc = sniff(&padded(b"ITOLITLS\x01\x00\x00\x00").as_slice(), "book.lit");
assert_eq!(doc.kind, Kind::Lit);
assert!(doc.notes.is_empty(), "{:?}", doc.notes);
assert_eq!(kind_of(b"ITOLITL", "book.lit"), Kind::Txt);
assert_eq!(kind_of(b" ITOLITLS", "book.lit"), Kind::Txt);
}
#[test]
fn a_fictionbook_is_recognised_and_other_xml_is_not() {
let book = b"<?xml version=\"1.0\" encoding=\"windows-1251\"?>\n\
<FictionBook xmlns=\"http://www.gribuser.ru/xml/fictionbook/2.0\">\
<book-title>\xcc\xe5\xeb\xfc\xed\xe8\xf6\xe0</book-title>";
let doc = sniff(&padded(book).as_slice(), "mill.fb2");
assert_eq!(doc.kind, Kind::Fb2);
assert!(doc.notes.is_empty(), "{:?}", doc.notes);
let feed = b"<?xml version=\"1.0\"?>\n<rss version=\"2.0\"><channel>";
assert_eq!(kind_of(feed, "feed.xml"), Kind::Html);
}
#[test]
fn a_fictionbook_named_as_something_else_is_remarked_on() {
let doc = sniff(&padded(b"<FictionBook>").as_slice(), "mill.txt");
assert_eq!(doc.kind, Kind::Fb2);
assert!(
doc.notes.iter().any(|note| note.contains("FictionBook")),
"{:?}",
doc.notes
);
}
#[test]
fn a_zip_holding_one_fictionbook_is_a_wrapped_one() {
let doc = archive_of(&["__MACOSX/._mill.fb2", "mill.fb2"], "mill.fb2.zip");
assert_eq!(doc.kind, Kind::ZipWrappedFb2);
assert_eq!(doc.declared_ext.as_deref(), Some("zip"));
assert!(
doc.notes
.iter()
.any(|note| note.contains("one FictionBook")),
"{:?}",
doc.notes
);
assert!(!doc.notes.iter().any(|note| note.contains("named .")));
let two = archive_of(&["one.fb2", "two.fb2"], "pair.zip");
assert_eq!(two.kind, Kind::Unknown);
}
#[test]
fn a_zip_of_page_images_is_a_comic() {
let doc = archive_of(
&[
"Issue 1/",
"Issue 1/001.jpg",
"Issue 1/002.JPEG",
"Issue 1/003.png",
"Issue 1/004.gif",
"Issue 1/005.webp",
"ComicInfo.xml",
"__MACOSX/Issue 1/._001.jpg",
"Issue 1/Thumbs.db",
".DS_Store",
],
"issue-1.cbz",
);
assert_eq!(doc.kind, Kind::Cbz);
assert!(
!doc.notes.iter().any(|note| note.contains("named .")),
"{:?}",
doc.notes
);
}
#[test]
fn a_comic_saved_as_a_zip_is_remarked_on() {
let doc = archive_of(&["001.jpg", "002.jpg"], "issue-1.zip");
assert_eq!(doc.kind, Kind::Cbz);
assert!(
doc.notes.iter().any(|note| note.contains("comic")),
"{:?}",
doc.notes
);
}
#[test]
fn a_zip_with_anything_but_images_is_not_a_comic() {
let one_pdf = archive_of(&["001.jpg", "002.jpg", "extras.pdf"], "issue-1.cbz");
assert_eq!(one_pdf.kind, Kind::Unknown);
assert!(
one_pdf
.notes
.iter()
.any(|note| note.contains("none of them a book")),
"{:?}",
one_pdf.notes
);
let readme = archive_of(&["001.jpg", "readme.txt"], "issue-1.cbz");
assert_eq!(readme.kind, Kind::Unknown);
// A name that ends like an image without being one.
let lookalike = archive_of(&["001.jpg", "notes.jpg.txt"], "issue-1.cbz");
assert_eq!(lookalike.kind, Kind::Unknown);
}
#[test]
fn a_zip_with_no_pages_is_not_a_comic() {
let catalogue_only = archive_of(&["ComicInfo.xml", "pages/"], "issue-1.cbz");
assert_eq!(catalogue_only.kind, Kind::Unknown);
let empty = archive_of(&[], "issue-1.cbz");
assert_eq!(empty.kind, Kind::Unknown);
}
#[test]
fn a_truncated_comic_does_not_panic() {
let whole = zip::tests::zip(&["001.jpg", "002.png", "ComicInfo.xml"]);
for cut in 0..whole.len() {
let _ = sniff(&&whole[..cut], "issue-1.cbz");
}
}
}
crates/booksniff/src/zip.rs
+2 -2
View File
@@ -78,11 +78,11 @@ fn directory_start(source: &impl Bytes) -> Option<(usize, u64)> {
}
#[cfg(test)]
mod tests {
pub(crate) mod tests {
use super::*;
/// Build a zip holding `names`, each with empty stored content.
fn zip(names: &[&str]) -> Vec<u8> {
pub(crate) fn zip(names: &[&str]) -> Vec<u8> {
let mut out: Vec<u8> = Vec::new();
let mut offsets = Vec::new();
for name in names {
crates/booksniff/tests/cli.rs
+24
View File
@@ -64,6 +64,30 @@ fn several_files_come_back_as_several_records() {
assert_eq!(kinds, ["\"pdf\"", "\"djvu\""]);
}
#[test]
fn the_formats_read_from_the_front_come_back_by_their_wire_names() {
let rtf = written("letter.rtf", &padded(br"{\rtf1\ansi"));
let lit = written("book.lit", &padded(b"ITOLITLS"));
let fb2 = written(
"mill.fb2",
&padded(b"<?xml version=\"1.0\"?>\n<FictionBook xmlns=\"http://www.gribuser.ru/xml/fictionbook/2.0\">"),
);
let out = run(&[
rtf.to_str().expect("utf-8 path"),
lit.to_str().expect("utf-8 path"),
fb2.to_str().expect("utf-8 path"),
]);
assert_eq!(out.status.code(), Some(0));
let kinds: Vec<String> = String::from_utf8(out.stdout)
.expect("utf-8")
.lines()
.map(|line| {
serde_json::from_str::<serde_json::Value>(line).expect("valid JSON")["kind"].to_string()
})
.collect();
assert_eq!(kinds, ["\"rtf\"", "\"lit\"", "\"fb2\""]);
}
#[test]
fn a_file_that_is_not_there_is_an_io_error_and_the_rest_still_run() {
let path = written("kept.pdf", &padded(b"%PDF-1.7"));
crates/colophon/Cargo.toml
+21
View File
@@ -0,0 +1,21 @@
[package]
name = "colophon"
description = "The record a book prints in its own front matter."
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
repository.workspace = true
[dependencies]
catena-cli.workspace = true
clap.workspace = true
corpus-core.workspace = true
bibclean.workspace = true
regex.workspace = true
schemars.workspace = true
serde.workspace = true
serde_json.workspace = true
[lints]
workspace = true
crates/colophon/README.md
+151
View File
@@ -0,0 +1,151 @@
# colophon
The record a book prints in its own front matter.
Part of [catena](../../README.md).
## What it does
A book states what it is twice before the text starts. The title page carries
the title as the publisher set it and the name of whoever wrote it. The
copyright page carries the imprint, the year, the ISBN, and — in most
English-language books printed since the 1970s — the cataloguing block, which
is a catalogue record a librarian made from the manuscript, subject headings
and all.
So this reads the pages rather than the file. A PDF whose embedded title is
`Untitled` and whose filename says `Oxford` opens on *The First World War: A
Very Short Introduction*, by Michael Howard, and that is what comes back.
Every field names the page it was read off. Nothing is inferred: a scan that
begins at the preface printed no title page, and the record of that is an
empty one with every field listed in `unsettled`.
## A shelf of your own
A cataloguing block states subjects the way a national library does:
```text
Political science – Early works to 1800
English literature—Middle English, 1100–1500—History and criticism
```
Nobody browses by those. `--shelf` takes the closed list you actually file by
and returns the headings from it that the book's own subjects hit:
```sh
colophon --shelf shelf.txt book.md
```
```text
# one heading per line; `= terms` gives the words that match it,
# and a heading with no terms matches its own name.
Byzantine History = byzantium, byzantine, constantinople
Political Science = political science, politics
Ethics
```
The longest term wins, so `Ancient Philosophy` beats `Philosophy` and a period
of thought does not collapse into its parent. Matching is on whole words after
punctuation is folded away, because a block separates with em dashes, en
dashes, full stops and hyphens, and OCR turns any of those into any other.
Headings that match nothing come back under `unshelved`, kept as printed. A
subject quietly dropped reads exactly like a book that printed none.
## When you would use it
- **Cataloguing scans and secondhand files**, where the embedded metadata is
whatever the last tool to touch the file happened to write.
- **Checking a catalogue against the books themselves.** A record that
disagrees with the book's own title page is a record worth looking at.
- **Recovering an ISBN** that only exists in print, which turns a book with
no identifier into one [`bibfetch`](../bibfetch/README.md) can look up.
- **Getting subject headings** that nobody had to invent, from the
cataloguing block of the books that print one.
## Install
```sh
cargo install --git https://git.pumpin.online/funkpower/catena colophon
```
## Example
```
$ colophon first-world-war.md
{"v":1,"path":"first-world-war.md","title":"The First World War","subtitle":"A Very Short Introduction","authors":["Michael Howard"],"editors":[],"isbns":["9780192803535"],"publisher":"Oxford University Press","place":"Oxford","year":2002,"subjects":[],"read_from":{"authors":"title_page","isbns":"copyright_page","publisher":"copyright_page","subtitle":"title_page","title":"title_page","year":"copyright_page"},"unsettled":["subjects"],"pages_read":16}
```
A library at once, and the books that say nothing about themselves:
```sh
colophon library/*.md | jq -r 'select(.title == null) | .path'
```
An ISBN that only exists in print, looked up:
```sh
colophon book.md | jq -r '.isbns[]' | idnorm | bibfetch --polite you@example.org
```
## What it will not read
Two pages in a book's front matter look like a title page and are not, and
both of them are how a catalogue ends up naming the wrong book. They are
found and refused.
The **series advertisement** lists other titles in the same series in the
same shape a title page uses — a work in capitals, a person beside it —
sometimes for four pages, with only the first saying what it is. The
**uniform title** in a translation's cataloguing block is the original
title, printed in brackets above the real one as a filing heading.
## Asking a model about the rest
Roughly a third of books print nothing this can read: an extraction that
starts at chapter one has no front matter in it at all. `--prompt` writes a
question about exactly the fields left `unsettled`, and `--merge` folds the
answer back in.
```sh
colophon --prompt book.md | ollama run qwen3:8b > reply.json
colophon --merge reply.json book.md
```
Nothing is sent from here: the question goes to stdout, and the model is
yours. What a model answers is marked `model` in `read_from`, so a record
always says which of its fields were read off a page and which were
answered.
The pages the question shows have had the advertising taken out of them
first. Handing a model a series list is how a book becomes catalogued as
whichever of two hundred other titles the model's eye fell on.
## Where this stops
It reports what the book printed, not what is true. `THE HUMAN CONDITION` is
reported in the capitals it was set in, and straightening that is
[`bibclean`](../bibclean/README.md)'s job. What a file's own metadata claims
is [`bookmeta`](../bookmeta/README.md)'s. What a register says about the
ISBN is [`bibfetch`](../bibfetch/README.md)'s. Which series a book belongs to
is [`seriesof`](../seriesof/README.md)'s.
## Options
| Option | Effect |
|---|---|
| `--pages <COUNT>` | How many pages in from the front to read. Default `16` |
| `--prompt` | Write the question to put to a model about what the pages left unsettled |
| `--merge <FILE>` | Fill in from a model's reply and write the completed record |
| `--quiet` | Say nothing; the exit code is the answer |
| `--man` | Write this page as a manual page, in roff |
## Exit codes
| Code | Meaning |
|---|---|
| `0` | The book printed something about itself |
| `1` | It printed nothing this could read |
| `65` | The input was not readable as text |
| `74` | The file could not be read |
crates/colophon/src/cip.rs
+274
View File
@@ -0,0 +1,274 @@
//! The cataloguing-in-publication block, which is a catalogue record the
//! publisher printed inside the book.
//!
//! Since the 1970s most English-language publishers have printed the record
//! the Library of Congress made from the manuscript on the copyright page,
//! and since 2013 they have printed it in labelled fields. Either way it
//! names the author as a library files him, the title with its subtitle, the
//! ISBN, and the subject headings the book was classified under — which is
//! the one place in a book where its subjects are stated rather than
//! inferred.
//!
//! It is a record, so it is read as one. Nothing here guesses.
use std::sync::LazyLock;
use regex::Regex;
/// What the block said.
#[derive(Debug, Default, Clone, PartialEq)]
pub struct Cip {
/// Title, without its subtitle.
pub title: Option<String>,
/// Subtitle, where the title statement had one.
pub subtitle: Option<String>,
/// Authors, in reading order rather than filing order.
pub authors: Vec<String>,
/// Editors, where the work is an edited one.
pub editors: Vec<String>,
/// ISBNs printed inside the block.
pub isbns: Vec<String>,
/// Subject headings the book was classified under.
pub subjects: Vec<String>,
}
impl Cip {
fn is_empty(&self) -> bool {
self.title.is_none() && self.authors.is_empty() && self.subjects.is_empty()
}
}
static HEAD: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)catalog(?:u)?ing[-\s]in[-\s]publication(?:\s+data)?").expect("a valid pattern")
});
/// `B105.C477D46 1991` / `126—dc20` — the call number, which ends the block.
static CALL_NUMBER: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)^([A-Z]{1,3}\d{1,4}(\.\w+)*\s+\d{4}|\d{3}(\.\d+)?\s*[—–-]{1,2}\s*dc\d{2})")
.expect("a valid pattern")
});
/// A name as a library files it: `Howard, Michael, 1922- author.`
static FILED_NAME: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(
r"^(?P<last>[A-Z][\p{L}'’\-]+(?:\s+[A-Z][\p{L}'’\-]+)?),\s+(?P<first>[A-Z][\p{L}'’\-\.]*(?:\s+[A-Z][\p{L}'’\-\.]*){0,3})\
(?:,\s*(?:\d{4}\s*[-–]\s*\d{0,4}|b\.\s*\d{4}|d\.\s*\d{4}))?\
(?:,?\s*(?:author|editor|translator|compiler)s?\.?)?\.?$",
)
.expect("a valid pattern")
});
/// `[Sur la traduction. English]` — the uniform title of a translation,
/// printed above the title it was translated into.
static UNIFORM_TITLE: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^\\?\[.*\\?\]\.?$").expect("a valid pattern"));
static LABEL: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)^(names|title|other titles|description|identifiers|subjects|classification|uniform title)\s*:\s*(.*)$")
.expect("a valid pattern")
});
/// `1. Consciousness. 2. Philosophy of mind. I. Title.`
static NUMBERED_SUBJECTS: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^\s*1\.\s+\S").expect("a valid pattern"));
static SUBJECT_SPLIT: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"\s*\b\d{1,2}\.\s+").expect("a valid pattern"));
/// The added entries a subject list ends with: `I. Title. II. Series.`
static ADDED_ENTRY: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"\s+[IVX]+\.\s+.*$").expect("a valid pattern"));
static ISBN_LINE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)isbn[:\s]*([0-9][0-9\s\-–]{8,20}[0-9Xx])").expect("a valid pattern")
});
/// The responsibility statement's separator, and what OCR makes of it.
///
/// A slash between the title and who wrote it is a solidus in a serif face,
/// and page scanners read it as a capital I or a lowercase l about as often
/// as they read it correctly.
static RESPONSIBILITY: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"\s+[/Il\\|]\s+").expect("a valid pattern"));
static EDITION_TAIL: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)\s*[.,]?\s*[—–-]{1,2}\s*(\d+(st|nd|rd|th)|first|second|third|new|rev\.?|revised)\b.*$")
.expect("a valid pattern")
});
fn tidy(s: &str) -> String {
s.trim()
.trim_matches(|c: char| c == '\\' || c == '"')
.trim()
.trim_end_matches(['.', ',', ';', ':', '/'])
.trim()
.to_string()
}
/// Split a statement of responsibility into who wrote and who edited.
fn people(statement: &str) -> (Vec<String>, Vec<String>) {
let mut authors = Vec::new();
let mut editors = Vec::new();
for part in statement.split(';') {
let part = part.trim();
let lower = part.to_lowercase();
let (target, rest) = if let Some(rest) = lower.strip_prefix("edited by ") {
(&mut editors, &part[part.len() - rest.len()..])
} else if lower.starts_with("translated by ")
|| lower.starts_with("with ")
|| lower.starts_with("foreword by ")
|| lower.starts_with("introduction by ")
{
continue;
} else {
(&mut authors, part)
};
for name in rest.split(" and ").flat_map(|n| n.split(',')) {
let name = tidy(name);
if name.len() >= 3 && name.split_whitespace().count() <= 5 {
target.push(name);
}
}
}
(authors, editors)
}
fn filed_to_reading(line: &str) -> Option<String> {
let caps = FILED_NAME.captures(line)?;
Some(format!(
"{} {}",
tidy(caps.name("first")?.as_str()),
tidy(caps.name("last")?.as_str())
))
}
fn split_title(statement: &str) -> (Option<String>, Option<String>, Vec<String>, Vec<String>) {
let mut pieces = RESPONSIBILITY.splitn(statement, 2);
let title_part = pieces.next().unwrap_or_default();
let (authors, editors) = pieces.next().map(people).unwrap_or_default();
let title_part = EDITION_TAIL.replace(title_part, "");
let mut halves = title_part.splitn(2, " : ");
let title = tidy(halves.next().unwrap_or_default());
let subtitle = halves.next().map(tidy).filter(|s| !s.is_empty());
(
Some(title).filter(|t| t.len() >= 2),
subtitle,
authors,
editors,
)
}
fn subjects_from(line: &str) -> Vec<String> {
let trimmed = ADDED_ENTRY.replace(line, "");
SUBJECT_SPLIT
.split(&trimmed)
.map(tidy)
.filter(|s| s.len() >= 3 && s.split_whitespace().count() <= 12)
.collect()
}
fn isbns_from(line: &str) -> Vec<String> {
ISBN_LINE
.captures_iter(line)
.filter_map(|c| {
let digits = bibclean::isbn::digits(&c[1]);
bibclean::isbn::valid(&digits).then_some(digits)
})
.collect()
}
/// Read the CIP block out of a book's front matter, if it printed one.
pub fn read(front: &str) -> Option<Cip> {
let start = HEAD.find(front)?.end();
let mut block: Vec<&str> = Vec::new();
for line in front[start..].lines().skip(1) {
let line = line.trim();
if line.is_empty() {
if block.len() > 2 {
break;
}
continue;
}
if CALL_NUMBER.is_match(line) {
break;
}
block.push(line);
if block.len() >= 24 {
break;
}
}
if block.is_empty() {
return None;
}
let mut cip = Cip::default();
let labelled = block.iter().any(|l| {
LABEL
.captures(l)
.is_some_and(|c| matches!(c[1].to_lowercase().as_str(), "names" | "title" | "subjects"))
});
if labelled {
for line in &block {
let Some(caps) = LABEL.captures(line) else {
continue;
};
let value = caps[2].trim();
match caps[1].to_lowercase().as_str() {
"names" => {
for name in value.split('|') {
if let Some(read) = filed_to_reading(tidy(name).as_str()) {
cip.authors.push(read);
}
}
}
"title" => {
let (title, subtitle, authors, editors) = split_title(value);
cip.title = title;
cip.subtitle = subtitle;
if cip.authors.is_empty() {
cip.authors = authors;
}
cip.editors = editors;
}
"identifiers" => cip.isbns.extend(isbns_from(value)),
"subjects" => {
let value = value.trim_start_matches("LCSH:").trim();
for subject in value.split('|') {
let subject = tidy(subject);
if subject.len() >= 3 {
cip.subjects.push(subject);
}
}
}
_ => {}
}
}
} else {
let mut seen_title = false;
for line in &block {
if UNIFORM_TITLE.is_match(line) {
continue;
}
if !seen_title {
if let Some(name) = filed_to_reading(line) {
cip.authors.push(name);
continue;
}
if ISBN_LINE.is_match(line) || line.to_lowercase().starts_with("isbn") {
cip.isbns.extend(isbns_from(line));
continue;
}
if RESPONSIBILITY.is_match(line) || line.contains(" : ") {
let (title, subtitle, authors, editors) = split_title(line);
cip.title = title;
cip.subtitle = subtitle;
if cip.authors.is_empty() {
cip.authors = authors;
}
cip.editors = editors;
seen_title = true;
continue;
}
}
if NUMBERED_SUBJECTS.is_match(line) {
cip.subjects = subjects_from(line);
}
cip.isbns.extend(isbns_from(line));
}
}
cip.isbns.sort();
cip.isbns.dedup();
if cip.is_empty() { None } else { Some(cip) }
}
crates/colophon/src/imprint.rs
+113
View File
@@ -0,0 +1,113 @@
//! The copyright page: who published it, where, when, and under what ISBN.
//!
//! Everything on this page is formula. The same dozen sentences appear in
//! every book printed in English, which is what makes them worth matching
//! exactly rather than interpreting.
use std::sync::LazyLock;
use regex::Regex;
/// What the copyright page stated.
#[derive(Debug, Default, Clone, PartialEq)]
pub struct Imprint {
/// The press.
pub publisher: Option<String>,
/// Where it publishes from, where the page named one place.
pub place: Option<String>,
/// The year of this edition, preferring first publication.
pub year: Option<u16>,
/// Every checksum-valid ISBN printed on the page, in the order printed.
pub isbns: Vec<String>,
}
static PUBLISHED_BY: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?im)^\s*(?:first\s+)?published(?:\s+(?:in\s+)?(?:1[5-9]|20)\d{2})?\s+by\s+(?P<who>[^,.\n]{3,60})")
.expect("a valid pattern")
});
static PRESS_LINE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?im)^\s*(?P<who>[A-Z][^\n]{2,48}?(?:University Press|Press|Publishers|Publishing|Books|Verlag))\s*[,.]?\s*$")
.expect("a valid pattern")
});
/// A sentence that merely mentions a press is not the press.
///
/// `Oxford is a registered trade mark of Oxford University Press` ends in a
/// publisher's name and is a legal notice.
static NOT_AN_IMPRINT: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)( is | are | was |registered|trade mark|trademark|imprint of|division of|©|\(c\)|@|;|\d{4})")
.expect("a valid pattern")
});
static FIRST_PUBLISHED: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)first published[^.\n]{0,40}?((?:1[5-9]|20)\d{2})").expect("a valid pattern")
});
static COPYRIGHT_YEAR: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)(?:©|\(c\)|copyright)\s*(?:by\s+)?((?:1[5-9]|20)\d{2})")
.expect("a valid pattern")
});
static ISBN: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)isbn(?:[-\s]?1[03])?[:\s]*([0-9][0-9\s\-–—]{8,24}[0-9Xx])")
.expect("a valid pattern")
});
/// `Oxford : Oxford University Press, 2007` — the labelled CIP's description,
/// which states all three at once.
static DESCRIPTION: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?im)^\s*Description\s*:\s*(?P<place>[^:|\n]{2,40}?)\s*:\s*(?P<who>[^,|\n]{3,60}?)\s*,\s*(?P<year>(?:1[5-9]|20)\d{2})")
.expect("a valid pattern")
});
fn tidy(s: &str) -> String {
s.trim()
.trim_end_matches([',', '.', ';'])
.trim()
.to_string()
}
/// Every checksum-valid ISBN announced in a piece of text.
pub fn isbns(text: &str) -> Vec<String> {
let mut out: Vec<String> = Vec::new();
for caps in ISBN.captures_iter(text) {
let digits = bibclean::isbn::digits(&caps[1]);
if matches!(digits.len(), 10 | 13)
&& bibclean::isbn::valid(&digits)
&& !out.contains(&digits)
{
out.push(digits);
}
}
out
}
/// Read the copyright page.
pub fn read(text: &str) -> Imprint {
let mut out = Imprint {
isbns: isbns(text),
..Imprint::default()
};
if let Some(caps) = DESCRIPTION.captures(text) {
out.place = Some(tidy(&caps["place"]));
out.publisher = Some(tidy(&caps["who"]));
out.year = caps["year"].parse().ok();
return out;
}
out.publisher = PUBLISHED_BY
.captures_iter(text)
.chain(PRESS_LINE.captures_iter(text))
.map(|caps| tidy(&caps["who"]))
.find(|who| !NOT_AN_IMPRINT.is_match(who) && who.split_whitespace().count() <= 8);
out.year = FIRST_PUBLISHED
.captures(text)
.or_else(|| COPYRIGHT_YEAR.captures(text))
.and_then(|caps| caps[1].parse().ok());
out.place = text
.lines()
.map(str::trim)
.find(|line| crate::title::PLACE.is_match(line) && line.split_whitespace().count() <= 4)
.and_then(|line| {
crate::title::PLACE
.captures(line)
.map(|caps| tidy(&caps["city"]))
});
out
}
crates/colophon/src/lib.rs
+362
View File
@@ -0,0 +1,362 @@
//! The record a book prints in its own front matter.
//!
//! A book states its own title twice before the text starts, and states it
//! properly: on the title page, set by the publisher, and again in the
//! cataloguing block on the copyright page, where a librarian has already
//! decided what the title is, who the author is, and what the book is about.
//! A file's embedded metadata may say `Untitled` and a filename may say
//! `Oxford`, but the printed page says *The First World War: A Very Short
//! Introduction*, by Michael Howard.
//!
//! So this reads the pages, not the file. What comes back is what the book
//! printed, with the page it was printed on named for every field, and
//! nothing at all where the book printed nothing — a scan that starts at the
//! preface has no title page, and the honest record of that is an empty one.
//!
//! Two traps are worth naming, because both of them are how a reader that
//! does not know better ends up cataloguing the wrong book:
//!
//! - **The series advertisement.** A paperback's front matter lists two
//! hundred other books in the same series, in the same shape a title page
//! uses: a work in capitals, a person beside it. A title taken off that
//! page belongs to somebody else. Those pages are found and refused,
//! including the ones that continue the list without repeating its heading.
//! - **The uniform title.** A translation's cataloguing block prints the
//! original title in brackets above the real one. It is a heading, not the
//! title of the book in your hand.
//!
//! What this does not do is decide whether what the book printed is any
//! good. `THE HUMAN CONDITION` is reported in the capitals it was set in;
//! straightening that is [`bibclean`]'s job. Nor does it look anything up:
//! an ISBN read off the copyright page is a key, and [`bibfetch`] is what
//! turns a key into a catalogue record.
//!
//! [`bibclean`]: https://git.pumpin.online/funkpower/catena
//! [`bibfetch`]: https://git.pumpin.online/funkpower/catena
pub mod cip;
pub mod imprint;
pub mod page;
pub mod prompt;
pub mod shelf;
pub mod title;
use std::collections::BTreeMap;
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
/// How many pages in from the front to read.
///
/// Front matter that has not finished by here is not front matter. Reading
/// further costs nothing but finds chapter headings, which look enough like
/// titles to be believed.
pub const PAGES: usize = 16;
/// Which page a field was read off.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "snake_case")]
pub enum Source {
/// The cataloguing-in-publication block: a catalogue record, printed.
Cip,
/// The title page, as the publisher set it.
TitlePage,
/// The copyright page, outside the cataloguing block.
CopyrightPage,
/// A model, answering about the pages this could not read. Never
/// written by a read: only [`prompt::merge`] puts it on a record.
Model,
}
impl Source {
/// What it is called.
pub fn name(self) -> &'static str {
match self {
Source::Cip => "cip",
Source::TitlePage => "title_page",
Source::CopyrightPage => "copyright_page",
Source::Model => "model",
}
}
}
/// What one book printed about itself.
///
/// Every field is written on every record, empty where the book said
/// nothing, and every field that was read names the page it came from in
/// `read_from`. The fields a catalogue needs and this could not find are
/// listed in `unsettled`, so an absence is legible rather than inferred from
/// a missing key.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct Printed {
/// Record version.
#[serde(default = "version")]
pub v: u32,
/// The file this was read from, where it was read from one.
pub path: Option<String>,
/// The title, without its subtitle.
pub title: Option<String>,
/// The subtitle, where the page set one.
pub subtitle: Option<String>,
/// Whoever the book says wrote it, in reading order.
pub authors: Vec<String>,
/// Whoever the book says edited it.
pub editors: Vec<String>,
/// Checksum-valid ISBNs printed in the front matter, as printed.
pub isbns: Vec<String>,
/// The press.
pub publisher: Option<String>,
/// Where it publishes from.
pub place: Option<String>,
/// The year of publication, preferring first publication.
pub year: Option<u16>,
/// The subject headings the cataloguing block filed it under, as printed.
pub subjects: Vec<String>,
/// Those headings filed onto a caller's own shelf, when one was given.
///
/// Empty without `--shelf`, and empty too when the book's subjects match
/// nothing on it — which is an answer, and the reason `unshelved` exists.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub shelved: Vec<String>,
/// Printed headings that matched nothing on the shelf, kept verbatim so a
/// subject is never silently dropped.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub unshelved: Vec<String>,
/// Which page each field was read off.
pub read_from: BTreeMap<String, Source>,
/// The fields a catalogue wants that this book did not print.
pub unsettled: Vec<String>,
/// How many pages were read.
pub pages_read: usize,
}
fn version() -> u32 {
corpus_core::V
}
impl corpus_core::Record for Printed {
fn version(&self) -> u32 {
self.v
}
}
impl Printed {
/// Did the book print anything at all?
pub fn is_empty(&self) -> bool {
self.title.is_none()
&& self.authors.is_empty()
&& self.isbns.is_empty()
&& self.subjects.is_empty()
}
fn note(&mut self, field: &str, source: Source) {
self.read_from.insert(field.to_string(), source);
}
}
/// Which page holds the title, given where the copyright page is.
///
/// The title page is the last page before the copyright page that is a page
/// of set lines rather than a list or a contents — publishers have put it
/// there for two centuries. Where there is no copyright page to count back
/// from, the first such page before the text starts is all there is to go on.
fn title_page(pages: &[page::Page], copyright: Option<usize>) -> Option<usize> {
let usable = |i: usize| pages.get(i).is_some_and(|p| p.kind == page::Kind::Title);
if let Some(c) = copyright {
if let Some(found) = (0..c).rev().find(|i| usable(*i)) {
return Some(found);
}
}
pages
.iter()
.position(|p| p.kind == page::Kind::Text)
.map_or_else(
|| (0..pages.len()).find(|i| usable(*i)),
|end| (0..end).find(|i| usable(*i)),
)
}
/// The longest printing of the same title anywhere in the front matter.
///
/// A title page breaks its title across lines and sets it in capitals; the
/// half title two leaves earlier prints the whole of it on one line, with
/// its subtitle, in the case the publisher meant. They are the same title,
/// and the unbroken one is the one to keep.
fn longest_printing<'a>(pages: &'a [page::Page], title: &str) -> Option<&'a str> {
let wanted = squash(title);
if wanted.len() < 4 {
return None;
}
pages
.iter()
.filter(|p| matches!(p.kind, page::Kind::Title | page::Kind::Copyright))
.flat_map(|p| p.lines.iter())
.filter(|line| {
let candidate = squash(line);
candidate.starts_with(&wanted) && candidate.len() > wanted.len()
})
.min_by_key(|line| line.len())
.map(String::as_str)
}
/// A string reduced to the letters and digits in it, for comparing two
/// printings of one title across a change of case and punctuation.
fn squash(s: &str) -> String {
s.chars()
.filter(|c| c.is_alphanumeric())
.flat_map(char::to_lowercase)
.collect()
}
/// Read a book's front matter, and file its subjects onto a shelf.
///
/// `budget` is how many pages in to read; [`PAGES`] is the sensible default.
/// Passing a [`Shelf`](shelf::Shelf) fills `shelved` and `unshelved`; without
/// one, `subjects` is still read and both stay empty.
pub fn read_onto(text: &str, budget: usize, shelf: Option<&shelf::Shelf>) -> Printed {
let mut out = read(text, budget);
if let Some(shelf) = shelf.filter(|s| !s.is_empty()) {
let filed = shelf.file(&out.subjects);
if !filed.on.is_empty() {
out.note("shelved", Source::Cip);
}
out.shelved = filed.on;
out.unshelved = filed.unfiled;
}
out
}
/// Read a book's front matter.
///
/// `budget` is how many pages in to read; [`PAGES`] is the sensible default.
pub fn read(text: &str, budget: usize) -> Printed {
let pages = page::split(text, budget);
let mut out = Printed {
v: version(),
path: None,
title: None,
subtitle: None,
authors: Vec::new(),
editors: Vec::new(),
isbns: Vec::new(),
publisher: None,
place: None,
year: None,
subjects: Vec::new(),
shelved: Vec::new(),
unshelved: Vec::new(),
read_from: BTreeMap::new(),
unsettled: Vec::new(),
pages_read: pages.len(),
};
let front: String = pages
.iter()
.map(|p| p.text.as_str())
.collect::<Vec<_>>()
.join("\n");
let copyright = pages.iter().position(|p| p.kind == page::Kind::Copyright);
// The cataloguing block is a record somebody else already made, so it
// outranks the title page wherever the two disagree.
if let Some(found) = cip::read(&front) {
if let Some(t) = found.title {
out.title = Some(t);
out.note("title", Source::Cip);
}
if let Some(s) = found.subtitle {
out.subtitle = Some(s);
out.note("subtitle", Source::Cip);
}
if !found.authors.is_empty() {
out.authors = found.authors;
out.note("authors", Source::Cip);
}
if !found.editors.is_empty() {
out.editors = found.editors;
out.note("editors", Source::Cip);
}
if !found.subjects.is_empty() {
out.subjects = found.subjects;
out.note("subjects", Source::Cip);
}
out.isbns = found.isbns;
if !out.isbns.is_empty() {
out.note("isbns", Source::Cip);
}
}
if let Some(page) = title_page(&pages, copyright).and_then(|i| title::read(&pages[i].lines)) {
if out.title.is_none() {
out.title = match page
.title
.as_deref()
.and_then(|t| longest_printing(&pages, t))
{
Some(longer) => {
let (title, subtitle) = longer
.split_once(" : ")
.or_else(|| longer.split_once(": "))
.map_or((longer.to_string(), None), |(head, tail)| {
(head.trim().to_string(), Some(tail.trim().to_string()))
});
out.subtitle = subtitle;
Some(title)
}
None => page.title,
};
if out.title.is_some() {
out.note("title", Source::TitlePage);
}
if out.subtitle.is_none() && page.subtitle.is_some() {
out.subtitle = page.subtitle;
out.note("subtitle", Source::TitlePage);
}
}
if out.authors.is_empty() && !page.authors.is_empty() {
out.authors = page.authors;
out.note("authors", Source::TitlePage);
}
if out.editors.is_empty() && !page.editors.is_empty() {
out.editors = page.editors;
out.note("editors", Source::TitlePage);
}
}
let imprint_text = copyright.map_or(front.as_str(), |i| pages[i].text.as_str());
let stated = imprint::read(imprint_text);
for isbn in stated.isbns {
if !out.isbns.contains(&isbn) {
out.isbns.push(isbn);
out.read_from
.entry("isbns".into())
.or_insert(Source::CopyrightPage);
}
}
if let Some(publisher) = stated.publisher {
out.publisher = Some(publisher);
out.note("publisher", Source::CopyrightPage);
}
if let Some(place) = stated.place {
out.place = Some(place);
out.note("place", Source::CopyrightPage);
}
if let Some(year) = stated.year {
out.year = Some(year);
out.note("year", Source::CopyrightPage);
}
for (field, missing) in [
("title", out.title.is_none()),
("authors", out.authors.is_empty() && out.editors.is_empty()),
("isbns", out.isbns.is_empty()),
("publisher", out.publisher.is_none()),
("year", out.year.is_none()),
("subjects", out.subjects.is_empty()),
] {
if missing {
out.unsettled.push(field.to_string());
}
}
out
}
crates/colophon/src/main.rs
+167
View File
@@ -0,0 +1,167 @@
//! The record a book prints in its own front matter.
use std::path::PathBuf;
use std::process::ExitCode;
use clap::Parser;
use corpus_core::exit;
/// Read what a book prints about itself in its opening pages.
///
/// The input is the book as markdown — the whole of it, or as much of the
/// front as you care to pass. The title page and the cataloguing block on
/// the copyright page are read, and what they state comes back as one
/// record, with the page each field was read off named in `read_from`.
///
/// Nothing is inferred. A scan that starts at chapter one prints no title
/// page, and what comes back then is a record with nothing in it and every
/// field named in `unsettled` — which is the answer, and a usable one.
#[derive(Parser)]
#[command(
version,
about,
after_help = "\
Exit codes: 0 the book printed something about itself, 1 it printed nothing
this could read, 65 the input was not readable as text, 74 the file could not
be read.
Example:
colophon book.md | jq -r '[.title, (.authors|join(\", \"))] | @tsv'
colophon *.md | jq -r 'select(.unsettled | index(\"title\")) | .path'
colophon book.md | jq -r '.isbns[]' | idnorm | bibfetch"
)]
struct Args {
/// The books to read. Standard input when none are given.
paths: Vec<PathBuf>,
/// Say nothing; the exit code is the answer.
#[arg(long, short)]
quiet: bool,
/// Write the question to put to a model about the fields these pages
/// did not settle, instead of the record. Nothing is sent: it goes to
/// stdout. One book at a time, because the pages differ.
#[arg(long)]
prompt: bool,
/// Fill in what the pages did not settle from a model's reply, written
/// earlier by `--prompt`, and write the completed record.
#[arg(long, value_name = "FILE")]
merge: Option<PathBuf>,
/// How many pages in from the front to read.
#[arg(long, value_name = "COUNT", default_value_t = colophon::PAGES)]
pages: usize,
/// A closed list of headings to file the book's printed subjects under.
///
/// One heading per line, optionally `Heading = term, term` to give the
/// words that should match it; a heading with no terms matches its own
/// name. A cataloguing block states subjects in Library of Congress form,
/// which is no use as a shelf, so this says what the shelf is. What
/// matches nothing comes back under `unshelved` rather than being dropped.
#[arg(long, value_name = "FILE")]
shelf: Option<PathBuf>,
}
fn main() -> ExitCode {
let args: Args = match catena_cli::parse() {
Ok(args) => args,
Err(code) => return code,
};
if args.pages == 0 {
eprintln!("colophon: --pages 0 would read nothing");
return ExitCode::from(exit::USAGE);
}
if args.prompt && args.merge.is_some() {
eprintln!("colophon: --prompt composes a question and --merge answers one");
return ExitCode::from(exit::USAGE);
}
if (args.prompt || args.merge.is_some()) && args.paths.len() > 1 {
eprintln!("colophon: one book at a time for --prompt and --merge");
return ExitCode::from(exit::USAGE);
}
// Read the shelf before any book, so a bad path fails at once rather than
// after the whole library has been read with no shelf applied.
let shelf = match args.shelf.as_deref() {
None => None,
Some(path) => match std::fs::read_to_string(path) {
Ok(text) => {
let s = colophon::shelf::Shelf::parse(&text);
if s.is_empty() {
eprintln!("colophon: {} lists no headings", path.display());
return ExitCode::from(exit::USAGE);
}
Some(s)
}
Err(e) => {
eprintln!("colophon: {}: {e}", path.display());
return ExitCode::from(exit::USAGE);
}
},
};
let inputs: Vec<Option<PathBuf>> = if args.paths.is_empty() {
vec![None]
} else {
args.paths.iter().cloned().map(Some).collect()
};
let mut records = Vec::with_capacity(inputs.len());
let mut last_text = String::new();
for path in inputs {
let text = match catena_cli::read_text(path.as_deref()) {
Ok(text) => text,
Err((code, why)) => {
eprintln!("colophon: {why}");
return ExitCode::from(code);
}
};
let mut record = colophon::read_onto(&text, args.pages, shelf.as_ref());
record.path = path.as_deref().map(|p| p.display().to_string());
records.push(record);
last_text = text;
}
if args.prompt {
let record = &records[0];
return match colophon::prompt::compose(record, &last_text, args.pages) {
// Nothing left to ask is not a failure, and a question nobody
// needs would waste a model's time.
None => ExitCode::from(exit::NO),
Some(question) => catena_cli::text("colophon", &question),
};
}
if let Some(path) = args.merge.as_deref() {
let reply = match std::fs::read_to_string(path) {
Ok(reply) => reply,
Err(e) => {
eprintln!("colophon: {}: {e}", path.display());
return ExitCode::from(exit::IO);
}
};
match colophon::prompt::merge(&mut records[0], &reply) {
Err(why) => {
eprintln!("colophon: {why}");
return ExitCode::from(exit::DATA);
}
Ok(0) => eprintln!("colophon: the reply filled nothing in"),
Ok(filled) => eprintln!("colophon: {filled} field(s) filled in from the reply"),
}
}
let anything = records.iter().any(|r| !r.is_empty());
if args.quiet {
return ExitCode::from(if anything { exit::OK } else { exit::NO });
}
if !anything {
eprintln!("colophon: nothing in these pages says what the book is");
for record in &records {
let _ = catena_cli::stream("colophon", std::slice::from_ref(record));
}
return ExitCode::from(exit::NO);
}
catena_cli::stream("colophon", &records)
}
crates/colophon/src/page.rs
+278
View File
@@ -0,0 +1,278 @@
//! The opening of a book, cut into pages and each page named for what it is.
//!
//! Front matter is not prose and does not read like it. A title page is four
//! lines and no sentence; a copyright page is a dense block of formulae that
//! recur from publisher to publisher; a series advertisement is a list of
//! other people's books. Telling them apart is most of the work, because the
//! title that matters is on one of them and the titles that will mislead you
//! are on another.
use std::sync::LazyLock;
use regex::Regex;
/// What a page of front matter is.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Kind {
/// Nothing legible on it.
Blank,
/// Other books, advertised. The trap: it names two hundred works that
/// are not this one, in the same shapes a title page uses.
SeriesList,
/// Publication facts: the imprint, the copyright, the ISBN, the CIP block.
Copyright,
/// The book's own table of contents.
Contents,
/// Few lines, no sentences — a title page, a half title, or a dedication.
Title,
/// Running text. The front matter is over.
Text,
}
/// One page, cleaned of what the converter added.
#[derive(Debug, Clone)]
pub struct Page {
/// Page number as the marker gave it, or position when there was none.
pub number: u32,
/// Legible lines, in order, with markup and typesetting slugs removed.
pub lines: Vec<String>,
/// The page's text as it arrived, for the block readers.
pub text: String,
/// What the page is.
pub kind: Kind,
}
static PAGE_MARKER: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?im)^\s*<!--\s*page\s+(\d+)\s*-->\s*$").expect("a valid pattern")
});
static IMAGE: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"!\[[^\]]*\]\([^)]*\)").expect("a valid pattern"));
static HEADING: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^#{1,6}\s+").expect("a valid pattern"));
/// `DESA01 19/9/06 6:02 PM Page i` — the typesetter's slug, printed by
/// accident and carried by every page of some books.
static SLUG: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)^[a-z]{2,10}[-_ ]?\d{1,6}\s+\d{1,2}/\d{1,2}/\d{2,4}.*page\s+[ivxlcdm\d]+\s*$")
.expect("a valid pattern")
});
/// A converted EPUB's navigation: `[Begin Reading](dedication.xhtml#ded)`,
/// and the page links some converters leave behind.
static NAVIGATION: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)^(<|\[[^\]]*\]\([^)]*\)|https?://|www\.|previous page|next page|page_\w+)")
.expect("a valid pattern")
});
static FOLIO: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^[\d\s.\-–—ivxlcdmIVXLCDM|]+$").expect("a valid pattern"));
static BLANK_NOTE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)^(this page (has been )?intentionally left blank|blank page)\.?$")
.expect("a valid pattern")
});
static LIST_CUE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(
r"(?i)(available now|available soon|also by|also in this series|also available|\
other titles|other books|forthcoming|in the same series|praise for|advance praise|\
very short introductions|series editors?|titles in this series)",
)
.expect("a valid pattern")
});
static COPYRIGHT_CUE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(
r"(?i)(all rights reserved|first published|first edition|copyright\s*[©®(]?|[©®]\s*\d{4}|\(c\)\s*\d{4}|\
library of congress|british library|catalog(u)?ing[- ]in[- ]publication|\
printed (in|and bound)|isbn|no part of this (book|publication)|\
typeset (in|by)|a cip (record|catalogue))",
)
.expect("a valid pattern")
});
static CONTENTS_CUE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)^(table of )?contents$|\.{4,}\s*\d+\s*$").expect("a valid pattern")
});
/// A line of a series list: a work, then whoever wrote it, on one line.
static LISTED_WORK: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"^[A-Z][A-Za-z'’\-]*(?:[ ,][A-Z][A-Za-z'’\-]*){0,7}\s+[A-Z][a-z'’\-]+\s+[A-Z][A-Za-z'’\-]+\.?$")
.expect("a valid pattern")
});
/// A person's name alone on a line.
pub static PERSON: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"^(?:[A-Z][A-Za-z'’\-\.]{0,20}\s+){1,3}[A-Z][A-Za-z'’\-]{1,25}\.?$")
.expect("a valid pattern")
});
/// Letters spread across a line by a PDF's glyph positions, put back together.
///
/// `A N E W T R A N SL AT ION` is one word to a reader and eleven tokens to a
/// parser, and the line it ruins is usually the title.
pub fn despread(line: &str) -> String {
let tokens: Vec<&str> = line.split_whitespace().collect();
if tokens.len() < 4 {
return line.to_string();
}
let singles = tokens.iter().filter(|t| t.chars().count() == 1).count();
if singles * 2 < tokens.len() {
return line.to_string();
}
let mut out: Vec<String> = Vec::new();
let mut run = String::new();
for token in tokens {
if token.chars().count() <= 3 && token.chars().all(char::is_alphabetic) {
run.push_str(token);
} else {
if !run.is_empty() {
out.push(std::mem::take(&mut run));
}
out.push(token.to_string());
}
}
if !run.is_empty() {
out.push(run);
}
out.join(" ")
}
/// Is the line so damaged that reading it would invent something?
///
/// A font whose glyphs map into the private use area comes out of extraction
/// as a line of unassigned code points. It looks like text to a length check
/// and is not text.
fn illegible(line: &str) -> bool {
let total = line.chars().filter(|c| !c.is_whitespace()).count();
if total == 0 {
return true;
}
let bad = line
.chars()
.filter(|c| {
let n = *c as u32;
(0xE000..=0xF8FF).contains(&n) || *c == '\u{FFFD}'
})
.count();
bad * 5 > total
}
fn clean_line(raw: &str) -> Option<String> {
let without_images = IMAGE.replace_all(raw, " ");
let stripped = HEADING.replace(without_images.trim(), "");
let line = despread(
stripped
.trim()
.trim_matches(|c| c == '*' || c == '_' || c == '|')
.trim(),
);
if line.is_empty()
|| illegible(&line)
|| SLUG.is_match(&line)
|| NAVIGATION.is_match(&line)
|| FOLIO.is_match(&line)
|| BLANK_NOTE.is_match(&line)
{
return None;
}
Some(line)
}
fn looks_like_prose(line: &str) -> bool {
let words = line.split_whitespace().count();
words > 14 || (words > 8 && line.trim_end().ends_with(['.', ';', ',']))
}
fn classify(lines: &[String], text: &str) -> Kind {
if lines.is_empty() {
return Kind::Blank;
}
if COPYRIGHT_CUE.find_iter(text).count() >= 2 {
return Kind::Copyright;
}
if LIST_CUE.is_match(text) && lines.len() >= 5 {
return Kind::SeriesList;
}
let listed = lines.iter().filter(|l| LISTED_WORK.is_match(l)).count();
if lines.len() >= 6 && listed * 5 >= lines.len() * 2 {
return Kind::SeriesList;
}
if lines.iter().take(3).any(|l| CONTENTS_CUE.is_match(l))
|| lines.iter().filter(|l| CONTENTS_CUE.is_match(l)).count() >= 4
{
return Kind::Contents;
}
if lines.len() <= 12 && !lines.iter().any(|l| looks_like_prose(l)) {
return Kind::Title;
}
Kind::Text
}
/// Cut the opening into pages and name each one.
///
/// Page markers are used where the converter left them. Where it did not —
/// an EPUB has no pages to mark — the blank lines are the only boundary
/// there is, and blocks stand in for pages. That is weaker, which is why a
/// book converted from EPUB usually comes back with nothing but its ISBN:
/// the metadata is in its OPF, and [`bookmeta`] reads that.
///
/// [`bookmeta`]: https://git.pumpin.online/funkpower/catena
pub fn split(text: &str, budget: usize) -> Vec<Page> {
let mut pages = Vec::new();
if PAGE_MARKER.is_match(text) {
let mut last_end = 0;
let mut number = 0u32;
for m in PAGE_MARKER.captures_iter(text) {
let whole = m.get(0).expect("group zero is always there");
if number > 0 || whole.start() > 0 {
push(&mut pages, number, &text[last_end..whole.start()]);
}
number = m[1].parse().unwrap_or(number + 1);
last_end = whole.end();
if pages.len() > budget {
break;
}
}
push(&mut pages, number, &text[last_end..]);
} else {
for (i, block) in text.split("\n\n").enumerate() {
push(&mut pages, i as u32 + 1, block);
if pages.len() > budget {
break;
}
}
}
pages.truncate(budget);
continuations(&mut pages);
pages
}
fn push(pages: &mut Vec<Page>, number: u32, text: &str) {
let lines: Vec<String> = text.lines().filter_map(clean_line).collect();
let kind = classify(&lines, text);
pages.push(Page {
number,
lines,
text: text.to_string(),
kind,
});
}
/// A series list runs over several pages and only the first one announces
/// itself. The pages after it are the same list and have to be refused too,
/// or the title taken off one of them is another book's.
fn continuations(pages: &mut [Page]) {
for i in 1..pages.len() {
if pages[i - 1].kind != Kind::SeriesList {
continue;
}
let page = &pages[i];
if page.kind == Kind::Blank || page.lines.len() < 6 {
continue;
}
// Only a page that is itself a list continues one. Counting bare
// names here would take a title page for advertising, because a
// title set in capitals and broken across lines is, line by line,
// the same shape as a person's name.
let listed = page
.lines
.iter()
.filter(|l| LISTED_WORK.is_match(l))
.count();
if page.lines.len() >= 6 && listed * 2 >= page.lines.len() {
pages[i].kind = Kind::SeriesList;
}
}
}
crates/colophon/src/prompt.rs
+194
View File
@@ -0,0 +1,194 @@
//! A question to put to a model, for what the pages did not settle.
//!
//! Nothing here sends anything. The prompt is composed and printed, and what
//! happens to it is the caller's business — their model, their machine,
//! their GPU. The tool stays what every tool here is: the same output for
//! the same input, on every machine, with no network and nothing installed.
//!
//! What the model is shown is not the front matter as it arrived. The pages
//! that advertise other books are taken out first, because handing them to a
//! model is how a book ends up catalogued as whichever of two hundred other
//! titles the model's eye fell on. It is shown the pages that describe this
//! book, told what was already read off them, and asked only for the rest.
use crate::{Printed, Source, page};
/// The version of the prompt below.
///
/// Bumped when the wording changes in a way that could change an answer, so
/// a caller keeping results can tell which prompt produced them.
pub const PROMPT_VERSION: u32 = 1;
/// How much of the front matter to show. Enough for a title page, a
/// copyright page and a contents; not enough to wander into the text.
const SHOWN: usize = 6000;
/// The fields a model may be asked for.
const ASKABLE: &[&str] = &["title", "authors", "publisher", "year", "subjects"];
fn known(record: &Printed) -> String {
let mut out = String::new();
if let Some(title) = &record.title {
out.push_str(&format!("title already read: {title}\n"));
}
if !record.authors.is_empty() {
out.push_str(&format!(
"authors already read: {}\n",
record.authors.join("; ")
));
}
if !record.isbns.is_empty() {
out.push_str(&format!(
"isbn printed in the book: {}\n",
record.isbns.join(", ")
));
}
if let Some(publisher) = &record.publisher {
out.push_str(&format!("publisher already read: {publisher}\n"));
}
out
}
/// The front matter with the advertising taken out.
pub fn readable_pages(text: &str, budget: usize) -> String {
let mut out = String::new();
for page in page::split(text, budget) {
if matches!(page.kind, page::Kind::SeriesList | page::Kind::Blank) || page.lines.is_empty()
{
continue;
}
out.push_str(&format!("--- page {}\n", page.number));
for line in &page.lines {
out.push_str(line);
out.push('\n');
}
out.push('\n');
if out.len() >= SHOWN {
break;
}
}
out.truncate(SHOWN);
out
}
/// Compose the question, or decline to.
///
/// `None` when the pages settled everything a model could help with, because
/// there is nothing to ask and a call to a model would be waste.
pub fn compose(record: &Printed, text: &str, budget: usize) -> Option<String> {
let wanted: Vec<&str> = ASKABLE
.iter()
.copied()
.filter(|field| record.unsettled.iter().any(|f| f == field))
.collect();
if wanted.is_empty() {
return None;
}
let pages = readable_pages(text, budget);
if pages.trim().is_empty() {
return None;
}
let mut out = String::from(
"Below are the opening pages of one book, as text. Pages advertising \
other books have already been removed.\n\n",
);
out.push_str(&pages);
out.push('\n');
let already = known(record);
if !already.is_empty() {
out.push_str("Read off these pages already:\n");
out.push_str(&already);
out.push('\n');
}
out.push_str(&format!("wanted: {}\n\n", wanted.join(", ")));
out.push_str(
"Reply with one JSON object and nothing else, using only the keys \
listed as wanted. `title` is the book's own title, without its \
subtitle. `authors` is a list of the people who wrote it, as the \
book writes their names. `publisher` is the press. `year` is a \
four-digit number. `subjects` is a list of at most five subject \
headings for what the book is about.\n\n\
Answer only from the pages above. Use null for anything they do not \
settle: a guess is worse than an absence, because nobody downstream \
can tell them apart. A title listed beside other titles is another \
book's, not this one's.\n",
);
Some(out)
}
/// Fill in what the pages did not settle from a model's reply.
///
/// Only the fields that were unsettled are filled, and each one is marked as
/// having come from a model — a record that cannot say which of its fields
/// were read and which were answered is a record nobody can apply a policy
/// to. Returns how many fields were filled.
pub fn merge(record: &mut Printed, reply: &str) -> Result<usize, String> {
let start = reply.find('{').ok_or("the reply holds no JSON object")?;
let end = reply.rfind('}').ok_or("the reply holds no JSON object")?;
let parsed: serde_json::Value = serde_json::from_str(&reply[start..=end])
.map_err(|e| format!("the reply is not JSON: {e}"))?;
let object = parsed.as_object().ok_or("the reply is not a JSON object")?;
let mut filled = 0;
let wanted = |record: &Printed, field: &str| record.unsettled.iter().any(|f| f == field);
if wanted(record, "title") {
if let Some(title) = object.get("title").and_then(|v| v.as_str()) {
let title = title.trim();
if !title.is_empty() {
record.title = Some(title.to_string());
record.read_from.insert("title".into(), Source::Model);
filled += 1;
}
}
}
if wanted(record, "publisher") {
if let Some(publisher) = object.get("publisher").and_then(|v| v.as_str()) {
let publisher = publisher.trim();
if !publisher.is_empty() {
record.publisher = Some(publisher.to_string());
record.read_from.insert("publisher".into(), Source::Model);
filled += 1;
}
}
}
if wanted(record, "year") {
if let Some(year) = object.get("year").and_then(serde_json::Value::as_u64) {
if (1400..=2200).contains(&year) {
record.year = Some(year as u16);
record.read_from.insert("year".into(), Source::Model);
filled += 1;
}
}
}
for (field, into) in [("authors", true), ("subjects", false)] {
if !wanted(record, field) {
continue;
}
let Some(list) = object.get(field).and_then(|v| v.as_array()) else {
continue;
};
let values: Vec<String> = list
.iter()
.filter_map(|v| v.as_str())
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect();
if values.is_empty() {
continue;
}
if into {
record.authors = values;
} else {
record.subjects = values;
}
record.read_from.insert(field.into(), Source::Model);
filled += 1;
}
record
.unsettled
.retain(|field| !record.read_from.contains_key(field));
Ok(filled)
}
crates/colophon/src/shelf.rs
+264
View File
@@ -0,0 +1,264 @@
//! Filing a book's printed subject headings onto a shelf someone actually has.
//!
//! A cataloguing block states subjects in Library of Congress form, which is
//! precise, deep, and no use as a shelf. Real blocks from real books read:
//!
//! ```text
//! Political science – Early works to 1800
//! English literature—Middle English, 1100–1500—History and criticism
//! Methodology. 2.Anthropic principle
//! Hobbes, Thomas, 1588–1679. Leviathan
//! ```
//!
//! Nobody browses by those. A library is browsed by a short closed list its
//! owner chose, and the job here is to get from the first to the second.
//!
//! The mapping is deliberately dumb: a heading is matched against the terms of
//! the shelf, longest first, and the first hit wins. It infers nothing, invents
//! nothing, and returns nothing at all rather than guess — a heading that
//! matches no term is reported unfiled, with its raw text kept, because a
//! subject silently dropped is indistinguishable from a book with no subjects.
use std::collections::BTreeSet;
/// A closed list of headings, with the words that file a book under each.
///
/// Parsed from a plain text file: one heading per line, optionally followed by
/// `=` and the comma-separated terms that should match it. A heading with no
/// terms matches its own name.
///
/// ```text
/// Byzantine History = byzantium, byzantine, constantinople, eastern roman
/// Patristics = church fathers, patristic, patrology
/// Ethics
/// ```
#[derive(Debug, Clone, Default)]
pub struct Shelf {
/// (term, heading), longest term first so the most specific wins.
terms: Vec<(String, String)>,
headings: BTreeSet<String>,
}
impl Shelf {
/// Read a shelf from the list format above. Blank lines and `#` comments
/// are skipped.
pub fn parse(text: &str) -> Self {
let mut terms: Vec<(String, String)> = Vec::new();
let mut headings = BTreeSet::new();
for line in text.lines() {
let line = line.trim();
if line.is_empty() || line.starts_with('#') {
continue;
}
let (heading, rest) = match line.split_once('=') {
Some((h, r)) => (h.trim(), r.trim()),
None => (line, ""),
};
if heading.is_empty() {
continue;
}
headings.insert(heading.to_string());
terms.push((fold(heading), heading.to_string()));
for term in rest.split(',') {
let term = fold(term);
if !term.is_empty() {
terms.push((term, heading.to_string()));
}
}
}
// Longest first: "ancient philosophy" must beat "philosophy", or every
// period of thought files under the parent and the shelf does no work.
terms.sort_by(|a, b| b.0.len().cmp(&a.0.len()).then_with(|| a.0.cmp(&b.0)));
terms.dedup_by(|a, b| a.0 == b.0);
Self { terms, headings }
}
/// Whether the shelf lists no headings, in which case it files nothing.
pub fn is_empty(&self) -> bool {
self.terms.is_empty()
}
/// Whether a name is one of the shelf's own headings.
pub fn has(&self, heading: &str) -> bool {
self.headings.contains(heading)
}
/// The heading a single raw subject files under, if any.
pub fn file_one(&self, raw: &str) -> Option<&str> {
let hay = fold(raw);
if hay.is_empty() {
return None;
}
self.terms
.iter()
.find(|(term, _)| contains_word(&hay, term))
.map(|(_, heading)| heading.as_str())
}
/// File a book's raw headings, in the shelf's own order, without repeats.
///
/// Returns the headings that matched and the raw subjects that did not, so
/// a caller can tell "this book is not about anything on the shelf" from
/// "this book printed no subjects".
pub fn file(&self, raw: &[String]) -> Filed {
let mut on = Vec::new();
let mut unfiled = Vec::new();
for subject in raw {
match self.file_one(subject) {
Some(h) => {
if !on.iter().any(|x: &String| x == h) {
on.push(h.to_string());
}
}
None => unfiled.push(subject.clone()),
}
}
Filed { on, unfiled }
}
}
/// What a shelf made of a book's printed subjects.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct Filed {
/// Headings from the shelf, in the order the book's subjects hit them.
pub on: Vec<String>,
/// Subjects that matched no term, kept verbatim.
pub unfiled: Vec<String>,
}
/// Lowercase, and reduce every run of non-alphanumerics to one space.
///
/// Cataloguing blocks separate with em dashes, en dashes, full stops and plain
/// hyphens, sometimes three of them in one heading, and OCR turns any of those
/// into any other. None of that should decide whether a book gets shelved.
fn fold(s: &str) -> String {
let mut out = String::with_capacity(s.len());
let mut space = true; // no leading space
for c in s.chars() {
if c.is_alphanumeric() {
for l in c.to_lowercase() {
out.push(l);
}
space = false;
} else if !space {
out.push(' ');
space = true;
}
}
while out.ends_with(' ') {
out.pop();
}
out
}
/// Whether `needle` appears in `hay` on whole-word boundaries.
///
/// Substring matching would file "Islam" under any book mentioning "Islamic"
/// — which is right — but would also file "Art" under "Cartography", which is
/// not. Both strings are already folded, so the boundary is a space.
fn contains_word(hay: &str, needle: &str) -> bool {
if needle.is_empty() || needle.len() > hay.len() {
return false;
}
hay.match_indices(needle).any(|(i, _)| {
let before = i == 0 || hay.as_bytes()[i - 1] == b' ';
let end = i + needle.len();
let after = end == hay.len() || hay.as_bytes()[end] == b' ';
before && after
})
}
#[cfg(test)]
mod tests {
use super::*;
fn shelf() -> Shelf {
Shelf::parse(
"# a shelf\n\
Philosophy = philosophy, philosophical\n\
Ancient Philosophy = ancient philosophy, plato, aristotle\n\
Political Science = political science, politics\n\
Theology = theology, christianity\n\
Byzantine History = byzantium, byzantine, constantinople\n\
Ethics\n",
)
}
#[test]
fn a_heading_with_no_terms_matches_its_own_name() {
assert_eq!(shelf().file_one("Ethics — early works"), Some("Ethics"));
}
#[test]
fn the_most_specific_term_wins() {
// Both "ancient philosophy" and "philosophy" match; the longer one has
// to win or every period files under the parent.
assert_eq!(
shelf().file_one("Ancient philosophy—History"),
Some("Ancient Philosophy")
);
}
#[test]
fn real_cataloguing_headings_file() {
let s = shelf();
for (raw, want) in [
(
"Political science – Early works to 1800",
"Political Science",
),
("Political science — Philosophy", "Political Science"),
("Byzantine Empire—History—527-1081", "Byzantine History"),
("Christianity and culture", "Theology"),
] {
assert_eq!(s.file_one(raw), Some(want), "{raw}");
}
}
#[test]
fn a_dash_is_a_dash_however_it_was_printed() {
// em, en, hyphen, and the double hyphen OCR leaves behind.
let s = shelf();
for raw in [
"Philosophy—Introductions",
"Philosophy – Introductions",
"Philosophy - Introductions",
"Philosophy -- Introductions",
"Philosophy. Introductions",
] {
assert_eq!(s.file_one(raw), Some("Philosophy"), "{raw}");
}
}
#[test]
fn a_term_must_match_a_whole_word() {
// "politics" must not file a book on "geopolitics", and "art" must not
// file one on cartography.
let s = Shelf::parse("Art\nPolitical Science = politics\n");
assert_eq!(s.file_one("Cartography—History"), None);
assert_eq!(s.file_one("Art—Byzantine"), Some("Art"));
}
#[test]
fn what_matches_nothing_is_reported_rather_than_dropped() {
let filed = shelf().file(&[
"Political science – Early works to 1800".into(),
"Selectivity (Psychology)".into(),
]);
assert_eq!(filed.on, ["Political Science"]);
assert_eq!(filed.unfiled, ["Selectivity (Psychology)"]);
}
#[test]
fn a_heading_is_not_repeated() {
let filed = shelf().file(&["Philosophy—Ancient".into(), "Philosophy—Modern".into()]);
assert_eq!(filed.on, ["Philosophy"]);
}
#[test]
fn an_empty_shelf_files_nothing_and_says_so() {
let s = Shelf::parse(" \n# nothing but a comment\n");
assert!(s.is_empty());
assert_eq!(s.file(&["Philosophy".into()]).unfiled, ["Philosophy"]);
}
}
crates/colophon/src/title.rs
+594
View File
@@ -0,0 +1,594 @@
//! The title page, read as the block it is.
//!
//! A title page is laid out rather than written: the title, sometimes broken
//! across lines by nothing but the width of the page, then whoever wrote it,
//! then the publisher at the foot. Reading it is a matter of knowing where
//! the title stops, which is at the first line that is a person or a press.
use std::sync::LazyLock;
use regex::Regex;
use crate::page::PERSON;
/// What a title page printed.
#[derive(Debug, Default, Clone, PartialEq)]
pub struct TitlePage {
/// The title, with lines the typesetter broke put back together.
pub title: Option<String>,
/// What followed it, where the page gave a subtitle.
pub subtitle: Option<String>,
/// Whoever it says wrote the book.
pub authors: Vec<String>,
/// Whoever it says edited it.
pub editors: Vec<String>,
}
/// `(1714)`, `, 1922-2019` — when a title page dates its author.
static LIFESPAN: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(
r"\s*[(,]\s*(?:1[0-9]|20)\d{2}\s*(?:[-\u{2013}]\s*(?:(?:1[0-9]|20)\d{2})?)?\s*\)?\s*$",
)
.expect("a valid pattern")
});
/// `350 BC`, `1971` — a date alone on a line, which is not a title.
static BARE_DATE: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"(?i)^\d{1,4}\s*(bc|ad|bce|ce)?\.?$").expect("a valid pattern"));
/// `Copyright © 2013 by`, `\u{a9} 2013 Richard Swinburne`, `Copyright 1971 by`.
/// `1. General Remarks`, `2 James W. McAllister` — a numbered heading.
static NUMBERED_HEADING: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^\s*\d+\s*[.)]\s+\S|^\s*\d+\s{2,}\S").expect("a valid pattern"));
/// `5222 What Is This`, `827 North Fourth Street` — a number that is not a
/// year, leading a line.
static LEADING_NUMBER: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^\s*(\d{1,5})\s+\S").expect("a valid pattern"));
/// `Page xxxvi`, `Chapter I`, `Fig. 3` — a marker, not a name.
static MARKER: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(
r"(?i)^\s*(page|pages|p\.|pp\.|fig\.|figure|plate|chapter|ch\.|section|part)\s+[\divxlcm]",
)
.expect("a valid pattern")
});
/// `Sidebottom, Harry. Ancient warfare` — a bibliography entry, which front
/// matter prints in its cataloguing block and in its further reading.
static CATALOGUE_ENTRY: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"^[A-Z][A-Za-z'\-]+,\s+[A-Z][A-Za-z'.\-]*\.\s+\S").expect("a valid pattern")
});
/// `To Mackenzie,` — a dedication, set alone on its own page.
static DEDICATION: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)^\s*(to|for|in memory of)\s+\S.*,\s*$").expect("a valid pattern")
});
static COPYRIGHT_NOTICE: LazyLock<Regex> = LazyLock::new(|| {
// The year need not follow the symbol directly: "(c) Alfredo Ferrarin
// 2004" puts the holder between them, and was read as a title.
Regex::new(r"(?i)(\u{a9}|\(c\)|copyright)[^\n]{0,60}?(1[0-9]|20)\d{2}")
.expect("a valid pattern")
});
/// `Edited by`, with whoever it was set on the line beneath.
static ROLE_ONLY: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)^(edited|translated|written|selected|compiled)\s+by:?$")
.expect("a valid pattern")
});
static BYLINE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)^(?P<role>by|edited by|written by|translated by|selected by|compiled by)\s+(?P<who>.{3,90})$")
.expect("a valid pattern")
});
/// A press, an imprint, or the line that announces one. The foot of the page.
pub static IMPRINT: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(
r"(?i)\b(university press|[a-z]+ press\b|publish(?:er|ers|ing)\b|publications\b|\
verlag\b|éditions\b|editions\b|books? (?:ltd|inc|group)\b|\
routledge|blackwell|wiley|springer|palgrave|macmillan|penguin|\
random house|harpercollins|w\.? ?w\.? norton|doubleday|bloomsbury|eerdmans|\
brill\b|sage\b|vintage\b|knopf|faber|crossway|zondervan|clarendon|\
foundation\b|institute\b|society\b|trust\b|distributed by|\
cover (design|art|photograph)|jacket design|thank you for buying|\
typeset|designed by|illustrated by|printed by|\
copyright|all rights reserved|printed in)\b",
)
.expect("a valid pattern")
});
/// A city a publisher puts on a title page, alone on its line.
pub static PLACE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(
r"(?i)^(?P<city>oxford|cambridge|london|new york|new haven|chicago|boston|princeton|\
berlin|paris|munich|rome|madrid|grand rapids|edinburgh|dublin|toronto|sydney|\
melbourne|minneapolis|philadelphia|san francisco|los angeles|leiden|amsterdam|\
copenhagen|stockholm|moscow|beijing|tokyo|delhi|nairobi)\b[\s,·•|and]*",
)
.expect("a valid pattern")
});
/// Words that name a part of a book rather than the book.
static STRUCTURAL: &[&str] = &[
"abbreviations",
"about the author",
"abstract",
"acknowledgements",
"acknowledgments",
"afterword",
"appendix",
"author",
"authors",
"bibliography",
"contents",
"dedication",
"epigraph",
"epilogue",
"foreword",
"frontispiece",
"glossary",
"half title",
"illustrations",
"index",
"introduction",
"keywords",
"list of contributors",
"list of illustrations",
"maps",
"notes",
"postscript",
"preface",
"prologue",
"subtitle",
"table of contents",
"title",
"title page",
"translator's note",
];
fn is_structural(line: &str) -> bool {
let norm: String = line
.chars()
.filter(|c| c.is_alphanumeric() || c.is_whitespace())
.collect::<String>()
.trim()
.to_lowercase();
let norm = strip_folio(&norm);
STRUCTURAL.binary_search(&norm).is_ok()
}
/// A heading with its page number taken off either end.
///
/// Front matter is folioed in roman, and the folio sits on the same line as
/// the heading often enough to matter: `vi CONTENTS` and `viii CONTENTS`
/// are contents pages, and were being read as titles because they are not the
/// bare word.
fn strip_folio(norm: &str) -> &str {
let is_folio = |w: &str| {
!w.is_empty()
&& (w.chars().all(|c| c.is_ascii_digit())
|| w.chars()
.all(|c| matches!(c, 'i' | 'v' | 'x' | 'l' | 'c' | 'd' | 'm')))
};
let mut out = norm.trim();
loop {
let trimmed = match out.split_once(' ') {
Some((head, rest)) if is_folio(head) => rest.trim_start(),
_ => match out.rsplit_once(' ') {
Some((rest, tail)) if is_folio(tail) => rest.trim_end(),
_ => return out,
},
};
if trimmed == out {
return out;
}
out = trimmed;
}
}
/// Lines that are printed matter rather than anything a book is called.
///
/// Every one of these was read as a title off a real book in a real library:
/// a copyright notice, a translator's credit, a part heading in brackets, a
/// span of HTML a converter left behind, and a volume entry from a collected
/// edition's own list of volumes.
fn is_furniture(line: &str) -> bool {
let t = line.trim();
if t.starts_with('[') || t.starts_with('\\') {
return true;
}
// A converter that left its markup in. Nothing printed on a title page is
// a tag, a stylesheet declaration, a markdown link or a filename, and a
// title that appears to be one is a fragment of the conversion rather than
// of the book. Rejecting only the first such line is no use: the next line
// of a mangled page is mangled too, so the whole shape has to go.
if t.contains("<span")
|| t.contains("<div")
|| t.contains("<p ")
|| t.contains("</")
|| t.contains("](") // a markdown link the converter split
|| t.contains("&quot;")
|| t.contains("&amp;")
|| t.contains("mso-") // Word's stylesheet, via HTML export
|| t.contains("font-family")
|| t.contains("font-size")
|| t.contains("style=")
|| t.contains("_epub_")
|| t.contains(".html")
|| t.contains(".xhtml")
{
return true;
}
// Sentences a copyright page sets, which are not notices and so not caught
// by the notice pattern.
if NUMBERED_HEADING.is_match(t) || MARKER.is_match(t) || CATALOGUE_ENTRY.is_match(t) {
return true;
}
// A line opening with a number that is not a year. "5222 What Is This" and
// "827 North Fourth Street" are a stray folio and an address; "1984
// Revisited" is a book, and the difference is whether the number could be
// the year of one.
if let Some(c) = LEADING_NUMBER.captures(t) {
let n: u32 = c[1].parse().unwrap_or(0);
if !(1000..=2099).contains(&n) {
return true;
}
}
if DEDICATION.is_match(t) {
return true;
}
let lower = t.to_lowercase();
for phrase in [
"moral rights of the author",
"no part of this publication",
"british library cataloguing",
"library of congress cataloging",
"printed in the united",
"typeset by",
"a cip catalogue record",
] {
if lower.contains(phrase) {
return true;
}
}
let low = t.to_lowercase();
// The notice, not the word. "Copyright and Its Discontents" is a book;
// "Copyright © 2013 by" is the page it would be printed on. What tells
// them apart is the symbol or the year, never the word alone.
if COPYRIGHT_NOTICE.is_match(t)
|| low.contains("all rights reserved")
|| low.starts_with("first published")
|| low.starts_with("reprinted")
{
return true;
}
// "Translated from the French by Willard R. Trask" — a credit, not a name
// and not a title. ROLE_ONLY catches the bare "Translated by"; this is the
// whole sentence, which a title page sets on one line.
if low.starts_with("translated from") || low.starts_with("translated by") {
return true;
}
// "Vol. 11. HENRY JAMES, JR." — an entry in a collected edition's list of
// its own volumes, printed in the front matter of every volume.
if low.starts_with("vol.") || low.starts_with("volume ") && low.contains('.') {
return true;
}
false
}
fn people(who: &str) -> Vec<String> {
who.split(&[';', '&'][..])
.flat_map(|part| part.split(" and "))
.map(|name| name.trim().trim_end_matches(['.', ',']).trim().to_string())
.filter(|name| name.len() >= 3 && name.split_whitespace().count() <= 5)
.collect()
}
/// Words a line ends on when the typesetter broke a title across it.
///
/// `Esotericism in` / `Western Culture` is one title set on two lines, and
/// nothing but the dangling preposition says so.
const CONTINUES: &[&str] = &[
"a", "an", "and", "at", "between", "by", "for", "from", "in", "into", "of", "on", "or", "the",
"to", "under", "with", "without",
];
/// Endings no surname has and every abstract noun does.
///
/// Two capitalised words look exactly like a name — `Martin Cohen` and
/// `Political Philosophy` are the same shape — and on a title page the
/// second one is the top half of the title. English morphology is what
/// tells them apart without a dictionary of names to consult.
const NOT_A_NAME: &[&str] = &[
"ance", "ence", "ics", "ism", "ity", "ment", "ness", "ography", "ology", "osophy", "sion",
"tion", "ture",
];
fn could_be_a_person(line: &str) -> bool {
let line = LIFESPAN.replace(line, "");
let line = line.trim().trim_end_matches('.');
if !PERSON.is_match(line) {
return false;
}
let tokens: Vec<&str> = line.split_whitespace().collect();
if tokens.len() > 4 {
return false;
}
!tokens.iter().any(|token| {
let word = token.trim_end_matches('.').to_lowercase();
word.len() > 4 && NOT_A_NAME.iter().any(|end| word.ends_with(end))
})
}
fn wraps(line: &str) -> bool {
if line.ends_with('-') {
return true;
}
line.rsplit_once(char::is_whitespace)
.is_some_and(|(_, last)| CONTINUES.contains(&last.to_lowercase().as_str()))
}
/// Put a title the page broke across lines back together.
fn unwrap_lines(block: &[&str]) -> Vec<String> {
let mut out: Vec<String> = Vec::new();
for line in block {
match out.last_mut() {
Some(previous) if wraps(previous) => {
if previous.ends_with('-') {
previous.pop();
previous.push_str(line);
} else {
previous.push(' ');
previous.push_str(line);
}
}
_ => out.push((*line).to_string()),
}
}
out
}
/// Read a title page, or decline to.
///
/// The page is taken as a block that ends at the imprint — a press, a place,
/// or the first line of running prose. What is left is the title, whoever
/// wrote it, and nothing else, because that is all a title page holds.
///
/// Where the byline sits is the publisher's choice and both choices are
/// common, so it is looked for at the top of the block and at the bottom and
/// nowhere between: a name in the middle of a title page is a title that
/// happens to be two capitalised words.
///
/// Declines when the block is empty or runs past five lines, at which point
/// this is a page of something else and a title taken off it would be a
/// guess with a page number attached.
pub fn read(lines: &[String]) -> Option<TitlePage> {
let mut out = TitlePage::default();
let mut block: Vec<&str> = Vec::new();
let mut waiting: Option<String> = None;
for line in lines.iter().take(14) {
// `Edited by` on its own line, with the names set under it.
if let Some(role) = waiting.take() {
let who = people(line);
if !who.is_empty() {
if role.starts_with("edited") {
out.editors.extend(who);
} else if !role.starts_with("translated") {
out.authors.extend(who);
}
continue;
}
}
if is_structural(line) || is_furniture(line) || BARE_DATE.is_match(line) {
if block.is_empty() {
continue;
}
break;
}
if let Some(caps) = BYLINE.captures(line) {
let role = caps["role"].to_lowercase();
let who = people(&caps["who"]);
if who.is_empty() {
waiting = Some(role);
continue;
}
if role.starts_with("edited") {
out.editors.extend(who);
} else if !role.starts_with("translated") {
out.authors.extend(who);
}
continue;
}
if ROLE_ONLY.is_match(line) {
waiting = Some(line.to_lowercase());
continue;
}
if IMPRINT.is_match(line) || PLACE.is_match(line) {
if block.is_empty() {
continue;
}
break;
}
block.push(line);
if block.len() > 5 {
return None;
}
}
if block.is_empty() {
return None;
}
// The byline is above the title or below it, never inside it.
if out.authors.is_empty() && out.editors.is_empty() && block.len() > 1 {
if could_be_a_person(block[0]) {
out.authors = people(block[0]);
block.remove(0);
} else if could_be_a_person(block[block.len() - 1]) {
out.authors = people(block[block.len() - 1]);
block.pop();
}
}
let units = unwrap_lines(&block);
let mut title = units.first()?.clone();
let mut subtitle = units.get(1).cloned();
let split = title
.split_once(" : ")
.or_else(|| title.split_once(": "))
.map(|(head, tail)| (head.trim().to_string(), tail.trim().to_string()));
if let Some((head, tail)) = split {
title = head;
subtitle = Some(tail);
}
// A fragment is not a title. Two letters of one, left behind by a font
// the extractor could not read, is worse than nothing: it is a title
// page's authority attached to noise.
out.title = Some(title).filter(|t| {
t.chars().filter(|c| c.is_alphabetic()).count() >= 4
&& t.split_whitespace()
.any(|w| w.chars().filter(|c| c.is_alphabetic()).count() >= 3)
});
out.subtitle = subtitle.filter(|s| s.len() >= 2);
out.title.as_ref()?;
Some(out)
}
#[cfg(test)]
mod furniture_tests {
use super::*;
/// Every string here was read as a title off a book in a real library.
#[test]
fn printed_matter_is_not_a_title() {
for line in [
"vi CONTENTS",
"viii CONTENTS",
"xiv ACKNOWLEDGEMENTS",
"12 INDEX",
"Translated from the French by Willard R. Trask",
"Copyright © 2013 by Barry Davies, BEM",
"© Alfredo Ferrarin 2004",
"\u{a9} Cambridge University Press 2011",
"[PART ONE:",
"Piper Verlag GmbH, Munich<span style=\"font-size\">",
"Vol. 11. HENRY JAMES, JR.",
"All rights reserved",
"First published 1971",
] {
assert!(
is_structural(line) || is_furniture(line),
"read as a title: {line:?}"
);
}
}
#[test]
fn a_real_title_survives_all_of_that() {
for line in [
"The First World War",
"A Very Short Introduction",
"Christianity and Classical Culture",
"The Problems of Philosophy",
"Volume and Void",
"Contents of Consciousness",
"August 1914",
"Copyright and Its Discontents",
] {
assert!(
!(is_structural(line) || is_furniture(line)),
"rejected a real title: {line:?}"
);
}
}
#[test]
fn a_folio_comes_off_either_end() {
assert_eq!(strip_folio("vi contents"), "contents");
assert_eq!(strip_folio("contents vi"), "contents");
assert_eq!(strip_folio("contents"), "contents");
// A title that happens to be roman numerals throughout is not a folio
// with nothing left; it keeps whatever it had.
assert_eq!(strip_folio("i"), "i");
}
}
#[cfg(test)]
mod conversion_tests {
use super::*;
/// A mangled page is mangled on every line, so rejecting the first
/// leftover only moves the misread down one. Each of these is the line
/// colophon fell through to after the line above it was rejected.
#[test]
fn what_a_converter_left_behind_is_never_a_title() {
for line in [
"[PART ONE:",
"RULERS AND REDNECKS](Webb_9780767922951_epub_p01_r1.html)",
"[AFRICAN SLAVERY IN AMERICA - \\[1775\\]](index_split_004.html)",
"Piper Verlag GmbH, Munich<span style=\"font-family:&quot;Times\"",
"mso-bidi-font-family:&quot;Times New Roman&quot;;mso-ansi-language",
"The moral rights of the author have been asserted",
"No part of this publication may be reproduced",
"British Library Cataloguing in Publication Data",
"Typeset by SPI Publisher Services",
] {
assert!(is_furniture(line), "read as a title: {line:?}");
}
}
#[test]
fn a_title_that_merely_sounds_like_the_copyright_page_survives() {
for line in [
"Copyright and Its Discontents",
"The Moral Rights of Man",
"Printed Matter",
"A Publication of the Institute",
] {
assert!(!is_furniture(line), "rejected a real title: {line:?}");
}
}
}
#[cfg(test)]
mod review_pile_tests {
use super::*;
/// Everything here was proposed as a title by a real run over this
/// library, and sat in the review queue waiting for a person to say no.
#[test]
fn what_the_review_pile_was_full_of() {
for line in [
"1. General Remarks",
"2 James W. McAllister",
"5222 What Is This",
"827 North Fourth Street",
"Page xxxvi",
"Page xxxviii",
"Chapter I",
"title",
"Sidebottom, Harry. Ancient warfare",
"To Mackenzie,",
] {
assert!(
is_structural(line) || is_furniture(line),
"still read as a title: {line:?}"
);
}
}
/// And none of it catches a book. The numbers are the delicate part: a
/// year can open a title, and a folio cannot.
#[test]
fn a_book_is_still_a_book() {
for line in [
"1984 Revisited",
"1914: The Year the World Ended",
"Chapter and Verse",
"Part of the Furniture",
"To Kill a Mockingbird",
"For Whom the Bell Tolls",
"The Title of This Book",
"Pages from a Cold Island",
"Sections of a Broken Whole",
"In Memory of Bread",
] {
assert!(
!(is_structural(line) || is_furniture(line)),
"rejected a real title: {line:?}"
);
}
}
}
crates/colophon/tests/frontmatter.rs
+299
View File
@@ -0,0 +1,299 @@
//! What real front matter looks like, and what has to come off it.
//!
//! Every case here is a shape taken from a book that was catalogued wrongly
//! without it.
use colophon::{PAGES, Source, read};
/// A paperback series prints two hundred other titles before its own, in the
/// same shape a title page uses. Catalogued off that page, this book is by
/// whichever author the reader's eye landed on.
#[test]
fn a_series_advertisement_is_not_the_title_page() {
let front = "\
<!-- Page 2 -->
The First World War: A Very Short Introduction
<!-- Page 3 -->
Very Short Introductions available now:
ANARCHISM Colin Ward
ANCIENT EGYPT Ian Shaw
ARISTOTLE Jonathan Barnes
THE ELEMENTS Philip Ball
CONSCIOUSNESS Susan Blackmore
COSMOLOGY Peter Coles
<!-- Page 4 -->
CLASSICS Mary Beard
DADA AND SURREALISM David Hopkins
DARWIN Jonathan Howard
THE DEAD SEA SCROLLS Timothy Lim
RUSSIAN LITERATURE Catriona Kelly
<!-- Page 6 -->
Michael Howard
THE FIRST
WORLD WAR
A Very Short Introduction
<!-- Page 7 -->
Great Clarendon Street, Oxford ox2 6dp
Oxford University Press is a department of the University of Oxford.
First published 2002
All rights reserved
ISBN 978-0-19-280353-5
";
let found = read(front, PAGES);
// The title page breaks the title across two lines and sets it in
// capitals; the half title prints the whole of it, cased as the
// publisher meant. Both are this book, and the unbroken one is kept.
assert_eq!(found.title.as_deref(), Some("The First World War"));
assert_eq!(found.subtitle.as_deref(), Some("A Very Short Introduction"));
assert_eq!(found.authors, vec!["Michael Howard"]);
assert_eq!(found.isbns, vec!["9780192803535"]);
assert_eq!(found.year, Some(2002));
assert_eq!(found.read_from.get("title"), Some(&Source::TitlePage));
}
/// The cataloguing block is a catalogue record somebody else already made.
/// The solidus between the title and who wrote it is read as a capital I by
/// every scanner ever built.
#[test]
fn a_cataloguing_block_outranks_the_page_it_is_printed_on() {
let front = "\
<!-- Page 4 -->
Consciousness Explained
Daniel C. Dennett
<!-- Page 5 -->
Copyright © 1991 by Daniel C. Dennett
All rights reserved
First published 1991
Library of Congress Cataloging-in-Publication Data
Dennett, Daniel C.
Consciousness explained I Daniel C. Dennett. — 1st ed.
p. cm.
Includes bibliographical references and index.
ISBN 0-316-18065-3
1. Consciousness. 2. Philosophy of mind. I. Title.
B105.C477D46 1991
126—dc20
";
let found = read(front, PAGES);
assert_eq!(found.title.as_deref(), Some("Consciousness explained"));
assert_eq!(found.authors, vec!["Daniel C. Dennett"]);
assert_eq!(found.read_from.get("title"), Some(&Source::Cip));
assert_eq!(found.subjects, vec!["Consciousness", "Philosophy of mind"]);
assert_eq!(found.isbns, vec!["0316180653"]);
}
/// Since 2013 the block comes in labelled fields, and the subject headings
/// with it — the one place a book states what it is about.
#[test]
fn the_labelled_block_gives_its_subjects() {
let front = "\
<!-- Page 4 -->
Library of Congress Cataloging-in-Publication Data
Names: Howard, Michael, 1922- author.
Title: The First World War : a very short introduction / Michael Howard.
Description: Oxford : Oxford University Press, 2007.
Identifiers: ISBN 9780192803535
Subjects: LCSH: World War, 1914-1918. | Europe--History--1871-1918.
All rights reserved
";
let found = read(front, PAGES);
assert_eq!(found.title.as_deref(), Some("The First World War"));
assert_eq!(found.subtitle.as_deref(), Some("a very short introduction"));
assert_eq!(found.authors, vec!["Michael Howard"]);
assert_eq!(found.publisher.as_deref(), Some("Oxford University Press"));
assert_eq!(found.place.as_deref(), Some("Oxford"));
assert_eq!(found.year, Some(2007));
assert_eq!(found.subjects.len(), 2);
}
/// A translation's block prints the original title in brackets above the
/// real one. It is a heading, not the book in your hand.
#[test]
fn a_uniform_title_is_not_the_title() {
let front = "\
<!-- Page 4 -->
All rights reserved
Library of Congress Cataloging-in-Publication Data
Ricoeur, Paul.
[Sur la traduction. English]
On translation / Paul Ricoeur ; translated by Eileen Brennan.
ISBN 0-415-35777-X
";
let found = read(front, PAGES);
assert_eq!(found.title.as_deref(), Some("On translation"));
assert_eq!(found.authors, vec!["Paul Ricoeur"]);
}
/// A title broken across two lines by the width of the page, with the second
/// half looking exactly like a person's name.
#[test]
fn a_wrapped_title_is_one_title() {
let front = "\
<!-- Page 4 -->
Esotericism in
Western Culture
Counter-Normativity and
Rejected Knowledge
Wouter J. Hanegraaff
<!-- Page 5 -->
BLOOMSBURY ACADEMIC
First published 2012
All rights reserved
";
let found = read(front, PAGES);
assert_eq!(
found.title.as_deref(),
Some("Esotericism in Western Culture")
);
assert_eq!(
found.subtitle.as_deref(),
Some("Counter-Normativity and Rejected Knowledge")
);
assert_eq!(found.authors, vec!["Wouter J. Hanegraaff"]);
}
/// Two capitalised words are a name or they are half a title, and English
/// morphology is the only thing on the page that says which.
#[test]
fn an_abstract_noun_is_not_a_person() {
let front = "\
<!-- Page 1 -->
Political Philosophy
From Plato to Mao
Martin Cohen
Pluto Press
LONDON
<!-- Page 2 -->
First published 2001 by Pluto Press
All rights reserved
";
let found = read(front, PAGES);
assert_eq!(found.title.as_deref(), Some("Political Philosophy"));
assert_eq!(found.subtitle.as_deref(), Some("From Plato to Mao"));
assert_eq!(found.authors, vec!["Martin Cohen"]);
assert_eq!(found.publisher.as_deref(), Some("Pluto Press"));
}
/// A scan that starts at the preface prints no title page, and the honest
/// record of that is an empty one.
#[test]
fn a_book_that_says_nothing_about_itself_is_reported_as_saying_nothing() {
let front = "\
<!-- Page 2 -->
Preface
Some mental states have qualitative phenomenal characters, and since the
dawn of Behaviorism some philosophers have doubted the ability of any
physicalist account of the mental to accommodate this fact.
";
let found = read(front, PAGES);
assert!(found.is_empty());
assert!(found.title.is_none());
for field in ["title", "authors", "isbns", "publisher", "year", "subjects"] {
assert!(
found.unsettled.iter().any(|f| f == field),
"{field} should be unsettled"
);
}
}
/// An ISBN whose check digit disagrees is a mistyped one, and reporting it
/// would hand the next stage a key that opens nothing.
#[test]
fn an_isbn_that_fails_its_check_digit_is_not_reported() {
let front = "\
<!-- Page 3 -->
All rights reserved
First published 1999
ISBN 978-0-19-280353-9
";
let found = read(front, PAGES);
assert!(found.isbns.is_empty());
}
/// What a model answers is filled in only where the pages said nothing, and
/// is marked as having come from a model.
#[test]
fn a_reply_fills_only_what_the_pages_left() {
let front = "\
<!-- Page 4 -->
Esotericism in
Western Culture
Wouter J. Hanegraaff
<!-- Page 5 -->
All rights reserved
First published 2012
";
let mut found = read(front, PAGES);
assert!(found.subjects.is_empty());
let filled = colophon::prompt::merge(
&mut found,
"{\"title\": \"Something Else\", \"subjects\": [\"Occultism\", \"Europe\"]}",
)
.expect("a readable reply");
assert_eq!(filled, 1);
assert_eq!(
found.title.as_deref(),
Some("Esotericism in Western Culture")
);
assert_eq!(found.subjects, vec!["Occultism", "Europe"]);
assert_eq!(found.read_from.get("subjects"), Some(&Source::Model));
assert!(!found.unsettled.iter().any(|f| f == "subjects"));
}
/// The question a model is asked never includes the pages that would
/// mislead it.
#[test]
fn the_question_leaves_out_the_advertising() {
let front = "\
<!-- Page 3 -->
Very Short Introductions available now:
ANARCHISM Colin Ward
THE ELEMENTS Philip Ball
CONSCIOUSNESS Susan Blackmore
ARISTOTLE Jonathan Barnes
COSMOLOGY Peter Coles
<!-- Page 6 -->
Some pages of a preface that names no title at all, running on for long
enough that the classifier takes it for text rather than a title page.
";
let found = read(front, PAGES);
let question = colophon::prompt::compose(&found, front, PAGES).expect("a question");
assert!(!question.contains("THE ELEMENTS"));
assert!(!question.contains("Philip Ball"));
assert!(question.contains("wanted: title"));
}
crates/corpus-core/src/doc.rs
+15
View File
@@ -77,6 +77,12 @@ pub enum Kind {
Azw3,
/// Amazon MOBI.
Mobi,
/// Microsoft Reader.
Lit,
/// FictionBook, the XML ebook format.
Fb2,
/// Rich Text Format.
Rtf,
/// Plain text.
Txt,
/// HTML or XHTML.
@@ -85,6 +91,10 @@ pub enum Kind {
Ole2,
/// A zip archive whose only real content is a single PDF.
ZipWrappedPdf,
/// A zip archive whose only real content is a single FictionBook.
ZipWrappedFb2,
/// A comic book archive: a zip of page images.
Cbz,
/// Recognised as none of the above.
Unknown,
}
@@ -118,6 +128,11 @@ mod tests {
(Kind::Ole2, "\"ole2\""),
(Kind::Azw3, "\"azw3\""),
(Kind::ZipWrappedPdf, "\"zip-wrapped-pdf\""),
(Kind::ZipWrappedFb2, "\"zip-wrapped-fb2\""),
(Kind::Fb2, "\"fb2\""),
(Kind::Lit, "\"lit\""),
(Kind::Rtf, "\"rtf\""),
(Kind::Cbz, "\"cbz\""),
(Kind::Unknown, "\"unknown\""),
];
for (kind, wire) in cases {
crates/corpus-core/src/exit.rs
+2
View File
@@ -30,6 +30,8 @@ pub const UNAVAILABLE: u8 = 69;
pub const SOFTWARE: u8 = 70;
/// Reading or writing failed.
pub const IO: u8 = 74;
/// A service refused for now, and asking again later will work.
pub const TEMPFAIL: u8 = 75;
/// The answer a predicate tool gives.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
crates/idnorm/src/isbn.rs
+99
View File
@@ -0,0 +1,99 @@
//! An ISBN's two forms, ten digits and thirteen.
//!
//! A book printed before 2007 carries a ten-digit ISBN, one printed since
//! carries thirteen, and every ten-digit one has a thirteen-digit twin: `978`
//! in front and the check digit worked out again. A catalogue may file a
//! book under either, so asking about one form and not the other misses
//! books that are there.
//!
//! Only a well-formed ISBN is converted. Working out a fresh check digit for
//! one whose own check digit is wrong would not correct the typing error, it
//! would turn the number into a different book's.
/// Just the digits of an ISBN, with a check character of ten as `X`.
pub fn digits(text: &str) -> String {
bibclean::isbn::digits(text).to_ascii_uppercase()
}
/// The thirteen-digit form of an ISBN, from either form.
///
/// `None` when the text is not an ISBN whose check digit agrees.
pub fn to_13(text: &str) -> Option<String> {
let digits = digits(text);
if !bibclean::isbn::valid(&digits) {
return None;
}
match digits.len() {
13 => Some(digits),
10 => {
let body = format!("978{}", &digits[..9]);
let sum: u32 = body
.bytes()
.enumerate()
.map(|(at, b)| u32::from(b - b'0') * if at % 2 == 0 { 1 } else { 3 })
.sum();
Some(format!("{body}{}", (10 - sum % 10) % 10))
}
_ => None,
}
}
/// The ten-digit form of an ISBN, from either form.
///
/// `None` when the text is not an ISBN whose check digit agrees, or when it
/// is a thirteen-digit one starting `979`, which was never issued in ten.
pub fn to_10(text: &str) -> Option<String> {
let digits = digits(text);
if !bibclean::isbn::valid(&digits) {
return None;
}
match digits.len() {
10 => Some(digits),
13 if digits.starts_with("978") => {
let body = &digits[3..12];
let sum: u32 = body
.bytes()
.enumerate()
.map(|(at, b)| u32::from(b - b'0') * (10 - at as u32))
.sum();
let check = match (11 - sum % 11) % 11 {
10 => 'X',
n => char::from(b'0' + n as u8),
};
Some(format!("{body}{check}"))
}
_ => None,
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn each_form_becomes_the_other() {
assert_eq!(to_13("0-306-40615-2").as_deref(), Some("9780306406157"));
assert_eq!(to_10("978-0-306-40615-7").as_deref(), Some("0306406152"));
assert_eq!(to_13("9780306406157").as_deref(), Some("9780306406157"));
assert_eq!(to_10("0306406152").as_deref(), Some("0306406152"));
}
#[test]
fn a_check_digit_of_ten_is_written_x() {
assert_eq!(to_10("9780804429573").as_deref(), Some("080442957X"));
assert_eq!(to_13("080442957x").as_deref(), Some("9780804429573"));
}
#[test]
fn a_979_isbn_has_no_ten_digit_form() {
assert_eq!(to_10("979-10-90636-07-1"), None);
assert_eq!(to_13("979-10-90636-07-1").as_deref(), Some("9791090636071"));
}
#[test]
fn a_mistyped_isbn_is_not_converted_into_a_different_book() {
assert_eq!(to_13("0-306-40615-3"), None);
assert_eq!(to_10("978-0-306-40615-8"), None);
assert_eq!(to_13("12345"), None);
}
}
crates/idnorm/src/lib.rs
+1
View File
@@ -13,6 +13,7 @@
pub mod arxiv;
pub mod doi;
pub mod isbn;
pub mod issn;
use schemars::JsonSchema;
docs/contract.md
+2 -1
View File
@@ -47,6 +47,7 @@ that was concatenated or written on Windows still reads.
| `69` | A program it needs is not installed |
| `70` | Its own fault |
| `74` | Reading or writing failed |
| `75` | A service refused for now; try again later |
Failures start where `sysexits.h` puts them, so "no" and "broke" are never the
same number. That is what makes this work:
@@ -64,7 +65,7 @@ No network, no database, no configuration file, no state kept between runs.
Five exceptions, each declared in `--help`:
- [`emb`](../crates/emb/README.md) makes an HTTP call, which is the whole of what it does.
- [`bibfetch`](../crates/bibfetch/README.md) asks a register about an identifier, likewise.
- [`bibfetch`](../crates/bibfetch/README.md) asks a register about an identifier or a title, likewise.
- [`pdftriage`](../crates/pdftriage/README.md) runs `pdftotext` and `pdfinfo`.
- [`pdfgutter`](../crates/pdfgutter/README.md) runs `pdftoppm` to look at a page.
- [`avplan`](../crates/avplan/README.md) runs `ffprobe` to read a file's streams, and
docs/decisions.md
+18
View File
@@ -20,6 +20,7 @@ to any of them is a change to a decision rather than to an accident.
- [pdfgutter — a page tree that does not add up is refused](#pdfgutter--a-page-tree-that-does-not-add-up-is-refused)
- [pdfgutter — a sheet on its side is cut the way it is read](#pdfgutter--a-sheet-on-its-side-is-cut-the-way-it-is-read)
- [pdfgutter — an unreadable file is an answer, not a false](#pdfgutter--an-unreadable-file-is-an-answer-not-a-false)
- [bookmeta — a cover is what its bytes say it is](#bookmeta--a-cover-is-what-its-bytes-say-it-is)
- [Every pattern — two regex engines, two Unicode tables](#every-pattern--two-regex-engines-two-unicode-tables)
## cuenorm — settings after the end time are the format, not a malformation
@@ -243,6 +244,23 @@ answers, and the [contract](contract.md) gives them three different numbers. A
caller wanting them collapsed treats 1 and 2 alike; a caller wanting to know
which files it failed on can now find out.
## bookmeta — a cover is what its bytes say it is
An EPUB's manifest gives every image a media type, and the cover's is not
used: the type comes from the image's first bytes, and a disagreement is a
note. Elsewhere `bookmeta` reports what the file says rather than what is
true, and this is the one place it does not.
The difference is that a title nobody believes is still a title, while a
media type that is wrong is an image that does not open. A caller handed
`image/png` for a JPEG writes `cover.png`, serves it as PNG, or hands it to a
decoder that refuses it. An image can be replaced by a converter or by hand
without its manifest entry changing, and the note keeps what the file said,
so nothing is lost by correcting it.
The same reasoning goes the other way: bytes that are not an image are not a
cover, whatever the manifest calls them.
## Every pattern — two regex engines, two Unicode tables
Patterns here spell their character classes explicitly rather than leaning on
docs/recipes.md
+92
View File
@@ -6,6 +6,7 @@ this suite, because a real pipeline has a database at the end of it.
- [A shelf of files, sorted before anything else touches it](#a-shelf-of-files-sorted-before-anything-else-touches-it)
- [A catalogue out of what the files already say](#a-catalogue-out-of-what-the-files-already-say)
- [Shelving a library by what its books printed](#shelving-a-library-by-what-its-books-printed)
- [Asking a model, without this suite calling one](#asking-a-model-without-this-suite-calling-one)
- [A scanned book into searchable passages](#a-scanned-book-into-searchable-passages)
- [A collected edition into one record per work](#a-collected-edition-into-one-record-per-work)
@@ -98,6 +99,97 @@ with:
langid converted/*.md | jq -r 'select(.language != "en") | [.path, .name] | @tsv'
```
## Shelving a library by what its books printed
A catalogue's subjects come from wherever the file came from, which is to say
from nowhere in particular. A book's own cataloguing block is the one place its
subjects are stated rather than guessed — and stated in Library of Congress
form, which is precise and useless for browsing:
```
$ colophon converted/hobbes.md | jq -r '.subjects[]'
Hobbes, Thomas, 1588–1679. Leviathan
Political science – Early works to 1800
```
Nobody has a shelf called that. Give `colophon` the list you actually file by
and it files them for you:
```
$ cat shelf.txt
Political Science = political science, politics
Byzantine History = byzantium, byzantine, constantinople
Theology = theology, christianity
Ethics
$ colophon --shelf shelf.txt converted/*.md | jq -c '{title, shelved, unshelved}'
{"title":"Leviathan","shelved":["Political Science"],"unshelved":["Hobbes, Thomas, 1588–1679. Leviathan"]}
{"title":"Christianity and Classical Culture","shelved":["Theology"],"unshelved":[]}
```
`unshelved` is the point of the exercise. A subject quietly dropped reads
exactly like a book that printed none, and the difference decides whether you
go looking for a heading your shelf is missing.
**The title page is not evidence on its own.** A scan that begins mid-frontmatter
gives up a running head, a copyright line, or the author's name where the title
should be. What makes a printed record trustworthy is a second one agreeing,
and the ISBN in the same block is how you get one:
```sh
colophon --shelf shelf.txt converted/*.md > printed.ndjson
jq -r 'select(.isbns != null) | .isbns[0]' printed.ndjson \
| idnorm --quiet \
| bibfetch --polite you@example.org > register.ndjson
```
Now two sources that cannot have copied each other have spoken, and you can
say which books are settled and which need a person:
```sh
jq -s -r '
(.[1] | map({key: .id, value: .}) | from_entries) as $reg
| .[0][]
| . as $printed
| ($reg[$printed.isbns[0] // ""] // {}) as $found
| [ $printed.title,
$found.title,
(if $printed.title and $found.title then "corroborated" else "one source" end)
] | @tsv
' printed.ndjson register.ndjson
```
```
The First World War The First World War: A Very Short Introduction corroborated
Document5 The Problems of Philosophy corroborated
Born Fighting — one source
```
The second row is the case worth the whole recipe: a file named `Document5`
whose author reads `-ducati996`, which the book itself and OpenLibrary agree
is Russell's *The Problems of Philosophy*. The third is the honest failure —
one source said something, nothing confirmed it, and it goes to a human.
In process, where a pipeline actually lives:
```python
import catena
shelf = open("shelf.txt").read()
printed = catena.read_colophon(markdown, shelf=shelf)
found = None
if printed["isbns"]:
found = catena.fetch_identifier(printed["isbns"][0], polite="you@example.org")
settled = bool(printed.get("title") and found and found.get("title"))
```
`catena.shelve(shelf, subjects)` files headings a catalogue already holds, by
the same rules, so a record's existing subjects and its book's printed ones are
judged alike.
## Asking a model, without this suite calling one
A tool here may compose a prompt. It does not send one. That keeps every
docs/records.md
+1 -1
View File
@@ -19,7 +19,7 @@ Part of [catena](../README.md).
| Field | Meaning |
|---|---|
| `path` | Where the file was read from, as given |
| `kind` | `pdf`, `epub`, `djvu`, `azw3`, `mobi`, `txt`, `html`, `ole2`, `zip-wrapped-pdf`, `unknown` |
| `kind` | `pdf`, `epub`, `djvu`, `azw3`, `mobi`, `lit`, `fb2`, `rtf`, `txt`, `html`, `ole2`, `cbz`, `zip-wrapped-pdf`, `zip-wrapped-fb2`, `unknown` |
| `declared_ext` | The extension the name claimed, lowercased, without the dot |
| `bytes` | Size on disk |
| `pages` | Page count, for paginated formats |
examples/README.md
+15
View File
@@ -15,6 +15,7 @@ NASA broadcast, which is a work of the US government.
| [`shell/03-search-and-rerank.sh`](shell/03-search-and-rerank.sh) | Passages into pgvector, then a search that asks twice and reconciles |
| [`rust/`](rust/src/main.rs) | The same book walked through in one process, with no pipe anywhere |
| [`python/book_to_citations.py`](python/book_to_citations.py) | A book to rows a database can take |
| [`python/shelve_a_library.py`](python/shelve_a_library.py) | What a book says about itself, shelved and checked against a register |
| [`python/search_and_rerank.py`](python/search_and_rerank.py) | The pgvector search, in the language a pipeline is usually written in |
| [`typescript/src/episode-to-chapters.ts`](typescript/src/episode-to-chapters.ts) | Captions into chapter marks and deep links for a player |
| [`typescript/src/search-and-rerank.ts`](typescript/src/search-and-rerank.ts) | Fusing two rankings app-side, and what re-ranking means |
@@ -83,6 +84,20 @@ python examples/python/book_to_citations.py
pip install catena-corpus psycopg
DATABASE_URL=postgresql://user@host/db \
python examples/python/search_and_rerank.py "why did the empire decline"
`shelve_a_library.py` is the metadata half: it reads the cataloguing block a
book prints in its own front matter, files those subjects onto a closed list
you supply, and asks the register that assigned the ISBN whether it agrees.
```sh
python examples/python/shelve_a_library.py converted/*.md
```
Neither source is worth acting on alone — a scan that starts mid-frontmatter
gives up a running head where the title should be, and an ISBN a catalogue has
been carrying wrongly returns a confidently wrong book. The example prints
`corroborated` only when both said the same thing, and the disagreements are
the interesting output.
```
## TypeScript
examples/python/shelve_a_library.py
+110
View File
@@ -0,0 +1,110 @@
"""What a book says about itself, checked against what a register says back.
A catalogue's subjects and titles come from wherever its files came from. Two
things can correct that without a model: the cataloguing block the book prints
in its own front matter, and the register that assigned its ISBN. Neither is
trustworthy alone — a scan that starts mid-frontmatter yields a running head
where the title should be, and an ISBN a catalogue has been carrying wrongly
returns a confidently wrong book. Together they are worth acting on.
python examples/python/shelve_a_library.py converted/*.md
With no arguments it reads a book from Project Gutenberg, which prints no
cataloguing block at all — the honest common case, and worth seeing.
"""
import pathlib
import re
import sys
import urllib.request
import catena
# The closed list this library is browsed by. `Heading = term, term` gives the
# words that file a book under it; a heading on its own matches its own name.
SHELF = """
Philosophy = philosophy, philosophical
Ancient Philosophy = ancient philosophy, plato, aristotle
Political Science = political science, politics, government
Theology = theology, christianity, religion
Church History = church history, ecclesiastical history
Byzantine History = byzantium, byzantine, constantinople
History = history, historiography
Literature = literature, fiction, poetry
Psychology = psychology
Ethics
Logic
"""
GUTENBERG = "https://www.gutenberg.org/cache/epub/5827/pg5827.txt"
def sample(into: pathlib.Path) -> str:
"""Locke's *Essay*, which Gutenberg strips the front matter from."""
book = into / "locke.txt"
if not book.exists():
into.mkdir(parents=True, exist_ok=True)
print(f"fetching {GUTENBERG}", file=sys.stderr)
with urllib.request.urlopen(GUTENBERG, timeout=120) as reply: # noqa: S310
book.write_bytes(reply.read())
return book.read_text(encoding="utf-8", errors="replace")
def same(a: str, b: str) -> bool:
"""Whether two printings of one title are the same title.
Compared on letters and digits alone. A register and a title page differ
over case, over a curly apostrophe against a straight one, and over
whether the subtitle is part of the title — none of which is a
disagreement, and all of which a plain string comparison calls one.
"""
fold = lambda s: re.sub(r"[^a-z0-9]+", "", (s or "").lower())
a, b = fold(a), fold(b)
return bool(a) and bool(b) and (a in b or b in a)
def look(text: str, name: str) -> None:
printed = catena.read_colophon(text, shelf=SHELF)
title = printed.get("title")
isbns = printed.get("isbns") or []
print(f"\n{name}")
print(f" printed title {title or '— (no title page in these pages)'}")
if printed.get("authors"):
print(f" printed authors {', '.join(printed['authors'])}")
if printed.get("shelved"):
print(f" shelved under {', '.join(printed['shelved'])}")
for raw in printed.get("unshelved", []):
# Not a failure: a heading the shelf has no place for is how you find
# out the shelf is missing one.
print(f" no shelf for {raw}")
if not printed.get("subjects"):
print(" subjects — (no cataloguing block printed)")
if not isbns:
print(" register — (nothing to look up; one source only)")
return
# The only call here that opens a socket, and it blocks.
found = catena.fetch_identifier(isbns[0], polite="[email protected]")
if not found or not found.get("found"):
print(f" register {isbns[0]} is unknown to every register asked")
return
print(f" register says {found['title']} / {', '.join(found.get('authors') or [])}")
agrees = bool(title) and same(title, found["title"])
print(f" verdict {'corroborated' if agrees else 'two sources disagree — ask a person'}")
def main() -> int:
paths = [pathlib.Path(p) for p in sys.argv[1:]]
if not paths:
look(sample(pathlib.Path("/tmp/catena-example")), "Locke, from Gutenberg")
return 0
for path in paths:
look(path.read_text(encoding="utf-8", errors="replace"), path.name)
return 0
if __name__ == "__main__":
raise SystemExit(main())