A lookup by ISBN took the year, publisher, language and page count from
Open Library's index, which answers for the work and gathers them from
every edition: Penguin's English War and Peace came back Spanish,
Norton's, and 1845. The index now gives only the title, the authors and
the subjects, and the edition record the rest (#6).
A cover must run to its end marker, FF D9 for a JPEG and IEND for a PNG,
so a body that only starts like an image is refused (#4). A relative
redirect whose query holds another address is resolved as relative (#5).
Closes#4, #5, #6.
Open Library's index answers about the work. Its publishers, languages,
first year and median page count are every edition's at once, and the
first of each is usually another edition's: ISBN 9780140447934, the
Penguin Classics War and Peace, came back Spanish, from W.W. Norton, in
1845.
The index is now asked only for what is true of the work (title,
subtitle, authors, subjects). A record from it is never complete, so
lookup always goes on to the edition record for container, language,
year and pages, then Google Books for whatever the edition record does
not say. A field neither gives is left out rather than filled from the
work.
Closes#6.
bookmeta gives an EPUB's cover by the routes its package can name one, and
marks a cover the file left open (a MOBI's first image, an EPUB image
found by its name) as guessed. booksniff knows RTF, Microsoft Reader,
FictionBook and comic book zips. bibfetch searches the registers by title
and author for a book with no identifier; covers by ISBN keep under Open
Library's rate limit, and a refusal for it is Error::Limited, not an
outage. The Python package gains fetch_cover, search_title and
catena.Limited.
Breaking: corpus_core::Kind has five new kinds; bibfetch::Error has
Limited; bookmeta::Cover has guessed; bibfetch::Covers has pace.
Open Library's title search, given an author as well, finds nothing for
The Ladder of Divine Ascent and four editions of Ladder of Divine Ascent.
When the title as given finds nothing, the title without a leading The, A
or An is asked, then the title and author as one free-text query, each
only when the one before found nothing. Google Books, whose anonymous
quota runs out, is asked only after all three.
A candidate whose edition gave an ISBN but no date or publisher took the
work's first_publish_year and first publisher, so one record could carry
one edition's ISBN and cover beside another's year and imprint. The id,
year, publisher, language and cover now come from the named edition only,
and are left out when it lacks them; the work gives only the title,
authors and subjects. A work with no edition named is named by its key,
not by an ISBN out of its editions' mixed list. The index is no longer
asked for the work-level fields, and gives no edition a length, so Open
Library candidates carry no pages.
When Open Library had nothing and Google Books failed, search answered
with no candidates and the CLI said no register had the title. The
failure is now the answer: Error::Limited for Google's 403, or 429 on
every attempt, which exits 75, and Error::Unreachable otherwise, exit 69.
The last route took any image whose id or address contained "cover", so
images/discover-ch1.png listed before images/cover.jpg won, and
recovery-map.png alone became the cover. A name now says cover only with
it as a word of its own: words break at anything not a letter or digit
and where a lower case letter meets a capital, so CoverImage and
Book_COVER still do. An image named cover outright comes first, then
manifest order. It is still a guess.
The guide's cover page took its first image, which can be a logo. Where
the page has several, the one whose file name, alt, id or class says
cover is taken; where none does, the first is, with a note saying how
many there were.
Each member was read no further than MAX_COVER, but nothing bounded how
many: two hundred cover-named images of 16 MiB took seven seconds to
inflate and left two hundred notes. One search now reads at most eight
members and stops with one note.
A percent-escape takes two hex digits and nothing else. from_str_radix
also takes a sign, so a%+fb decoded to a, 0x0f, b.
A zip whose members are all images was called cbz from their names alone,
so a folder of holiday photographs came back as a comic. It is now cbz
only when it carries a ComicInfo.xml or is itself named .cbz or .cbr;
otherwise it is unknown, with a note that it is a zip of N images and
would be a comic book if its name said .cbz. Only the archive's directory
is read, so members are still judged by name, and the README says so,
for a wrapped FictionBook too.
{\rtf has to be the whole control word: the byte after it must be the
version digit, a space or the next control word, so {\rtfoo is not RTF.
A FictionBook saved as UTF-16 was unknown. A byte-order mark in either
order now has the front decoded, at most 2048 code units, before the same
walk to the root element.
bookmeta: Cover::guessed says the file left the choice open — a Kindle
book's first image record, or an EPUB image picked because its name says
cover — so a caller with a register to ask can prefer that. A cover the
file names is never a guess. Cover::extension matches bibfetch's.
bibfetch: Open Library answers 403 to more than a hundred covers by ISBN
in five minutes from one address. That is now Error::Limited, not
Unreachable: the other form of the ISBN is not asked, the cover id still
is, since those are not limited, and the CLI exits 75. Covers asked by
ISBN are kept three seconds apart across the process (Covers::pace), which
stays under the limit however long a run goes. A deadline too long to add
to the clock no longer panics; it is no deadline.
bindings: fetch_cover and search_title for Python, and catena.Limited,
which fetch_identifier raises too when a register refuses for being asked
too often.
corpus-core: exit 75, a service refused for now.
A book that arrives as a file often has no identifier to ask about, only
a title and perhaps a name. `bibfetch search --title T [--author A]
[--limit N]` asks Open Library's index which books it has under those,
and Google Books only when Open Library has nothing, and prints one
record per candidate in the order the register gave them. From Rust,
`bibfetch::search(&agent, title, author, limit, attempts)`.
Each candidate is one edition. The index answers with works and, asked
for `editions.*`, names the edition of each that best matches; the ISBN,
publisher, language, year and cover come from that edition, the title,
authors, subjects and length from the work. The work's own ISBN list
mixes editions: the first War and Peace work lists four, while the
edition matched is a 1917 printing with none.
`id` is the first ISBN whose check digit agrees, in thirteen digits,
thirteen-digit entries taken first. A candidate with none is named by
the register's key: `olid` (OL…M for the edition, OL…W for the work when
no edition was named) or `googlebooks` (a volume id). Google's `OTHER`
shelfmarks are never read as ISBNs.
`cover_id` keeps the meaning it has in a lookup record: the edition's
own first cover, taken from `editions.cover_i` and only when the record
names that edition. The work's `cover_i` and `cover_edition_key` are not
asked for. A work with no edition named, and every Google Books
candidate, has none.
Open Library having nothing stands when Google Books is down; both down
is exit 69. Exit 0 a candidate, 1 none, 65 an empty title.
The retrying GET that lookups use is now `ask::get`, shared by both, and
no longer sleeps after its last attempt.
bookmeta reads MOBI 6, AZW and AZW3 from the MOBI header and its EXTH
records: title, authors, publisher, description, ISBN, subjects, date and
language, and the cover image the book carries (#1). EXTH 113 is filed by
its shape, an ASIN under asin and a calibre UUID under uuid. The EPUB
reader now keeps dc:description as well.
bibfetch fetches a book's cover from Open Library by ISBN, falling back to
the ISBN-10 form and then to the edition's own cover id, which a Record now
carries; `bibfetch cover` does the same from a terminal (#2). idnorm gains
ISBN-10 and ISBN-13 conversion.
Breaking:
- bookmeta::Metadata has two new public fields, description and cover, so
a struct literal without `..` no longer compiles.
- bibfetch::Record has a new public field, cover_id.
- bibfetch::Error has a new variant, Invalid, so an exhaustive match
without it no longer compiles.
- A bibfetch input file named `cover` is now read as the subcommand; pass
it as ./cover.
The cover is the first image reached by EPUB 3's cover-image property,
EPUB 2's <meta name="cover"> (by id, or by address where a converter wrote
one), the guide's cover page and the first <img> or SVG <image> on it, and
last any manifest image whose id or address says cover. A route that names
something unusable leaves a note and the next is asked.
Addresses are resolved against the document they are written in, with the
fragment off and percent-escapes undone; one that climbs out of the archive
or carries a scheme is not followed. Every member is read through a bound,
checked against the size the archive declares and enforced on the read
itself, so an understated entry stops at the cap too.
The media type is read off the bytes, now WebP as well, and a manifest that
disagrees is a note; bytes that are not an image are no cover. MAX_COVER
(16 MiB) is the cap for both formats, so a Kindle cover record larger than
that is left out as well.
Four formats a library of ebooks holds and this had no name for, so each
came back as text, as HTML or as unknown.
RTF is its opening group, after any byte-order mark and blank space.
Microsoft Reader is its signature at the front. A FictionBook is XML
whose root element is FictionBook, found by stepping over the
declaration, comments and doctype in the first four kilobytes. That walk
is over bytes rather than text, because a great many FictionBooks are
windows-1251 and would otherwise be called binary; other XML is still
HTML, as before.
A zip whose only real member is one FictionBook is zip-wrapped-fb2, the
same way a zipped PDF is zip-wrapped-pdf: handed to a converter as
FictionBook it is still a zip. A zip whose every real member is a JPEG,
PNG, GIF or WebP is cbz, once directories, ComicInfo.xml and desktop
clutter are passed over. One PDF or one readme among the pages and it is
not a comic. Only the names in the archive's directory are read, as for
every other zip.
The kind vocabulary grows by five: lit, fb2, rtf, cbz and
zip-wrapped-fb2.
When a MOBI or AZW3 file names no cover (no EXTH 201 or 202), the cover is
the record at the first image index if it is a JPEG, PNG or GIF, and
nothing otherwise, as calibre does (metadata/mobi.py:524-532). The walk to
the next image is gone: a later record may be an inline illustration, and
kindlegen puts a font or RESC record first.
EXTH 121 is no longer read, which also stops a stale KF8 boundary below
first_image from hiding the cover.
Review id 3 on catena#2.
P0: Record.cover_id came from the search index's cover_i, the cover
chosen for the whole work, and the index is asked first, so cover()
fell back to another edition's image (Penguin War and Peace,
9780140447934, got Standard Ebooks' Maude cover 12621906). cover_i is
no longer asked for or read; cover_id is the first positive id in the
edition record's covers, or None. An ISBN request that fails, as
opposed to a 404, now ends the fetch with that failure instead of
going on to the cover id.
Redirects go only over HTTPS on the standard port to
covers.openlibrary.org, archive.org or a host under archive.org, and
never to an address by number; the covers store the fetch began at is
still allowed.
The 1 KB size floor is gone. With default=false a missing cover is a
404, and the stand-in without it is a 43-byte GIF the media type check
refuses, so the floor could only drop real Small covers.
Covers.deadline (DEADLINE, two minutes) bounds a whole fetch: every
address, attempt, redirect, body and wait. It starts once the fetch
holds the process lock. The timeout test no longer times a span that
includes the lock wait.
EXTH 113 is an ASIN only when it is ten letters and digits; a UUID goes
under uuid, as the EPUB reader files it, and anything else under other.
With no EXTH 201 or 202, the first record from the first image index that
is a JPEG, PNG or GIF is the cover, stopping at the KF8 boundary (EXTH 121).
`bibfetch::cover(&agent, isbn, cover_id, size)` asks Open Library's covers
store by the ISBN's thirteen-digit form, then its ten-digit form, then by a
cover id a lookup record named, and returns the image or `Ok(None)`. The
answer is checked before it is trusted: JPEG or PNG by content type and by
magic bytes, between 1 KB and 10 MB, at most three redirects, none of them
off HTTPS. One request at a time per process; a 429 or 503 is waited out
for its Retry-After, up to a minute.
`idnorm::isbn` converts between the two forms, and only for an ISBN whose
check digit agrees. Lookup records carry `cover_id` from the edition record
and the search index. The CLI gains `bibfetch cover <isbn> -o <file>`.
A Kindle book's title, authors, publisher, description, ISBN, ASIN,
subjects, date and language come from the EXTH block, with the full name,
the Palm database name and the header's locale as fallbacks. The cover
image (EXTH 201, or the thumbnail) is returned as Metadata::cover with its
media type; it stays out of the JSON record. Every offset is bounds-checked:
a damaged header gives an empty record with a note, a broken EXTH block
keeps what came before the break, and an encrypted book is still read and
noted as drm.
Metadata gains description, which the EPUB reader now fills from
dc:description as well.
Fixtures are a story and cover written for the tests (CC0), converted with
calibre to MOBI 6 and AZW3, plus no-EXTH, DRM-flagged and truncated copies.
The commit that reordered the chain did not carry the version bump. The
bump was chained after check.sh with &&, the commit after a semicolon, so
a failing check skipped the bump and let the commit through. The tag then
pointed at a tree still calling itself 0.3.0, which the release workflow
asserts against and would have refused.
check.sh was failing because a test names which register is asked first
about an ISBN, which is exactly what changed. It now names the index, for
the reason the reorder happened.
The chain asked OpenLibrary's edition record first and its search index
second. The edition record is the slow one: measured against this
library it takes five to twenty-eight seconds, and it names its authors
by key, so a second request was needed anyway. The index, asked for its
fields by name, returns the title, the authors, the publisher, the
language, a page count and the subjects together, in about a quarter of
a second.
Reordering them turns three requests per book into one, and a pass over
five thousand books from fifty-three hours into well under one. The
edition record is still there for a subtitle or a publisher the index
did not give.
The chain asked three registers and returned whichever answered first
with a title and an author, discarding the rest. The registers are
complementary rather than redundant: OpenLibrary's edition record
carries the publisher, the pagination and the language and names its
authors by key, while its search index carries the authors and not the
publisher. One book, two half-answers, and the whole of it only if they
are put together.
They are merged now, first answer winning each field, so the order of
the chain is the order of trust. The record says which registers spoke.
It also carries what was being thrown away: subjects, language and a
page count, from whichever register has them. The index will not return
those unless asked for by name, which is how a language it held went
unread. Subjects are capped at twelve — the index lists every heading
every edition of a work has ever been given, which on a well-known book
is hundreds.
An edition record keeps its subtitle in a separate field, so a title
came back without it.
Read out of the review queue a real run produced, where each of these was
sitting as a proposed title waiting for a person to say no:
1. General Remarks a numbered heading
2 James W. McAllister a numbered heading
5222 What Is This a folio that ran into the line
827 North Fourth Street an address
Page xxxvi / Chapter I a marker
title a form label, printed literally
Sidebottom, Harry. Ancient a bibliography entry
To Mackenzie, a dedication
The numbers are the delicate part. A leading number that could be a year
is left alone, because 1984 Revisited is a book and 5222 What Is This is
a folio, and nothing else distinguishes them. A dedication has to end in
a comma, so To Kill a Mockingbird survives.
The copyright notices in the same pile are already fixed: those
proposals were written before 0.2.2 and are stale rather than wrong.
title_case counted small words from the start of the whole string, so
every subtitle that opened with one was set in lower case: 'Aesthetic in
Kant: a Critique', 'Born Fighting: the Scots-Irish'. A colon starts a
title of its own and the first word of one is capitalised however small
it is.
Found by book-pipeline, which had been proposing a title page's capitals
verbatim into a catalogue and needed this to stop.
'(c) Alfredo Ferrarin 2004' was read as a title. The notice pattern
wanted the year to follow the symbol directly, and a copyright line puts
the holder's name in between, which is the ordinary way to set one.
The Python surface gained enough this week that three different builds
were installed on two machines all calling themselves 0.1.0: moria had
read_colophon through merge_colophon, mirkwood had a partial from the
middle of the afternoon, and PyPI had neither. A version that does not
move is a version nobody can depend on, which is how book-pipeline ended
up running a wheel scp'd into place by hand.
New since 0.1.0: colophon --shelf and catena.shelve, which file a book's
printed Library of Congress headings onto a closed list; catena
.read_colophon; catena.fetch_identifier, the registers in process; and
catena.colophon_prompt / merge_colophon, the question about what the
pages did not settle and the reply folded back in.
read_colophon is several milliseconds of computation over a few hundred
kilobytes, and it held the GIL for all of them. A caller reading a
library through asyncio.to_thread therefore got no overlap whatsoever:
six books in flight, one book's worth of work at a time.
Measured on this library, that was most of why a concurrent pass still
paced at four and a half hours.
colophon has composed the question about what its pages did not settle
since it was written, and merged a reply back into the record it read.
Both were reachable only from a terminal, so a pipeline wanting the same
fallback would have had to write its own prompt — a second one, drifting
from the first, asking for a shape merge does not accept.
merge fills only the unsettled fields, so a model cannot overwrite what a
book printed about itself. That is the property that makes asking one
safe, and it belongs to colophon rather than to any caller.
Read against five thousand real books, the title page is often not the
first thing in the file, and what is there instead has shape. Six kinds
of it were being read as titles:
vi CONTENTS a folioed heading
Copyright (c) 2013 by Barry Davies, BEM a notice
Translated from the French by Willard R. Trask a credit
Vol. 11. HENRY JAMES, JR. a collected edition's list
[PART ONE: a part heading
Piper Verlag GmbH, Munich<span style=... a converter's leavings
Rejecting one line only moves the misread down one, because a page that
converted badly converted badly throughout, so the whole shape goes:
markdown links, stylesheet declarations, HTML entities, epub filenames,
and the sentences a copyright page sets.
The notice pattern wants the symbol or the year, never the word alone.
The first draft rejected 'Copyright and Its Discontents', which its own
test caught.
On sixty books this drops five claimed titles, all of them wrong, and
fixes six that were reading one homily's heading for every other
homily's title.
The suite could read a cataloguing block and could ask a register, and
nothing showed the two together — which is the only way either is worth
acting on. A title page alone gives up a running head when the scan
starts late; an ISBN alone returns a confidently wrong book when the
catalogue has been carrying the wrong one for years.
The example's title comparison folds to letters and digits rather than
comparing strings. A register and a title page differ over case, over a
curly apostrophe against a straight one, and over whether the subtitle
counts as part of the title. An example that called those disagreements
would teach the wrong lesson, and did until it was fixed.
It runs on this library and finds both of the cases worth finding: a file
called Document5 by -ducati996 that both sources agree is Russell's
Problems of Philosophy, and a Pelikan whose stored ISBN belongs to a book
about sculpture.
book-pipeline reaches for colophon and bibfetch on every book it repairs.
Shelling out would mean a process per book per tool across five thousand
books, and parsing back the JSON it is about to parse anyway.
bibfetch's fetch loop lived in main.rs, so binding it meant either
lifting it or writing a second one. A second implementation of 'which
register answers about this kind of identifier, and in what order' would
drift from the first, and the copy nobody runs from a terminal would
drift silently. It is now bibfetch::ask, and the command line calls it
too — verified by running the CLI before and after and diffing.
fetch_identifier releases the GIL around the request. It is the only
function here that opens a socket, and holding the interpreter for the
length of an HTTP call would stop every other thread in it.
bibfetch is optional and only the python feature turns it on: its HTTP
client does not build for wasm32, and an unconditional dependency would
have taken the JavaScript package down with it.
funkpower