funkpower/catena
1
0 Fork 0
mirror of https://git.pumpin.online/funkpower/catena.git synced 2026-11-13 23:55:40 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: funkpower/catena:bibfetch-search
funkpower/catena:main funkpower/catena:issues-3-6 funkpower/catena:work-edition-metadata funkpower/catena:fixes-040 funkpower/catena:fixes-040-search funkpower/catena:fixes-040-epub funkpower/catena:fixes-040-sniff funkpower/catena:bookmeta-epub-cover funkpower/catena:bibfetch-search funkpower/catena:release-0.4.0 funkpower/catena:booksniff-more-formats funkpower/catena:bibfetch-openlibrary-cover funkpower/catena:bookmeta-mobi-exth
funkpower/catena:v0.5.0 funkpower/catena:v0.4.0 funkpower/catena:v0.3.1 funkpower/catena:v0.3.0 funkpower/catena:v0.2.3 funkpower/catena:v0.2.2 funkpower/catena:v0.2.1 funkpower/catena:v0.2.0 funkpower/catena:v0.1.0
...
compare: funkpower/catena:main
funkpower/catena:main funkpower/catena:issues-3-6 funkpower/catena:work-edition-metadata funkpower/catena:fixes-040 funkpower/catena:fixes-040-search funkpower/catena:fixes-040-epub funkpower/catena:fixes-040-sniff funkpower/catena:bookmeta-epub-cover funkpower/catena:bibfetch-search funkpower/catena:release-0.4.0 funkpower/catena:booksniff-more-formats funkpower/catena:bibfetch-openlibrary-cover funkpower/catena:bookmeta-mobi-exth
funkpower/catena:v0.5.0 funkpower/catena:v0.4.0 funkpower/catena:v0.3.1 funkpower/catena:v0.3.0 funkpower/catena:v0.2.3 funkpower/catena:v0.2.2 funkpower/catena:v0.2.1 funkpower/catena:v0.2.0 funkpower/catena:v0.1.0
27 Commits
bibfetch-search ... main
Author SHA1 Message Date
funkpower 2d885b5afc Merge pull request 'bibfetch: a cover whole to its end marker, and relative redirects with an address in the query (#4, #5)' (#13) from issues-3-6 into main 2026-11-13 16:33:34 -05:00
funkpower 99f4f198af Merge remote-tracking branch 'origin/main' into issues-3-6 
# Conflicts:
#	crates/bibfetch/src/source.rs
 2026-11-13 16:28:23 -05:00
funkpower 038b0c70ba Merge pull request 'bibfetch: language, publisher and year of an ISBN from its edition, not the work (closes #6)' (#12) from work-edition-metadata into main 2026-11-13 16:28:10 -05:00
funkpower 6b5d9c20ee bibfetch: an edition's facts from its own record, and a cover whole to its end 
A lookup by ISBN took the year, publisher, language and page count from
Open Library's index, which answers for the work and gathers them from
every edition: Penguin's English War and Peace came back Spanish,
Norton's, and 1845. The index now gives only the title, the authors and
the subjects, and the edition record the rest (#6).

A cover must run to its end marker, FF D9 for a JPEG and IEND for a PNG,
so a body that only starts like an image is refused (#4). A relative
redirect whose query holds another address is resolved as relative (#5).

Closes #4, #5, #6.
 2026-11-13 16:23:48 -05:00
funkpower dd41a93c9b bibfetch: take an ISBN's language, publisher, year and length from the edition, not the work 
Open Library's index answers about the work. Its publishers, languages,
first year and median page count are every edition's at once, and the
first of each is usually another edition's: ISBN 9780140447934, the
Penguin Classics War and Peace, came back Spanish, from W.W. Norton, in
1845.

The index is now asked only for what is true of the work (title,
subtitle, authors, subjects). A record from it is never complete, so
lookup always goes on to the edition record for container, language,
year and pages, then Google Books for whatever the edition record does
not say. A field neither gives is left out rather than filled from the
work.

Closes #6.
 2026-11-13 16:19:54 -05:00
funkpower e9f9e75c1d Merge pull request '0.5.0: review fixes, Python bindings for covers and search, version bump' (#11) from fixes-040 into main 2026-11-13 16:08:45 -05:00
funkpower 65fd4adaad Merge remote-tracking branch 'origin/main' into fixes-040 2026-11-13 16:04:00 -05:00
funkpower 58573399e7 Merge pull request 'bibfetch: the books a register has under a title' (#9) from bibfetch-search into main 2026-11-13 15:59:31 -05:00
funkpower d06e13e2f3 Merge pull request 'bookmeta: read an EPUB's cover, by the four routes a package can name one' (#8) from bookmeta-epub-cover into main 2026-11-13 15:59:25 -05:00
funkpower 8ecc4766ee Merge pull request 'booksniff: RTF, Microsoft Reader, FictionBook and comic archives' (#7) from booksniff-more-formats into main 2026-11-13 15:59:20 -05:00
funkpower 3bd3ea11ae 0.5.0 — covers a caller can trust, more formats known, and books found by title 
bookmeta gives an EPUB's cover by the routes its package can name one, and
marks a cover the file left open (a MOBI's first image, an EPUB image
found by its name) as guessed. booksniff knows RTF, Microsoft Reader,
FictionBook and comic book zips. bibfetch searches the registers by title
and author for a book with no identifier; covers by ISBN keep under Open
Library's rate limit, and a refusal for it is Error::Limited, not an
outage. The Python package gains fetch_cover, search_title and
catena.Limited.

Breaking: corpus_core::Kind has five new kinds; bibfetch::Error has
Limited; bookmeta::Cover has guessed; bibfetch::Covers has pace.
 2026-11-13 15:58:49 -05:00
funkpower b7feefb704 Merge remote-tracking branch 'origin/main' into fixes-040 2026-11-13 15:58:00 -05:00
funkpower f43b9d4900 bibfetch: a title Open Library will not find as written is asked again 
Open Library's title search, given an author as well, finds nothing for
The Ladder of Divine Ascent and four editions of Ladder of Divine Ascent.
When the title as given finds nothing, the title without a leading The, A
or An is asked, then the title and author as one free-text query, each
only when the one before found nothing. Google Books, whose anonymous
quota runs out, is asked only after all three.
 2026-11-13 15:56:49 -05:00
funkpower 6844f10e0e Merge remote-tracking branch 'origin/fixes-040-sniff' into fixes-040 2026-11-13 15:51:01 -05:00
funkpower 4aae5b6ac0 Merge remote-tracking branch 'origin/fixes-040-search' into fixes-040 2026-11-13 15:51:01 -05:00
funkpower 3fd48d4999 bibfetch: a search candidate is one edition, and a refused fallback says so 
A candidate whose edition gave an ISBN but no date or publisher took the
work's first_publish_year and first publisher, so one record could carry
one edition's ISBN and cover beside another's year and imprint. The id,
year, publisher, language and cover now come from the named edition only,
and are left out when it lacks them; the work gives only the title,
authors and subjects. A work with no edition named is named by its key,
not by an ISBN out of its editions' mixed list. The index is no longer
asked for the work-level fields, and gives no edition a length, so Open
Library candidates carry no pages.

When Open Library had nothing and Google Books failed, search answered
with no candidates and the CLI said no register had the title. The
failure is now the answer: Error::Limited for Google's 403, or 429 on
every attempt, which exits 75, and Error::Unreachable otherwise, exit 69.
 2026-11-13 15:50:46 -05:00
funkpower 0e83c6ffc1 bookmeta: an EPUB cover by the word, not the letters 
The last route took any image whose id or address contained "cover", so
images/discover-ch1.png listed before images/cover.jpg won, and
recovery-map.png alone became the cover. A name now says cover only with
it as a word of its own: words break at anything not a letter or digit
and where a lower case letter meets a capital, so CoverImage and
Book_COVER still do. An image named cover outright comes first, then
manifest order. It is still a guess.

The guide's cover page took its first image, which can be a logo. Where
the page has several, the one whose file name, alt, id or class says
cover is taken; where none does, the first is, with a note saying how
many there were.

Each member was read no further than MAX_COVER, but nothing bounded how
many: two hundred cover-named images of 16 MiB took seven seconds to
inflate and left two hundred notes. One search now reads at most eight
members and stops with one note.

A percent-escape takes two hex digits and nothing else. from_str_radix
also takes a sign, so a%+fb decoded to a, 0x0f, b.
 2026-11-13 15:49:18 -05:00
funkpower 48478fe84e booksniff: a zip of pictures is not a comic until something says so 
A zip whose members are all images was called cbz from their names alone,
so a folder of holiday photographs came back as a comic. It is now cbz
only when it carries a ComicInfo.xml or is itself named .cbz or .cbr;
otherwise it is unknown, with a note that it is a zip of N images and
would be a comic book if its name said .cbz. Only the archive's directory
is read, so members are still judged by name, and the README says so,
for a wrapped FictionBook too.

{\rtf has to be the whole control word: the byte after it must be the
version digit, a space or the next control word, so {\rtfoo is not RTF.

A FictionBook saved as UTF-16 was unknown. A byte-order mark in either
order now has the front decoded, at most 2048 code units, before the same
walk to the root element.
 2026-11-13 15:49:17 -05:00
funkpower de23a7abad bookmeta, bibfetch, bindings: what a caller needs to trust a cover 
bookmeta: Cover::guessed says the file left the choice open — a Kindle
book's first image record, or an EPUB image picked because its name says
cover — so a caller with a register to ask can prefer that. A cover the
file names is never a guess. Cover::extension matches bibfetch's.

bibfetch: Open Library answers 403 to more than a hundred covers by ISBN
in five minutes from one address. That is now Error::Limited, not
Unreachable: the other form of the ISBN is not asked, the cover id still
is, since those are not limited, and the CLI exits 75. Covers asked by
ISBN are kept three seconds apart across the process (Covers::pace), which
stays under the limit however long a run goes. A deadline too long to add
to the clock no longer panics; it is no deadline.

bindings: fetch_cover and search_title for Python, and catena.Limited,
which fetch_identifier raises too when a register refuses for being asked
too often.

corpus-core: exit 75, a service refused for now.
 2026-11-13 15:44:50 -05:00
funkpower 5f2a471f58 Merge pull request 'Release 0.4.0' (#10) from release-0.4.0 into main 2026-11-13 15:37:47 -05:00
funkpower 7aa1783a3e Merge remote-tracking branch 'origin/bibfetch-search' into fixes-040 2026-11-13 15:32:45 -05:00
funkpower 4d21cc5927 Merge remote-tracking branch 'origin/bookmeta-epub-cover' into fixes-040 2026-11-13 15:32:45 -05:00
funkpower 6510973850 Merge remote-tracking branch 'origin/booksniff-more-formats' into fixes-040 2026-11-13 15:32:45 -05:00
funkpower 897b2210b6 Merge remote-tracking branch 'origin/main' into bookmeta-epub-cover 
# Conflicts:
#	crates/bookmeta/src/mobi.rs
 2026-11-13 15:29:53 -05:00
funkpower bd104834ed 0.4.0 — MOBI and AZW books read without calibre, and a cover from Open Library 
bookmeta reads MOBI 6, AZW and AZW3 from the MOBI header and its EXTH
records: title, authors, publisher, description, ISBN, subjects, date and
language, and the cover image the book carries (#1). EXTH 113 is filed by
its shape, an ASIN under asin and a calibre UUID under uuid. The EPUB
reader now keeps dc:description as well.

bibfetch fetches a book's cover from Open Library by ISBN, falling back to
the ISBN-10 form and then to the edition's own cover id, which a Record now
carries; `bibfetch cover` does the same from a terminal (#2). idnorm gains
ISBN-10 and ISBN-13 conversion.

Breaking:
- bookmeta::Metadata has two new public fields, description and cover, so
  a struct literal without `..` no longer compiles.
- bibfetch::Record has a new public field, cover_id.
- bibfetch::Error has a new variant, Invalid, so an exhaustive match
  without it no longer compiles.
- A bibfetch input file named `cover` is now read as the subcommand; pass
  it as ./cover.
 2026-11-13 15:27:46 -05:00
funkpower 57ce9137e6 bookmeta: read an EPUB's cover, by the four routes a package can name one 
The cover is the first image reached by EPUB 3's cover-image property,
EPUB 2's <meta name="cover"> (by id, or by address where a converter wrote
one), the guide's cover page and the first <img> or SVG <image> on it, and
last any manifest image whose id or address says cover. A route that names
something unusable leaves a note and the next is asked.

Addresses are resolved against the document they are written in, with the
fragment off and percent-escapes undone; one that climbs out of the archive
or carries a scheme is not followed. Every member is read through a bound,
checked against the size the archive declares and enforced on the read
itself, so an understated entry stops at the cap too.

The media type is read off the bytes, now WebP as well, and a manifest that
disagrees is a note; bytes that are not an image are no cover. MAX_COVER
(16 MiB) is the cap for both formats, so a Kindle cover record larger than
that is left out as well.
 2026-11-13 15:26:49 -05:00
funkpower 5c4f770d76 booksniff: RTF, Microsoft Reader, FictionBook and comic archives 
Four formats a library of ebooks holds and this had no name for, so each
came back as text, as HTML or as unknown.

RTF is its opening group, after any byte-order mark and blank space.
Microsoft Reader is its signature at the front. A FictionBook is XML
whose root element is FictionBook, found by stepping over the
declaration, comments and doctype in the first four kilobytes. That walk
is over bytes rather than text, because a great many FictionBooks are
windows-1251 and would otherwise be called binary; other XML is still
HTML, as before.

A zip whose only real member is one FictionBook is zip-wrapped-fb2, the
same way a zipped PDF is zip-wrapped-pdf: handed to a converter as
FictionBook it is still a zip. A zip whose every real member is a JPEG,
PNG, GIF or WebP is cbz, once directories, ComicInfo.xml and desktop
clutter are passed over. One PDF or one readme among the pages and it is
not a comic. Only the names in the archive's directory are read, as for
every other zip.

The kind vocabulary grows by five: lit, fb2, rtf, cbz and
zip-wrapped-fb2.
 2026-11-13 15:25:33 -05:00
26 changed files with 2759 additions and 268 deletions
Expand all files Collapse all files

No files matched your search

Cargo.lock
Generated
+26 -26
View File
@@ -101,7 +101,7 @@ checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
[[package]] [[package]]
name = "avplan" name = "avplan"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -126,7 +126,7 @@ checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5"
[[package]] [[package]]
name = "bibclean" name = "bibclean"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -143,7 +143,7 @@ dependencies = [
[[package]] [[package]]
name = "bibfetch" name = "bibfetch"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -196,7 +196,7 @@ dependencies = [
[[package]] [[package]]
name = "bookmeta" name = "bookmeta"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -210,7 +210,7 @@ dependencies = [
[[package]] [[package]]
name = "booksniff" name = "booksniff"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -250,7 +250,7 @@ checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04"
[[package]] [[package]]
name = "catena" name = "catena"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"bibclean", "bibclean",
"bibfetch", "bibfetch",
@@ -277,7 +277,7 @@ dependencies = [
[[package]] [[package]]
name = "catena-cli" name = "catena-cli"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"clap", "clap",
"clap_mangen", "clap_mangen",
@@ -312,7 +312,7 @@ checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
[[package]] [[package]]
name = "chunkmd" name = "chunkmd"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -334,7 +334,7 @@ dependencies = [
[[package]] [[package]]
name = "citefind" name = "citefind"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"chunkmd", "chunkmd",
@@ -399,7 +399,7 @@ dependencies = [
[[package]] [[package]]
name = "colophon" name = "colophon"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"bibclean", "bibclean",
"catena-cli", "catena-cli",
@@ -448,7 +448,7 @@ dependencies = [
[[package]] [[package]]
name = "corpus-core" name = "corpus-core"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"schemars", "schemars",
"serde", "serde",
@@ -492,7 +492,7 @@ dependencies = [
[[package]] [[package]]
name = "cuekit" name = "cuekit"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"corpus-core", "corpus-core",
"regex", "regex",
@@ -504,7 +504,7 @@ dependencies = [
[[package]] [[package]]
name = "cuenorm" name = "cuenorm"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -514,7 +514,7 @@ dependencies = [
[[package]] [[package]]
name = "cuepack" name = "cuepack"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -525,7 +525,7 @@ dependencies = [
[[package]] [[package]]
name = "cueseg" name = "cueseg"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -612,7 +612,7 @@ dependencies = [
[[package]] [[package]]
name = "emb" name = "emb"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -876,7 +876,7 @@ dependencies = [
[[package]] [[package]]
name = "idnorm" name = "idnorm"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"bibclean", "bibclean",
"catena-cli", "catena-cli",
@@ -942,7 +942,7 @@ dependencies = [
[[package]] [[package]]
name = "langid" name = "langid"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -1022,7 +1022,7 @@ dependencies = [
[[package]] [[package]]
name = "mdscrub" name = "mdscrub"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -1034,7 +1034,7 @@ dependencies = [
[[package]] [[package]]
name = "mdsplit" name = "mdsplit"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -1113,7 +1113,7 @@ checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
[[package]] [[package]]
name = "pdfgutter" name = "pdfgutter"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -1126,7 +1126,7 @@ dependencies = [
[[package]] [[package]]
name = "pdftriage" name = "pdftriage"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -1370,7 +1370,7 @@ checksum = "323c417e1d9665a65b263ec744ba09030cfb277e9daa0b018a4ab62e57bc8189"
[[package]] [[package]]
name = "rrf" name = "rrf"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -1517,7 +1517,7 @@ dependencies = [
[[package]] [[package]]
name = "seriesof" name = "seriesof"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -1634,7 +1634,7 @@ checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca"
[[package]] [[package]]
name = "textkit" name = "textkit"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"regex", "regex",
"tiktoken-rs", "tiktoken-rs",
@@ -1724,7 +1724,7 @@ checksum = "fd3ca314f692efd6c868f8408f53fe444634a845f96c028b97d35f6a1f79f0ee"
[[package]] [[package]]
name = "tocfind" name = "tocfind"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
Cargo.toml
+27 -27
View File
@@ -3,39 +3,39 @@ resolver = "3"
members = ["crates/*"] members = ["crates/*"]
[workspace.package] [workspace.package]
version = "0.3.1" version = "0.5.0"
edition = "2024" edition = "2024"
rust-version = "1.85" rust-version = "1.85"
license = "MPL-2.0" license = "MPL-2.0"
repository = "https://git.pumpin.online/funkpower/catena" repository = "https://git.pumpin.online/funkpower/catena"
[workspace.dependencies] [workspace.dependencies]
catena-cli = { path = "crates/catena-cli", version = "0.3.1" } catena-cli = { path = "crates/catena-cli", version = "0.5.0" }
corpus-core = { path = "crates/corpus-core", version = "0.3.1" } corpus-core = { path = "crates/corpus-core", version = "0.5.0" }
cuekit = { path = "crates/cuekit", version = "0.3.1" } cuekit = { path = "crates/cuekit", version = "0.5.0" }
textkit = { path = "crates/textkit", version = "0.3.1" } textkit = { path = "crates/textkit", version = "0.5.0" }
chunkmd = { path = "crates/chunkmd", version = "0.3.1" } chunkmd = { path = "crates/chunkmd", version = "0.5.0" }
mdscrub = { path = "crates/mdscrub", version = "0.3.1" } mdscrub = { path = "crates/mdscrub", version = "0.5.0" }
mdsplit = { path = "crates/mdsplit", version = "0.3.1" } mdsplit = { path = "crates/mdsplit", version = "0.5.0" }
pdftriage = { path = "crates/pdftriage", version = "0.3.1" } pdftriage = { path = "crates/pdftriage", version = "0.5.0" }
cuenorm = { path = "crates/cuenorm", version = "0.3.1" } cuenorm = { path = "crates/cuenorm", version = "0.5.0" }
cueseg = { path = "crates/cueseg", version = "0.3.1" } cueseg = { path = "crates/cueseg", version = "0.5.0" }
rrf = { path = "crates/rrf", version = "0.3.1" } rrf = { path = "crates/rrf", version = "0.5.0" }
cuepack = { path = "crates/cuepack", version = "0.3.1" } cuepack = { path = "crates/cuepack", version = "0.5.0" }
bibclean = { path = "crates/bibclean", version = "0.3.1" } bibclean = { path = "crates/bibclean", version = "0.5.0" }
avplan = { path = "crates/avplan", version = "0.3.1" } avplan = { path = "crates/avplan", version = "0.5.0" }
pdfgutter = { path = "crates/pdfgutter", version = "0.3.1" } pdfgutter = { path = "crates/pdfgutter", version = "0.5.0" }
catena = { path = "crates/bindings", version = "0.3.1" } catena = { path = "crates/bindings", version = "0.5.0" }
seriesof = { path = "crates/seriesof", version = "0.3.1" } seriesof = { path = "crates/seriesof", version = "0.5.0" }
idnorm = { path = "crates/idnorm", version = "0.3.1" } idnorm = { path = "crates/idnorm", version = "0.5.0" }
bibfetch = { path = "crates/bibfetch", version = "0.3.1" } bibfetch = { path = "crates/bibfetch", version = "0.5.0" }
bookmeta = { path = "crates/bookmeta", version = "0.3.1" } bookmeta = { path = "crates/bookmeta", version = "0.5.0" }
langid = { path = "crates/langid", version = "0.3.1" } langid = { path = "crates/langid", version = "0.5.0" }
tocfind = { path = "crates/tocfind", version = "0.3.1" } tocfind = { path = "crates/tocfind", version = "0.5.0" }
booksniff = { path = "crates/booksniff", version = "0.3.1" } booksniff = { path = "crates/booksniff", version = "0.5.0" }
citefind = { path = "crates/citefind", version = "0.3.1" } citefind = { path = "crates/citefind", version = "0.5.0" }
colophon = { path = "crates/colophon", version = "0.3.1" } colophon = { path = "crates/colophon", version = "0.5.0" }
emb = { path = "crates/emb", version = "0.3.1" } emb = { path = "crates/emb", version = "0.5.0" }
clap = { version = "4", features = ["derive"] } clap = { version = "4", features = ["derive"] }
clap_mangen = "0.3" clap_mangen = "0.3"
crates/bibfetch/README.md
+56 -19
View File
@@ -86,7 +86,7 @@ is a failure — nobody replied, and the identifier may be perfectly good.
|---|---|---| |---|---|---|
| `doi` | Crossref | It assigns most scholarly DOIs | | `doi` | Crossref | It assigns most scholarly DOIs |
| `arxiv` | the arXiv | Its own register of preprints | | `arxiv` | the arXiv | Its own register of preprints |
| `isbn` | OpenLibrary's edition record, then its index, then Google Books | Books, without a key | | `isbn` | OpenLibrary's index, then its edition record, then Google Books | Books, without a key |
One register is enough for a DOI or a preprint: the body that assigns the One register is enough for a DOI or a preprint: the body that assigns the
identifier is the only one that has it. A book is different. An ISBN is identifier is the only one that has it. A book is different. An ISBN is
@@ -97,6 +97,14 @@ names them. So the registers are asked in turn and the first complete answer
wins. An incomplete one is kept in case nothing better comes, and a register wins. An incomplete one is kept in case nothing better comes, and a register
being down is a failure only when every register is. being down is a failure only when every register is.
OpenLibrary's index answers about the work, not the edition: its publishers,
languages, years and page counts are every edition's at once, and the first
of each is usually another edition's. So for a book it gives only the title,
the authors and the subjects, and the `container`, `language`, `year` and
`pages` of a record come from the edition record for that ISBN, or from
Google Books when the edition record does not say. A field neither gives is
left out rather than filled from the work.
Google Books is asked last because it answers anonymous callers only until a Google Books is asked last because it answers anonymous callers only until a
daily quota runs out, after which it replies `429` to everyone sharing the daily quota runs out, after which it replies `429` to everyone sharing the
address. address.
@@ -127,7 +135,8 @@ id is asked only when every form of the ISBN asked was answered with no cover; i
asking by ISBN failed, that failure is the result. asking by ISBN failed, that failure is the result.
What comes back is checked before it is written: a JPEG or PNG by its bytes What comes back is checked before it is written: a JPEG or PNG by its bytes
as well as by what the server says, no larger than ten megabytes, reached in at as well as by what the server says, whole to its end marker (`FF D9` for a
JPEG, `IEND` for a PNG), no larger than ten megabytes, reached in at
most three redirects, each over HTTPS to `covers.openlibrary.org`, most three redirects, each over HTTPS to `covers.openlibrary.org`,
`archive.org` or a host under `archive.org`, and never to an address by number. `archive.org` or a host under `archive.org`, and never to an address by number.
One request is made at a time, a `429` or `503` is waited out for as long as One request is made at a time, a `429` or `503` is waited out for as long as
@@ -136,6 +145,14 @@ redirect included, stops after two minutes. The two minutes start once it is
the fetch's turn, so a fetch queued behind others in the same process can wait the fetch's turn, so a fetch queued behind others in the same process can wait
that long again first. that long again first.
Open Library allows a hundred covers by ISBN every five minutes from one
address, and refuses the rest with `403`. Covers asked by ISBN are kept three
seconds apart across the whole process (`Covers::pace`), which stays under that
limit however long a run goes. A refusal all the same is not mistaken for an
outage: the other form of the ISBN is not asked, the cover id is, because
Open Library does not limit those, and with no cover found that way the fetch
fails with `Error::Limited`, which means ask again later.
| Option | Effect | | Option | Effect |
|---|---| |---|---|
| `-o`, `--output <FILE>` | Where to write the image | | `-o`, `--output <FILE>` | Where to write the image |
@@ -144,7 +161,8 @@ that long again first.
Exit `0` the cover was written, `1` Open Library has none, `65` the argument Exit `0` the cover was written, `1` Open Library has none, `65` the argument
was not an ISBN or the answer was not an image, `69` Open Library could not be was not an ISBN or the answer was not an image, `69` Open Library could not be
reached, `74` the file could not be written. reached, `74` the file could not be written, `75` Open Library refused for
being asked too often and will answer later.
From Rust, `bibfetch::cover(&agent, isbn, cover_id, size)` returns From Rust, `bibfetch::cover(&agent, isbn, cover_id, size)` returns
`Ok(None)` when there is no cover, and a [`Cover`](src/cover.rs) with the `Ok(None)` when there is no cover, and a [`Cover`](src/cover.rs) with the
@@ -154,7 +172,7 @@ bytes, the media type and the address asked when there is.
``` ```
$ bibfetch search --title 'The First World War' --author Howard --limit 2 $ bibfetch search --title 'The First World War' --author Howard --limit 2
{"v":1,"id":"9780192853622","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War","authors":["Michael Eliot Howard"],"year":2002,"container":"Oxford University Press","subjects":["World War, 1914-1918"],"language":"eng","pages":154,"cover_id":1123124} {"v":1,"id":"9780192853622","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War","authors":["Michael Eliot Howard"],"year":2002,"container":"Oxford University Press","subjects":["World War, 1914-1918"],"language":"eng","cover_id":1123124}
{"v":1,"id":"9780199205592","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War: A Very Short Introduction","authors":["Michael Howard"],"year":2007,"container":"Oxford University Press","language":"eng","cover_id":14606475} {"v":1,"id":"9780199205592","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War: A Very Short Introduction","authors":["Michael Howard"],"year":2007,"container":"Oxford University Press","language":"eng","cover_id":14606475}
``` ```
@@ -163,18 +181,33 @@ title and perhaps a name. This asks which books a register has under those,
and prints one record per candidate: Open Library's index first, and Google and prints one record per candidate: Open Library's index first, and Google
Books only when Open Library has nothing. Books only when Open Library has nothing.
Open Library's title search is strict. Given an author as well, it finds
nothing for *The Ladder of Divine Ascent* and four editions of *Ladder of
Divine Ascent*. So when the title as given finds nothing, the title without a
leading "The", "A" or "An" is asked next, and then the title and author as one
free-text query. Each is asked only when the one before found nothing, so a
title that matches costs one request. The looser questions can bring back
books that only share words with the title, which is one more reason the
candidates are for the caller to judge.
Each candidate is one edition. Open Library's index answers with works, and Each candidate is one edition. Open Library's index answers with works, and
names the edition of each that best matches the search; what belongs to an names the edition of each that best matches the search. The ISBN, publisher,
edition — its ISBN, publisher, language, year and cover — is that edition's, language, year and cover are that edition's and no other's: a field the
and what belongs to the work — the title, the authors, the subjects and the edition does not give is left out rather than filled from the work, whose
length — is the work's. Google Books answers with volumes, which are editions publishers, years and ISBNs belong to all its editions at once. Only the
already. title, the authors and the subjects, which describe the work, are the work's.
Open Library's index gives no length for an edition, so its candidates carry
no `pages`. Google Books answers with volumes, which are editions already.
| Field | When the candidate has an ISBN | When it has none | | Field | When the candidate has an ISBN | When it has none |
|---|---|---| |---|---|---|
| `id` | The first ISBN whose check digit agrees, in thirteen digits; one written in thirteen is taken over one written in ten | The register's own key: an Open Library id such as `OL37863121M` for the edition, or `OL267171W` for the work when no edition was named; a Google Books volume id | | `id` | The first ISBN of the edition whose check digit agrees, in thirteen digits; one written in thirteen is taken over one written in ten | The register's own key: an Open Library id such as `OL37863121M` for the edition, or `OL267171W` for the work when no edition was named; a Google Books volume id |
| `kind` | `isbn` | `olid` for Open Library, `googlebooks` for Google Books | | `kind` | `isbn` | `olid` for Open Library, `googlebooks` for Google Books |
A work the index names no edition of, or names one it gives neither an ISBN
nor a key for, is a candidate named by the work's key, with its title,
authors and subjects and nothing an edition would say.
An ISBN whose check digit is wrong is passed over rather than corrected, and An ISBN whose check digit is wrong is passed over rather than corrected, and
Google's shelfmarks, typed `OTHER`, are not read as ISBNs even when their Google's shelfmarks, typed `OTHER`, are not read as ISBNs even when their
digits would pass for one. A record of kind `isbn` can be handed straight back digits would pass for one. A record of kind `isbn` can be handed straight back
@@ -185,9 +218,8 @@ to `bibfetch` for everything the registers say about that edition; `olid` and
Open Library's record of the edition the record names, by that edition's ISBN Open Library's record of the edition the record names, by that edition's ISBN
or its key, so `bibfetch cover --cover-id` fetches that edition's cover and no or its key, so `bibfetch cover --cover-id` fetches that edition's cover and no
other. The cover the index picked for the whole work is never asked for, other. The cover the index picked for the whole work is never asked for,
because it is often another edition's. A work the index names no edition of because it is often another edition's. Google Books knows nothing of Open
gives the first ISBN it lists that reads, which may be any edition's, and no Library's covers and gives none.
cover id; Google Books knows nothing of Open Library's covers and gives none.
The candidates are what the register said, in the order it said them. This The candidates are what the register said, in the order it said them. This
makes no judgement about which one is the book in hand, and keeps one that is makes no judgement about which one is the book in hand, and keeps one that is
@@ -201,10 +233,14 @@ bibfetch search --title 'Anathem' --author Stephenson \
| jq -r 'select(.kind == "isbn") | .id' | head -1 | jq -r 'select(.kind == "isbn") | .id' | head -1
``` ```
Open Library answering that it has nothing is an answer, and it stands when Open Library having nothing is an answer only once Google Books has been
Google Books cannot be reached; a register being down is a failure only when asked and has nothing either. When Google Books cannot be reached, the search
both are. Each is asked the same way a lookup asks, retried with the same fails rather than say no register has the title, and when it refuses for its
waits. Open Library is sent the title and the author as written; Google Books quota (`403`, or `429` on every attempt) the failure says so and the command
exits `75`. Open Library being down is a failure only when Google Books has
nothing to offer in its place. Each is asked the same way a lookup asks,
retried with the same waits, except that a `403` from Google Books is not
asked again. Open Library is sent the title and the author as written; Google Books
is sent each word as its own `intitle:` or `inauthor:` term, so nothing in a is sent each word as its own `intitle:` or `inauthor:` term, so nothing in a
title is read as search syntax. title is read as search syntax.
@@ -215,8 +251,9 @@ title is read as search syntax.
| `--limit <COUNT>` | The most candidates to print, `1` to `40`. Default `10` | | `--limit <COUNT>` | The most candidates to print, `1` to `40`. Default `10` |
Exit `0` at least one candidate was printed, `1` no register has the title, Exit `0` at least one candidate was printed, `1` no register has the title,
`65` the title was empty, `69` no register could be reached, `74` the output `65` the title was empty, `69` a register that had to answer could not be
could not be written. reached, `74` the output could not be written, `75` Google Books refused for
its quota.
From Rust, `bibfetch::search(&agent, title, author, limit, attempts)` returns From Rust, `bibfetch::search(&agent, title, author, limit, attempts)` returns
the candidates, empty when no register has the title. the candidates, empty when no register has the title.
crates/bibfetch/src/cover.rs
+204 -11
View File
@@ -46,6 +46,11 @@ const REDIRECT_HOSTS: [&str; 2] = ["covers.openlibrary.org", "archive.org"];
/// agent's timeout each, and a minute's `Retry-After` between attempts. /// agent's timeout each, and a minute's `Retry-After` between attempts.
pub const DEADLINE: Duration = Duration::from_secs(120); pub const DEADLINE: Duration = Duration::from_secs(120);
/// How long to leave between two covers asked by ISBN. Open Library allows a
/// hundred every five minutes from one address and refuses the rest with
/// `403`; one every three seconds stays under that however long a run goes.
pub const PACE: Duration = Duration::from_secs(3);
/// The longest a `Retry-After` is honoured for. A register asking for more /// The longest a `Retry-After` is honoured for. A register asking for more
/// than this is better treated as down than waited on. /// than this is better treated as down than waited on.
const MAX_WAIT: Duration = Duration::from_secs(60); const MAX_WAIT: Duration = Duration::from_secs(60);
@@ -54,7 +59,10 @@ const MAX_WAIT: Duration = Duration::from_secs(60);
/// this, and limits covers asked by ISBN to a hundred every five minutes /// this, and limits covers asked by ISBN to a hundred every five minutes
/// from one address; threads asking at once would only meet the limit /// from one address; threads asking at once would only meet the limit
/// sooner and all back off together. /// sooner and all back off together.
static ONE_AT_A_TIME: Mutex<()> = Mutex::new(()); ///
/// It holds when the last cover was asked for by ISBN, so that [`PACE`] is
/// kept across fetches and threads.
static ONE_AT_A_TIME: Mutex<Option<Instant>> = Mutex::new(None);
/// Which of Open Library's three sizes to ask for. /// Which of Open Library's three sizes to ask for.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
@@ -127,6 +135,9 @@ pub struct Covers {
/// request is given whatever is left of it, or the agent's own timeout /// request is given whatever is left of it, or the agent's own timeout
/// if that is shorter, and a wait that would run past it is not waited. /// if that is shorter, and a wait that would run past it is not waited.
pub deadline: Duration, pub deadline: Duration,
/// The least time between two covers asked by ISBN; [`PACE`] by default.
/// Covers asked by id are not limited and are not spaced out.
pub pace: Duration,
} }
impl Default for Covers { impl Default for Covers {
@@ -135,6 +146,7 @@ impl Default for Covers {
base: OPEN_LIBRARY_COVERS.to_string(), base: OPEN_LIBRARY_COVERS.to_string(),
attempts: crate::ATTEMPTS, attempts: crate::ATTEMPTS,
deadline: DEADLINE, deadline: DEADLINE,
pace: PACE,
} }
} }
} }
@@ -162,6 +174,12 @@ impl Covers {
/// of the ISBN does not stop the other being asked, but it does stop the /// of the ISBN does not stop the other being asked, but it does stop the
/// cover id being asked: the ISBN's own cover may exist, and is the one /// cover id being asked: the ISBN's own cover may exist, and is the one
/// wanted. The failure is returned when no ISBN form gave a cover. /// wanted. The failure is returned when no ISBN form gave a cover.
///
/// Refusal for asking too often is the exception. It is
/// [`Error::Limited`], the other form is not asked, since it would only
/// be refused too, and the cover id is still asked, because Open Library
/// does not limit those. A cover found that way is returned; otherwise the
/// refusal is, so that the caller knows to ask again later.
pub fn fetch( pub fn fetch(
&self, &self,
agent: &ureq::Agent, agent: &ureq::Agent,
@@ -186,31 +204,54 @@ impl Covers {
.map(|form| format!("{}/b/isbn/{form}-{letter}.jpg?default=false", self.base)) .map(|form| format!("{}/b/isbn/{form}-{letter}.jpg?default=false", self.base))
.collect(); .collect();
let _turn = ONE_AT_A_TIME let mut last_by_isbn = ONE_AT_A_TIME
.lock() .lock()
.unwrap_or_else(|held| held.into_inner()); .unwrap_or_else(|held| held.into_inner());
// From here, not from the call: waiting for the lock is time another // From here, not from the call: waiting for the lock is time another
// fetch spent, and counting it would let a busy process starve this // fetch spent, and counting it would let a busy process starve this
// one of the time its own requests need. // one of the time its own requests need. A deadline too long to add
let until = Instant::now() + self.deadline; // to the clock is no deadline at all.
let now = Instant::now();
let until = now
.checked_add(self.deadline)
.unwrap_or_else(|| now + Duration::from_secs(365 * 24 * 60 * 60));
let mut failed = None; let mut failed = None;
for address in &addresses { for address in &addresses {
match self.ask(agent, address, until) { if let Some(ready) = last_by_isbn.and_then(|at| at.checked_add(self.pace)) {
let now = Instant::now();
if ready > now {
if ready >= until {
return Err(Error::Unreachable(format!(
"{address}: past the {:?} deadline, waiting to keep {:?} \
between covers asked by ISBN",
self.deadline, self.pace
)));
}
std::thread::sleep(ready - now);
}
}
let answer = self.ask(agent, address, until);
*last_by_isbn = Some(Instant::now());
match answer {
Ok(Answer::Image(cover)) => return Ok(Some(cover)), Ok(Answer::Image(cover)) => return Ok(Some(cover)),
Ok(Answer::Absent) => {} Ok(Answer::Absent) => {}
Err(why @ Error::Limited(_)) => {
failed = Some(why);
break;
}
Err(why) => failed = Some(why), Err(why) => failed = Some(why),
} }
} }
if let Some(why) = failed { if let Some(why) = failed.take_if(|why| !matches!(why, Error::Limited(_))) {
return Err(why); return Err(why);
} }
let Some(id) = cover_id else { let Some(id) = cover_id else {
return Ok(None); return failed.map_or(Ok(None), Err);
}; };
let address = format!("{}/b/id/{id}-{letter}.jpg?default=false", self.base); let address = format!("{}/b/id/{id}-{letter}.jpg?default=false", self.base);
match self.ask(agent, &address, until)? { match self.ask(agent, &address, until)? {
Answer::Image(cover) => Ok(Some(cover)), Answer::Image(cover) => Ok(Some(cover)),
Answer::Absent => Ok(None), Answer::Absent => failed.map_or(Ok(None), Err),
} }
} }
@@ -311,6 +352,15 @@ Open Library's covers or the Internet Archive"
// `default=false` makes a missing cover a 404 rather than a // `default=false` makes a missing cover a 404 rather than a
// stand-in image. // stand-in image.
404 | 410 => return Ok(Answer::Absent), 404 | 410 => return Ok(Answer::Absent),
// Open Library's answer to more than a hundred covers asked by
// ISBN in five minutes from one address. Asking again at once
// would only be refused again.
403 => {
return Err(Fault::Final(Error::Limited(format!(
"{at}: refused (403); Open Library allows a hundred covers \
by ISBN every five minutes from one address"
))));
}
429 | 503 => { 429 | 503 => {
let wait = header(&response, "retry-after") let wait = header(&response, "retry-after")
.and_then(|value| value.trim().parse::<u64>().ok()) .and_then(|value| value.trim().parse::<u64>().ok())
@@ -336,6 +386,23 @@ Open Library's covers or the Internet Archive"
} }
} }
/// Whether an image's bytes run to its end marker: `FF D9` for a JPEG, which
/// some encoders follow with zero padding, and the `IEND` chunk for a PNG,
/// which is its last twelve bytes.
fn ends_whole(media_type: &str, bytes: &[u8]) -> bool {
match media_type {
"image/jpeg" => {
let end = bytes.iter().rposition(|&b| b != 0).map_or(0, |at| at + 1);
end >= 5 && bytes[..end].ends_with(&[0xFF, 0xD9])
}
_ => {
bytes.len() >= 8 + 12
&& bytes[bytes.len() - 12..].starts_with(&[0, 0, 0, 0])
&& &bytes[bytes.len() - 8..bytes.len() - 4] == b"IEND"
}
}
}
/// Read and check a body that is meant to be a cover. /// Read and check a body that is meant to be a cover.
fn read(mut response: ureq::http::Response<ureq::Body>, address: &str) -> Result<Answer, Error> { fn read(mut response: ureq::http::Response<ureq::Body>, address: &str) -> Result<Answer, Error> {
let declared = header(&response, "content-type") let declared = header(&response, "content-type")
@@ -388,6 +455,14 @@ fn read(mut response: ureq::http::Response<ureq::Body>, address: &str) -> Result
"{address}: said {media_type} but is not one" "{address}: said {media_type} but is not one"
))); )));
} }
// The start of an image is not an image. A body sent with no length,
// or a stub, can begin right and stop short; what ends it says it is
// whole.
if !ends_whole(media_type, &bytes) {
return Err(Error::Shape(format!(
"{address}: a {media_type} that stops before its end"
)));
}
// No floor on the size. Open Library's stand-in for a missing cover is a // No floor on the size. Open Library's stand-in for a missing cover is a
// 43-byte, one-pixel GIF, and it serves that only when `default=false` // 43-byte, one-pixel GIF, and it serves that only when `default=false`
// is left off: with it, as every address here has it, a missing cover is // is left off: with it, as every address here has it, a missing cover is
@@ -462,7 +537,12 @@ fn origin(url: &str) -> Option<&str> {
/// Where a `Location` points, read against the address that sent it. /// Where a `Location` points, read against the address that sent it.
fn resolve(from: &str, location: &str) -> String { fn resolve(from: &str, location: &str) -> String {
if location.contains("://") { // Absolute only when the scheme comes first: a relative address can
// carry another address in its query.
if location
.find("://")
.is_some_and(|at| !location[..at].contains(['/', '?', '#']))
{
return location.to_string(); return location.to_string();
} }
if let Some(rest) = location.strip_prefix("//") { if let Some(rest) = location.strip_prefix("//") {
@@ -603,6 +683,7 @@ mod tests {
base: self.base.clone(), base: self.base.clone(),
attempts: 3, attempts: 3,
deadline: DEADLINE, deadline: DEADLINE,
pace: Duration::ZERO,
} }
} }
@@ -619,15 +700,19 @@ mod tests {
.agent() .agent()
} }
/// A JPEG's start, filler, and its end marker, `len` bytes in all.
fn jpeg(len: usize) -> Vec<u8> { fn jpeg(len: usize) -> Vec<u8> {
let mut bytes = vec![0xFF, 0xD8, 0xFF, 0xE0]; let mut bytes = vec![0xFF, 0xD8, 0xFF, 0xE0];
bytes.resize(len, 0x11); bytes.resize(len - 2, 0x11);
bytes.extend([0xFF, 0xD9]);
bytes bytes
} }
/// A PNG's signature, filler, and its `IEND` chunk, `len` bytes in all.
fn png(len: usize) -> Vec<u8> { fn png(len: usize) -> Vec<u8> {
let mut bytes = b"\x89PNG\r\n\x1a\n".to_vec(); let mut bytes = b"\x89PNG\r\n\x1a\n".to_vec();
bytes.resize(len, 0x22); bytes.resize(len - 12, 0x22);
bytes.extend(b"\0\0\0\0IEND\xAE\x42\x60\x82");
bytes bytes
} }
@@ -1052,6 +1137,105 @@ mod tests {
assert_eq!(stub.asked(), [ISBN13, ISBN10]); assert_eq!(stub.asked(), [ISBN13, ISBN10]);
} }
#[test]
fn a_refusal_for_asking_too_often_still_asks_the_cover_id() {
let stub = Stub::start(|_, path| match path {
ISBN13 => Reply::status(403),
"/b/id/42-L.jpg?default=false" => Reply::image("image/jpeg", jpeg(4096)),
_ => Reply::status(404),
});
let cover = stub
.covers()
.fetch(&agent(), "9780306406157", Some(42), CoverSize::Large)
.expect("an answer")
.expect("a cover");
assert!(cover.url.ends_with("/b/id/42-L.jpg?default=false"));
// The ten-digit form would only have been refused too.
assert_eq!(stub.asked(), [ISBN13, "/b/id/42-L.jpg?default=false"]);
}
#[test]
fn a_refusal_for_asking_too_often_is_not_an_absence() {
let stub = Stub::start(|_, path| match path {
ISBN13 => Reply::status(403),
_ => Reply::status(404),
});
let found = stub
.covers()
.fetch(&agent(), "9780306406157", None, CoverSize::Large)
.map(|cover| cover.map(|cover| cover.url));
assert!(matches!(found, Err(Error::Limited(_))), "{found:?}");
assert_eq!(stub.asked(), [ISBN13]);
// With a cover id that has no cover either, the refusal still stands:
// the ISBN's own cover may be there once the limit passes.
let found = stub
.covers()
.fetch(&agent(), "9780306406157", Some(42), CoverSize::Large)
.map(|cover| cover.map(|cover| cover.url));
assert!(matches!(found, Err(Error::Limited(_))), "{found:?}");
}
#[test]
fn a_deadline_too_long_to_count_is_no_deadline() {
let stub = Stub::start(|_, _| Reply::image("image/jpeg", jpeg(2048)));
let covers = Covers {
deadline: Duration::MAX,
..stub.covers()
};
let cover = covers
.fetch(&agent(), "9780306406157", None, CoverSize::Large)
.expect("an answer");
assert!(cover.is_some());
}
#[test]
fn covers_asked_by_isbn_are_spaced_out() {
let stub = Stub::start(|_, _| Reply::status(404));
let covers = Covers {
pace: Duration::from_millis(300),
..stub.covers()
};
let started = Instant::now();
let found = covers.fetch(&agent(), "9780306406157", None, CoverSize::Large);
assert!(matches!(found, Ok(None)), "{found:?}");
// Two forms of the ISBN, so at least one pause between them.
assert!(started.elapsed() >= Duration::from_millis(300));
assert_eq!(stub.asked(), [ISBN13, ISBN10]);
}
#[test]
fn a_magic_number_alone_is_not_a_cover() {
let mut cut_jpeg = jpeg(4096);
cut_jpeg.truncate(4000);
let mut cut_png = png(4096);
cut_png.truncate(4000);
for (media_type, body) in [
("image/jpeg", vec![0xFF, 0xD8, 0xFF]),
("image/png", b"\x89PNG\r\n\x1a\n".to_vec()),
("image/jpeg", cut_jpeg),
("image/png", cut_png),
] {
let stub = Stub::start(move |_, _| Reply::image(media_type, body.clone()));
let found = stub
.covers()
.fetch(&agent(), "9780306406157", None, CoverSize::Large)
.map(|cover| cover.map(|cover| cover.bytes.len()));
assert!(
matches!(found, Err(Error::Shape(_))),
"{media_type}: {found:?}"
);
}
}
#[test]
fn a_jpeg_padded_after_its_end_marker_is_whole() {
let mut padded = jpeg(2048);
padded.extend([0, 0, 0]);
assert!(ends_whole("image/jpeg", &padded));
assert!(!ends_whole("image/jpeg", &[0xFF, 0xD8, 0xFF, 0, 0]));
}
#[test] #[test]
fn a_redirect_to_an_address_by_number_is_refused() { fn a_redirect_to_an_address_by_number_is_refused() {
// A listener standing in for a service on the LAN; it only counts. // A listener standing in for a service on the LAN; it only counts.
@@ -1099,6 +1283,15 @@ mod tests {
assert_eq!(resolve(from, "//c.example/x"), "https://c.example/x"); assert_eq!(resolve(from, "//c.example/x"), "https://c.example/x");
assert_eq!(resolve(from, "/x?y=1"), "https://a.example/x?y=1"); assert_eq!(resolve(from, "/x?y=1"), "https://a.example/x?y=1");
assert_eq!(resolve(from, "x.jpg"), "https://a.example/b/isbn/x.jpg"); assert_eq!(resolve(from, "x.jpg"), "https://a.example/b/isbn/x.jpg");
// An address in a relative one's query does not make it absolute.
assert_eq!(
resolve(from, "/view_archive.php?from=https://archive.org/x"),
"https://a.example/view_archive.php?from=https://archive.org/x"
);
assert_eq!(
resolve(from, "x?u=https://c.example/"),
"https://a.example/b/isbn/x?u=https://c.example/"
);
assert_eq!( assert_eq!(
origin("http://127.0.0.1:8080/a?b"), origin("http://127.0.0.1:8080/a?b"),
Some("http://127.0.0.1:8080") Some("http://127.0.0.1:8080")
crates/bibfetch/src/lib.rs
+5 -2
View File
@@ -112,6 +112,9 @@ pub enum Error {
NoRegister(String), NoRegister(String),
/// What was asked about is not an identifier of the kind it should be. /// What was asked about is not an identifier of the kind it should be.
Invalid(String), Invalid(String),
/// The register refused because it has been asked too often from this
/// address. It is not down, and asking again later will work.
Limited(String),
} }
impl std::fmt::Display for Error { impl std::fmt::Display for Error {
@@ -123,6 +126,7 @@ impl std::fmt::Display for Error {
write!(f, "no register here answers about {kind} identifiers") write!(f, "no register here answers about {kind} identifiers")
} }
Error::Invalid(why) => write!(f, "{why}"), Error::Invalid(why) => write!(f, "{why}"),
Error::Limited(why) => write!(f, "{why}"),
} }
} }
} }
@@ -261,11 +265,10 @@ mod merge_tests {
let mut index = rec("openlibrary"); let mut index = rec("openlibrary");
index.title = Some("The First World War".into()); index.title = Some("The First World War".into());
index.authors = vec!["Michael Howard".into()]; index.authors = vec!["Michael Howard".into()];
index.year = Some(2007);
edition.fill_from(index); edition.fill_from(index);
assert_eq!(edition.authors, ["Michael Howard"]); assert_eq!(edition.authors, ["Michael Howard"]);
assert_eq!(edition.year, Some(2007)); assert_eq!(edition.language.as_deref(), Some("eng"));
// The fuller title arrived first and is not displaced by the shorter. // The fuller title arrived first and is not displaced by the shorter.
assert_eq!( assert_eq!(
edition.title.as_deref(), edition.title.as_deref(),
crates/bibfetch/src/main.rs
+2
View File
@@ -225,6 +225,7 @@ fn cover(
eprintln!("bibfetch: {isbn}: {why}"); eprintln!("bibfetch: {isbn}: {why}");
ExitCode::from(match why { ExitCode::from(match why {
bibfetch::Error::Unreachable(_) => exit::UNAVAILABLE, bibfetch::Error::Unreachable(_) => exit::UNAVAILABLE,
bibfetch::Error::Limited(_) => exit::TEMPFAIL,
_ => exit::DATA, _ => exit::DATA,
}) })
} }
@@ -255,6 +256,7 @@ fn search(
eprintln!("bibfetch: {why}"); eprintln!("bibfetch: {why}");
ExitCode::from(match why { ExitCode::from(match why {
bibfetch::Error::Invalid(_) => exit::DATA, bibfetch::Error::Invalid(_) => exit::DATA,
bibfetch::Error::Limited(_) => exit::TEMPFAIL,
_ => exit::UNAVAILABLE, _ => exit::UNAVAILABLE,
}) })
} }
crates/bibfetch/src/search.rs
+329 -105
View File
@@ -23,22 +23,34 @@ pub const MAX_LIMIT: usize = 40;
/// What Open Library's index is asked to say about each work, and about the /// What Open Library's index is asked to say about each work, and about the
/// one edition of it that best matches the search. /// one edition of it that best matches the search.
/// ///
/// Not the work's own `cover_i` or `cover_edition_key`: the cover the index /// Of the work, only what describes the work: its publishers, ISBNs,
/// chose for a work is often another edition's. The edition's `cover_i` is /// languages, years and page counts are every edition's at once, and a
/// the first cover in that edition's record, which is what a lookup by ISBN /// record made from them would describe no edition. Not the work's own
/// gives as `cover_id`. /// `cover_i` or `cover_edition_key` either: the cover the index chose for a
const OPEN_LIBRARY_FIELDS: &str = "key,title,subtitle,author_name,first_publish_year,publisher,\ /// work is often another edition's. The edition's `cover_i` is the first
isbn,language,subject,number_of_pages_median,editions,editions.key,editions.isbn,\ /// cover in that edition's record, which is what a lookup by ISBN gives as
editions.publisher,editions.language,editions.publish_date,editions.cover_i"; /// `cover_id`.
const OPEN_LIBRARY_FIELDS: &str = "key,title,subtitle,author_name,subject,editions,editions.key,\
editions.isbn,editions.publisher,editions.language,editions.publish_date,editions.cover_i";
/// Where Google Books is asked.
const GOOGLE_BOOKS: &str = "https://www.googleapis.com/books/v1/volumes";
/// Ask which books a register has under a title, and by an author if one is /// Ask which books a register has under a title, and by an author if one is
/// named. /// named.
/// ///
/// Open Library's index first, and Google Books only when that answered with /// Open Library's index first, and Google Books only when that answered with
/// nothing. A register being down is a failure only when every register /// nothing. Open Library's title search is strict: given an author too, it
/// asked is: Open Library answering that it has nothing is an answer, and it /// finds nothing for "The Ladder of Divine Ascent" that it finds for "Ladder
/// stands when Google Books cannot be reached. Asks nothing when `limit` is /// of Divine Ascent". So when the title as given finds nothing, the title
/// zero; a `limit` over [`MAX_LIMIT`] is taken as that. /// without a leading article is asked, and then the title and author as one
/// free-text query, before Google Books is. Each later question is asked only
/// when the one before it found nothing, and a looser one can return books
/// that only share words with the title, which the caller judges. Once Google Books has been asked, its failure is the answer:
/// [`Error::Limited`] when it refused for its quota, [`Error::Unreachable`]
/// when it could not be reached, since saying no register has the title
/// would be a claim no register made. Asks nothing when `limit` is zero; a
/// `limit` over [`MAX_LIMIT`] is taken as that.
pub fn search( pub fn search(
agent: &ureq::Agent, agent: &ureq::Agent,
title: &str, title: &str,
@@ -46,9 +58,7 @@ pub fn search(
limit: usize, limit: usize,
attempts: u32, attempts: u32,
) -> Result<Vec<Record>, Error> { ) -> Result<Vec<Record>, Error> {
search_with(title, author, limit, |url| { search_with(title, author, limit, |url| get(agent, url, attempts))
crate::ask::get(agent, url, attempts)
})
} }
/// [`search`], with how to fetch an address passed in. /// [`search`], with how to fetch an address passed in.
@@ -75,25 +85,104 @@ fn search_with(
None => Err(Error::Unreachable(format!("{url}: answered 404"))), None => Err(Error::Unreachable(format!("{url}: answered 404"))),
}; };
let open_library = let mut from_open_library = Ok(Vec::new());
fetch(open_library_url(title, author, limit)).and_then(|body| open_library(&body, limit)); for url in open_library_urls(title, author, limit) {
from_open_library = fetch(url).and_then(|body| open_library(&body, limit));
if !from_open_library.as_ref().is_ok_and(Vec::is_empty) {
break;
}
}
let open_library = from_open_library;
if open_library.as_ref().is_ok_and(|found| !found.is_empty()) { if open_library.as_ref().is_ok_and(|found| !found.is_empty()) {
return open_library; return open_library;
} }
let google = match google_url(title, author, limit) { let Some(url) = google_url(title, author, limit) else {
Some(url) => fetch(url).and_then(|body| google_books(&body, limit)),
// Nothing in the title Google could be asked by. // Nothing in the title Google could be asked by.
None => Ok(Vec::new()), return open_library;
}; };
match (open_library, google) { let google = fetch(url).and_then(|body| google_books(&body, limit))?;
(_, Ok(found)) if !found.is_empty() => Ok(found), if google.is_empty() {
(Ok(none), _) => Ok(none), // Open Library's nothing, or its failure, which Google Books having
(Err(why), _) => Err(why), // nothing either does not excuse.
return open_library;
} }
Ok(google)
} }
/// Where to ask Open Library. /// The body at an address, asked as a lookup asks, or `None` when the
/// register says there is no such thing.
///
/// Except that Google Books refusing for its quota is [`Error::Limited`]: a
/// `403` at once, since it is the day's quota and asking again within
/// seconds will not move it, and a `429` once every attempt has been refused.
fn get(agent: &ureq::Agent, url: &str, attempts: u32) -> Result<Option<String>, Error> {
let attempts = attempts.max(1);
let mut last = Error::Unreachable("no attempt was made".into());
for attempt in 0..attempts {
match agent.get(url).call() {
Ok(mut response) => {
return response
.body_mut()
.read_to_string()
.map(Some)
.map_err(|e| Error::Unreachable(e.to_string()));
}
Err(ureq::Error::StatusCode(404)) => return Ok(None),
Err(ureq::Error::StatusCode(status)) => match refused(url, status) {
Some(why) if status == 403 => return Err(why),
Some(why) => last = why,
None => last = Error::Unreachable(format!("{url}: answered {status}")),
},
Err(e) => last = Error::Unreachable(e.to_string()),
}
if attempt + 1 < attempts {
std::thread::sleep(std::time::Duration::from_secs(1 << attempt.min(5)));
}
}
Err(last)
}
/// The refusal a status from a register is, when it is one for asking too
/// often.
fn refused(url: &str, status: u16) -> Option<Error> {
(url.starts_with(GOOGLE_BOOKS) && matches!(status, 403 | 429)).then(|| {
Error::Limited(format!(
"Google Books refused ({status}): its quota for this address is spent for now"
))
})
}
/// Every question to put to Open Library, strictest first.
fn open_library_urls(title: &str, author: Option<&str>, limit: usize) -> Vec<String> {
let mut urls = vec![open_library_url(title, author, limit)];
if let Some(rest) = without_article(title) {
urls.push(open_library_url(rest, author, limit));
}
let words = match author {
Some(author) => format!("{title} {author}"),
None => title.to_string(),
};
urls.push(format!(
"https://openlibrary.org/search.json?q={}&fields={OPEN_LIBRARY_FIELDS}&limit={limit}",
source::encode(&words),
));
urls
}
/// A title without its leading English article, when it has one and
/// something follows it.
fn without_article(title: &str) -> Option<&str> {
let (first, rest) = title.split_once(char::is_whitespace)?;
let rest = rest.trim_start();
(["the", "a", "an"]
.iter()
.any(|article| first.eq_ignore_ascii_case(article))
&& !rest.is_empty())
.then_some(rest)
}
/// Where to ask Open Library by title and author.
fn open_library_url(title: &str, author: Option<&str>, limit: usize) -> String { fn open_library_url(title: &str, author: Option<&str>, limit: usize) -> String {
let author = author let author = author
.map(|author| format!("&author={}", source::encode(author))) .map(|author| format!("&author={}", source::encode(author)))
@@ -124,7 +213,7 @@ fn google_url(title: &str, author: Option<&str>, limit: usize) -> Option<String>
return None; return None;
} }
Some(format!( Some(format!(
"https://www.googleapis.com/books/v1/volumes?q={}&maxResults={limit}", "{GOOGLE_BOOKS}?q={}&maxResults={limit}",
source::encode(&terms.join(" ")) source::encode(&terms.join(" "))
)) ))
} }
@@ -162,18 +251,8 @@ struct Work {
#[serde(default)] #[serde(default)]
author_name: Vec<String>, author_name: Vec<String>,
#[serde(default)] #[serde(default)]
first_publish_year: Option<i64>,
#[serde(default)]
publisher: Vec<String>,
#[serde(default)]
isbn: Vec<String>,
#[serde(default)]
language: Vec<String>,
#[serde(default)]
subject: Vec<String>, subject: Vec<String>,
#[serde(default)] #[serde(default)]
number_of_pages_median: Option<u32>,
#[serde(default)]
editions: Option<Editions>, editions: Option<Editions>,
} }
@@ -212,10 +291,11 @@ fn olid(key: &str) -> Option<String> {
/// Read Open Library's answer into one candidate per work, in its order. /// Read Open Library's answer into one candidate per work, in its order.
/// ///
/// What belongs to an edition — its ISBN, publisher, language, year and /// What belongs to an edition — its ISBN, publisher, language, year and
/// cover — is taken from the edition the index chose as matching best; what /// cover — is taken from the edition the index chose as matching best, and
/// belongs to the work — its title, authors, subjects and length — from the /// left out when that edition does not give it; only the title, authors and
/// work. A work the index names no edition of gives its first ISBN that reads, /// subjects, which describe the work, are the work's. A work the index names
/// which may be any edition's, and no cover id. /// no edition of is named by the work's key and says nothing an edition
/// would.
fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> { fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
let reply: OpenLibraryReply = let reply: OpenLibraryReply =
serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?; serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?;
@@ -228,26 +308,47 @@ fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
} }
fn candidate(work: Work) -> Option<Record> { fn candidate(work: Work) -> Option<Record> {
let edition = work.editions.and_then(|e| e.docs.into_iter().next()); // The edition, and what it is named by: its ISBN, or its key when it has
// none. One with neither cannot be named, and is as good as none.
// The cover is kept only when the record names the edition it is the let edition = work
// cover of, by that edition's ISBN or its key. .editions
let (id, kind, edition_named) = match &edition { .and_then(|e| e.docs.into_iter().next())
Some(edition) => match first_isbn(edition.isbn.iter().map(String::as_str)) { .and_then(|edition| {
Some(isbn) => (isbn, "isbn", true), match first_isbn(edition.isbn.iter().map(String::as_str)) {
None => match edition.key.as_deref().and_then(olid) { Some(isbn) => Some((isbn, "isbn")),
Some(key) => (key, "olid", true), None => edition
None => (work.key.as_deref().and_then(olid)?, "olid", false), .key
}, .as_deref()
}, .and_then(olid)
None => match first_isbn(work.isbn.iter().map(String::as_str)) { .map(|key| (key, "olid")),
Some(isbn) => (isbn, "isbn", false), }
// A work with no key cannot be named, so it is not a candidate. .map(|named| (named, edition))
None => (work.key.as_deref().and_then(olid)?, "olid", false), });
},
};
let mut record = match edition {
Some(((id, kind), edition)) => {
let mut record = source::blank(&id, kind, Source::OpenLibrarySearch); let mut record = source::blank(&id, kind, Source::OpenLibrarySearch);
record.container = edition.publisher.into_iter().next();
record.language = edition.language.into_iter().next();
record.year = edition
.publish_date
.iter()
.find_map(|date| source::year_in(date));
record.cover_id = edition
.cover_i
.and_then(|id| u64::try_from(id).ok())
.filter(|id| *id > 0);
record
}
// The work's own ISBNs, publishers and years are its editions', and
// which belongs with which the index does not say.
None => source::blank(
// A work with no key cannot be named, so it is not a candidate.
&work.key.as_deref().and_then(olid)?,
"olid",
Source::OpenLibrarySearch,
),
};
record.title = match (work.title, work.subtitle) { record.title = match (work.title, work.subtitle) {
(Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")), (Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")),
(title, _) => title, (title, _) => title,
@@ -255,30 +356,6 @@ fn candidate(work: Work) -> Option<Record> {
record.authors = work.author_name; record.authors = work.author_name;
record.subjects = work.subject; record.subjects = work.subject;
record.subjects.truncate(SUBJECT_LIMIT); record.subjects.truncate(SUBJECT_LIMIT);
record.pages = work.number_of_pages_median;
let (publisher, language, year) = match edition {
Some(edition) => {
if edition_named {
record.cover_id = edition
.cover_i
.and_then(|id| u64::try_from(id).ok())
.filter(|id| *id > 0);
}
(
edition.publisher.into_iter().next(),
edition.language.into_iter().next(),
edition
.publish_date
.iter()
.find_map(|date| source::year_in(date)),
)
}
None => (None, None, None),
};
record.container = publisher.or_else(|| work.publisher.into_iter().next());
record.language = language.or_else(|| work.language.into_iter().next());
record.year = year.or(work.first_publish_year);
Some(record) Some(record)
} }
@@ -312,8 +389,9 @@ mod tests {
use super::*; use super::*;
/// Open Library's answer to `title=the first world war&author=howard`, /// Open Library's answer to `title=the first world war&author=howard`,
/// recorded 2026-11-13 with the fields above and `limit=2`, the ISBN and /// recorded 2026-11-13 with `limit=2`, the ISBN and subject lists
/// subject lists shortened. /// shortened, and with the work's own publishers, ISBNs, years and length,
/// which are not asked for and are read past when they come.
const HOWARD: &str = r#"{"numFound": 9, "start": 0, "numFoundExact": true, "num_found": 9, const HOWARD: &str = r#"{"numFound": 9, "start": 0, "numFoundExact": true, "num_found": 9,
"documentation_url": "https://openlibrary.org/dev/docs/api/search", "q": "", "offset": null, "documentation_url": "https://openlibrary.org/dev/docs/api/search", "q": "", "offset": null,
"docs": [ "docs": [
@@ -420,7 +498,8 @@ mod tests {
assert_eq!(first.container.as_deref(), Some("Oxford University Press")); assert_eq!(first.container.as_deref(), Some("Oxford University Press"));
assert_eq!(first.language.as_deref(), Some("eng")); assert_eq!(first.language.as_deref(), Some("eng"));
assert_eq!(first.subjects, ["World War, 1914-1918"]); assert_eq!(first.subjects, ["World War, 1914-1918"]);
assert_eq!(first.pages, Some(154)); // The work's median length is no edition's.
assert_eq!(first.pages, None);
assert_eq!(first.cover_id, Some(1123124)); assert_eq!(first.cover_id, Some(1123124));
let second = &found[1]; let second = &found[1];
@@ -474,17 +553,64 @@ mod tests {
} }
#[test] #[test]
fn a_work_with_no_edition_named_gives_its_isbn_but_no_cover() { fn what_the_edition_does_not_give_is_left_out_not_taken_from_the_work() {
let body = r#"{"docs": [{"key": "/works/OL1W", "title": "A Book", let body = r#"{"docs": [{"key": "/works/OL267171W", "title": "War and Peace",
"isbn": ["0-306-40615-2"], "publisher": ["Plenum"], "first_publish_year": 1981}]}"#; "author_name": ["Leo Tolstoy"], "subject": ["Napoleonic Wars"],
"first_publish_year": 1867, "publisher": ["Crowell"], "language": ["eng"],
"number_of_pages_median": 1296,
"editions": {"docs": [{"key": "/books/OL1M", "isbn": ["9780199232765"],
"cover_i": 5}]}}]}"#;
let found = open_library(body, 5).expect("candidates"); let found = open_library(body, 5).expect("candidates");
let only = &found[0];
assert_eq!( assert_eq!(
(found[0].id.as_str(), found[0].kind.as_str()), (only.id.as_str(), only.kind.as_str()),
("9780306406157", "isbn") ("9780199232765", "isbn")
); );
assert_eq!(found[0].container.as_deref(), Some("Plenum")); assert_eq!(only.cover_id, Some(5));
assert_eq!(found[0].year, Some(1981)); // 1867 and Crowell are other editions'.
assert_eq!(found[0].cover_id, None); assert_eq!(only.year, None);
assert_eq!(only.container, None);
assert_eq!(only.language, None);
assert_eq!(only.pages, None);
// What describes the work is the work's.
assert_eq!(only.title.as_deref(), Some("War and Peace"));
assert_eq!(only.authors, ["Leo Tolstoy"]);
assert_eq!(only.subjects, ["Napoleonic Wars"]);
}
#[test]
fn a_work_with_no_edition_named_is_named_by_its_key_and_says_nothing_an_edition_would() {
let work = r#""key": "/works/OL1W", "title": "A Book", "author_name": ["A. Writer"],
"isbn": ["0-306-40615-2"], "publisher": ["Plenum"], "first_publish_year": 1981,
"language": ["eng"], "number_of_pages_median": 300"#;
// No edition, and an edition with neither an ISBN nor a key.
for editions in [
"",
r#", "editions": {"docs": [{"publisher": ["Plenum"], "cover_i": 7}]}"#,
] {
let body = format!(r#"{{"docs": [{{{work}{editions}}}]}}"#);
let found = open_library(&body, 5).expect("candidates");
let only = &found[0];
// Not 9780306406157: which edition that is, the index does not say.
assert_eq!(
(only.id.as_str(), only.kind.as_str()),
("OL1W", "olid"),
"{body}"
);
assert_eq!(only.title.as_deref(), Some("A Book"));
assert_eq!(only.authors, ["A. Writer"]);
assert_eq!(
(
only.year,
only.container.as_deref(),
only.language.as_deref(),
only.pages,
only.cover_id
),
(None, None, None, None, None),
"{body}"
);
}
} }
#[test] #[test]
@@ -545,8 +671,19 @@ mod tests {
recorded(&mut asked, &bodies), recorded(&mut asked, &bodies),
) )
.expect("candidates"); .expect("candidates");
assert_eq!(asked.len(), 2); // The title as given, without its article, as free text, then Google.
assert!(asked[1].starts_with("https://www.googleapis.com/books/v1/volumes?q=")); assert_eq!(asked.len(), 4, "{asked:?}");
assert!(
asked[1].contains("?title=Google%20Story&author=Vise&"),
"{}",
asked[1]
);
assert!(
asked[2].contains("?q=The%20Google%20Story%20Vise&"),
"{}",
asked[2]
);
assert!(asked[3].starts_with("https://www.googleapis.com/books/v1/volumes?q="));
assert_eq!(found.len(), 2); assert_eq!(found.len(), 2);
let first = &found[0]; let first = &found[0];
@@ -593,15 +730,46 @@ mod tests {
} }
#[test] #[test]
fn open_library_having_nothing_stands_when_google_books_is_down() { fn google_books_down_after_open_library_had_nothing_is_a_failure() {
let mut asked = Vec::new(); let mut asked = Vec::new();
let bodies = [ let bodies = [
("openlibrary.org", Ok(NOTHING)), ("openlibrary.org", Ok(NOTHING)),
("googleapis.com", Err(down())), ("googleapis.com", Err(down())),
]; ];
let found = let failed = search_with("Nothing", None, 5, recorded(&mut asked, &bodies));
search_with("Nothing", None, 5, recorded(&mut asked, &bodies)).expect("an answer"); assert!(matches!(failed, Err(Error::Unreachable(_))), "{failed:?}");
assert!(found.is_empty()); // By title, as free text, then Google.
assert_eq!(asked.len(), 3, "{asked:?}");
}
#[test]
fn google_books_refusing_for_its_quota_is_said_to_be_that() {
let mut asked = Vec::new();
let limited = refused(&google_url("Nothing", None, 5).expect("a url"), 429);
let bodies = [
("openlibrary.org", Ok(NOTHING)),
("googleapis.com", Err(limited.expect("a refusal"))),
];
let failed = search_with("Nothing", None, 5, recorded(&mut asked, &bodies));
assert!(matches!(failed, Err(Error::Limited(_))), "{failed:?}");
}
#[test]
fn only_google_books_quota_statuses_are_refusals() {
let google = google_url("Anathem", None, 5).expect("a url");
for status in [403, 429] {
assert!(
matches!(refused(&google, status), Some(Error::Limited(_))),
"{status}"
);
}
for status in [400, 500, 503] {
assert!(refused(&google, status).is_none(), "{status}");
}
let open_library = open_library_url("Anathem", None, 5);
for status in [403, 429] {
assert!(refused(&open_library, status).is_none(), "{status}");
}
} }
#[test] #[test]
@@ -650,8 +818,50 @@ mod tests {
]; ];
search_with("Anathem", Some(" "), 5, recorded(&mut asked, &bodies)).expect("ok"); search_with("Anathem", Some(" "), 5, recorded(&mut asked, &bodies)).expect("ok");
assert!(!asked[0].contains("author="), "{}", asked[0]); assert!(!asked[0].contains("author="), "{}", asked[0]);
assert!(!asked[1].contains("inauthor"), "{}", asked[1]); assert!(asked[1].contains("?q=Anathem&"), "{}", asked[1]);
assert!(asked[1].contains("q=intitle%3AAnathem&"), "{}", asked[1]); assert!(!asked[2].contains("inauthor"), "{}", asked[2]);
assert!(asked[2].contains("q=intitle%3AAnathem&"), "{}", asked[2]);
}
#[test]
fn a_leading_article_is_dropped_when_the_title_as_given_finds_nothing() {
let mut asked = Vec::new();
let mut answered = 0;
let found = search_with(
"The Ladder of Divine Ascent",
Some("John Climacus"),
5,
|url: &str| {
asked.push(url.to_string());
answered += 1;
Ok(Some(
if answered == 2 { TOLSTOY } else { NOTHING }.to_string(),
))
},
)
.expect("candidates");
assert!(!found.is_empty());
assert_eq!(
asked.len(),
2,
"nothing is asked after a question finds something"
);
assert!(
asked[1].contains("?title=Ladder%20of%20Divine%20Ascent&author=John%20Climacus&"),
"{}",
asked[1]
);
}
#[test]
fn only_a_leading_article_with_words_after_it_is_dropped() {
assert_eq!(without_article("The Republic"), Some("Republic"));
assert_eq!(without_article("an Essay"), Some("Essay"));
assert_eq!(without_article("A"), None);
assert_eq!(without_article("The "), None);
assert_eq!(without_article("Theology"), None);
assert_eq!(without_article("Anathem"), None);
assert_eq!(open_library_urls("Anathem", None, 5).len(), 2);
} }
#[test] #[test]
@@ -723,7 +933,9 @@ intitle:y intitle:z inauthor:O'Brien inauthor:Sons";
let found = search_with("?!", Some("Someone"), 5, recorded(&mut asked, &bodies)) let found = search_with("?!", Some("Someone"), 5, recorded(&mut asked, &bodies))
.expect("an answer"); .expect("an answer");
assert!(found.is_empty()); assert!(found.is_empty());
assert_eq!(asked.len(), 1); // By title and as free text; never Google.
assert_eq!(asked.len(), 2, "{asked:?}");
assert!(asked.iter().all(|url| url.contains("openlibrary.org")));
} }
#[test] #[test]
@@ -744,13 +956,25 @@ intitle:y intitle:z inauthor:O'Brien inauthor:Sons";
let found = let found =
search_with("The Google Story", None, 1, recorded(&mut asked, &bodies)).expect("ok"); search_with("The Google Story", None, 1, recorded(&mut asked, &bodies)).expect("ok");
assert_eq!(found.len(), 1); assert_eq!(found.len(), 1);
assert!(asked[0].ends_with("&limit=1"), "{}", asked[0]); let google = asked.last().expect("Google was asked");
assert!(asked[1].ends_with("&maxResults=1"), "{}", asked[1]); assert!(
asked[..asked.len() - 1]
.iter()
.all(|url| url.ends_with("&limit=1")),
"{asked:?}"
);
assert!(google.ends_with("&maxResults=1"), "{google}");
let mut asked = Vec::new(); let mut asked = Vec::new();
search_with("The Google Story", None, 500, recorded(&mut asked, &bodies)).expect("ok"); search_with("The Google Story", None, 500, recorded(&mut asked, &bodies)).expect("ok");
assert!(asked[0].ends_with("&limit=40"), "{}", asked[0]); let google = asked.last().expect("Google was asked");
assert!(asked[1].ends_with("&maxResults=40"), "{}", asked[1]); assert!(
asked[..asked.len() - 1]
.iter()
.all(|url| url.ends_with("&limit=40")),
"{asked:?}"
);
assert!(google.ends_with("&maxResults=40"), "{google}");
let mut asked = Vec::new(); let mut asked = Vec::new();
let found = let found =
crates/bibfetch/src/source.rs
+119 -33
View File
@@ -17,9 +17,10 @@ pub enum Source {
Arxiv, Arxiv,
/// OpenLibrary, for books by ISBN. /// OpenLibrary, for books by ISBN.
OpenLibrary, OpenLibrary,
/// OpenLibrary's index, rather than its edition record. Asked when the /// OpenLibrary's index, rather than its edition record. Asked for what is
/// edition record has the book but not who wrote it, which is often: an /// true of the work — the title, the authors by name, the subjects —
/// edition names its authors by key, and the index names them. /// because an edition names its authors by key and the index names them.
/// Never for what differs between editions.
OpenLibrarySearch, OpenLibrarySearch,
/// Google Books, for books by ISBN. Asked last, because it answers /// Google Books, for books by ISBN. Asked last, because it answers
/// anonymous callers only until a daily quota runs out. /// anonymous callers only until a daily quota runs out.
@@ -55,14 +56,14 @@ impl Source {
match kind { match kind {
"doi" => &[Source::Crossref], "doi" => &[Source::Crossref],
"arxiv" => &[Source::Arxiv], "arxiv" => &[Source::Arxiv],
// The index first. Asked for the fields by name it answers with // The index first, for what is true of the work: the title, the
// the title, the authors, the publisher, the language, a page // authors by name, the subjects. Then the edition record, for
// count and the subjects in one request, which is the whole of // what is true of this edition only: the publisher, the language,
// what a catalogue wants; the edition record is then needed only // the year and the length. The index answers about the work, and
// for a subtitle or a publisher the index did not give. Asking it // its publishers, languages and years are every edition's at
// second cost three requests per book where one usually does, and // once, so they are not asked of it; the lookup goes on to the
// at three books in flight that is the difference between two // edition record for them. Google Books last, for whatever the
// days and two hours. // edition record does not say.
"isbn" => &[ "isbn" => &[
Source::OpenLibrarySearch, Source::OpenLibrarySearch,
Source::OpenLibrary, Source::OpenLibrary,
@@ -83,15 +84,16 @@ impl Source {
Source::OpenLibrary => { Source::OpenLibrary => {
format!("https://openlibrary.org/isbn/{}.json", encode(id)) format!("https://openlibrary.org/isbn/{}.json", encode(id))
} }
// `fields` or the index answers with a handful of defaults and // Only what is true of the work. Not `publisher`, `language`,
// keeps the rest, which is how a language it holds went unread. // `first_publish_year` or `number_of_pages_median`: those are every
// Not `cover_i`: that is the cover chosen for the whole work, and // edition's at once, and the first of each is usually another
// on a work with hundreds of editions it is usually another // edition's (the Penguin *War and Peace* came back Spanish, from
// edition's, in another translation or language. // W.W. Norton, in 1845). Not `cover_i`: that is the cover chosen
// for the whole work, often another edition's in another
// translation or language.
Source::OpenLibrarySearch => format!( Source::OpenLibrarySearch => format!(
"https://openlibrary.org/search.json?q=isbn:{}\ "https://openlibrary.org/search.json?q=isbn:{}\
&fields=title,subtitle,author_name,first_publish_year,publisher,language,subject,\ &fields=title,subtitle,author_name,subject&limit=1",
number_of_pages_median&limit=1",
encode(id) encode(id)
), ),
Source::GoogleBooks => format!( Source::GoogleBooks => format!(
@@ -295,6 +297,8 @@ struct SearchReply {
docs: Vec<SearchDoc>, docs: Vec<SearchDoc>,
} }
/// What the index says about a work. Only the fields true of every edition
/// are read, so an answer that carries the others anyway cannot leak them.
#[derive(Deserialize)] #[derive(Deserialize)]
struct SearchDoc { struct SearchDoc {
#[serde(default)] #[serde(default)]
@@ -304,15 +308,7 @@ struct SearchDoc {
#[serde(default)] #[serde(default)]
author_name: Vec<String>, author_name: Vec<String>,
#[serde(default)] #[serde(default)]
first_publish_year: Option<i64>,
#[serde(default)]
publisher: Vec<String>,
#[serde(default)]
language: Vec<String>,
#[serde(default)]
subject: Vec<String>, subject: Vec<String>,
#[serde(rename = "number_of_pages_median", default)]
pages: Option<u32>,
} }
/// How many of a work's subject headings are worth carrying. /// How many of a work's subject headings are worth carrying.
@@ -324,6 +320,11 @@ pub(crate) const SUBJECT_LIMIT: usize = 12;
/// A search that matched nothing answers with no documents rather than a /// A search that matched nothing answers with no documents rather than a
/// status, so an empty reply is the absence and not a failure. /// status, so an empty reply is the absence and not a failure.
///
/// The record says what is true of the work and nothing that belongs to one
/// edition: no publisher, language, year or page count. Those come from the
/// edition record, which [`crate::ask::lookup`] asks next because a record
/// without them is never complete.
fn open_library_search(id: &str, kind: &str, body: &str) -> Result<Record, Error> { fn open_library_search(id: &str, kind: &str, body: &str) -> Result<Record, Error> {
let reply: SearchReply = serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?; let reply: SearchReply = serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?;
let Some(doc) = reply.docs.into_iter().next() else { let Some(doc) = reply.docs.into_iter().next() else {
@@ -336,14 +337,10 @@ fn open_library_search(id: &str, kind: &str, body: &str) -> Result<Record, Error
(title, _) => title, (title, _) => title,
}; };
record.authors = doc.author_name; record.authors = doc.author_name;
record.year = doc.first_publish_year;
record.container = doc.publisher.into_iter().next();
record.language = doc.language.into_iter().next();
// The index lists every subject any edition of the work carries, which // The index lists every subject any edition of the work carries, which
// runs to hundreds on a well-known book. A shelf needs a handful. // runs to hundreds on a well-known book. A shelf needs a handful.
record.subjects = doc.subject; record.subjects = doc.subject;
record.subjects.truncate(SUBJECT_LIMIT); record.subjects.truncate(SUBJECT_LIMIT);
record.pages = doc.pages;
Ok(record) Ok(record)
} }
@@ -475,9 +472,8 @@ mod tests {
fn each_kind_goes_to_the_body_that_assigns_it() { fn each_kind_goes_to_the_body_that_assigns_it() {
assert_eq!(Source::for_kind("doi"), Some(Source::Crossref)); assert_eq!(Source::for_kind("doi"), Some(Source::Crossref));
assert_eq!(Source::for_kind("arxiv"), Some(Source::Arxiv)); assert_eq!(Source::for_kind("arxiv"), Some(Source::Arxiv));
// The index, not the edition record: it answers with the whole of // The index first, for the authors by name; the edition record names
// what a catalogue wants in one request, where the edition record // them by key, and is asked next for what only the edition knows.
// takes up to twenty-eight seconds and names its authors by key.
assert_eq!(Source::for_kind("isbn"), Some(Source::OpenLibrarySearch)); assert_eq!(Source::for_kind("isbn"), Some(Source::OpenLibrarySearch));
// No register answers about an ISSN without a key. // No register answers about an ISSN without a key.
assert_eq!(Source::for_kind("issn"), None); assert_eq!(Source::for_kind("issn"), None);
@@ -599,6 +595,96 @@ mod tests {
assert!(!Source::OpenLibrarySearch.url("1").contains("cover_i")); assert!(!Source::OpenLibrarySearch.url("1").contains("cover_i"));
} }
/// What the index said for ISBN 9780140447934, the Penguin Classics *War
/// and Peace*, on 2026-11-13, asked for the fields the lookup asked for
/// before #6: the publishers, languages and first year of every edition
/// of the work.
const WAR_AND_PEACE_INDEX: &str = r#"{"docs":[{"title":"War and Peace",
"author_name":["Leo Tolstoy"],"first_publish_year":1845,
"publisher":["W.W. Norton","IndyPublish.com","Canterbury Classics"],
"language":["spa","gre","dut","eng"],"subject":["Fiction"],
"number_of_pages_median":1296}]}"#;
/// The edition record for the same ISBN, `/isbn/9780140447934.json`, with
/// the English language its `languages` field should name.
const WAR_AND_PEACE_EDITION: &str = r#"{"title":"War and Peace (Penguin Classics)",
"publishers":["Penguin Books"],"publish_date":"September 27, 2007",
"languages":[{"key":"/languages/eng"}],"number_of_pages":1440}"#;
/// Index first, then the edition record, merged as `ask::lookup` merges.
fn looked_up(edition: &str) -> Record {
let mut kept =
open_library_search("9780140447934", "isbn", WAR_AND_PEACE_INDEX).expect("a record");
kept.fill_from(open_library("9780140447934", "isbn", edition).expect("a record"));
kept
}
#[test]
fn the_index_says_nothing_only_an_edition_can() {
let record =
open_library_search("9780140447934", "isbn", WAR_AND_PEACE_INDEX).expect("a record");
// What is true of the work is kept.
assert_eq!(record.title.as_deref(), Some("War and Peace"));
assert_eq!(record.authors, ["Leo Tolstoy"]);
// What belongs to one edition is not taken from all of them.
assert_eq!(record.language, None);
assert_eq!(record.container, None);
assert_eq!(record.year, None);
assert_eq!(record.pages, None);
}
#[test]
fn a_lookup_by_isbn_gives_the_editions_language_publisher_and_year() {
let record = looked_up(WAR_AND_PEACE_EDITION);
assert_eq!(record.language.as_deref(), Some("eng"));
assert_eq!(record.container.as_deref(), Some("Penguin Books"));
assert_eq!(record.year, Some(2007));
assert_eq!(record.pages, Some(1440));
assert_eq!(record.authors, ["Leo Tolstoy"]);
}
#[test]
fn what_the_edition_does_not_say_is_missing_rather_than_the_works() {
// The real edition record today names no language at all.
let record = looked_up(
r#"{"title":"War and Peace (Penguin Classics)","publishers":["Penguin Books"],
"publish_date":"September 27, 2007","number_of_pages":1440}"#,
);
assert_eq!(record.language, None);
assert_eq!(record.container.as_deref(), Some("Penguin Books"));
assert_eq!(record.year, Some(2007));
// Nor a date: the work's first year is not this edition's.
let record = looked_up(r#"{"title":"War and Peace (Penguin Classics)"}"#);
assert_eq!(record.year, None);
assert_eq!(record.container, None);
assert_eq!(record.pages, None);
}
#[test]
fn the_index_alone_never_ends_a_lookup() {
// `lookup` stops at the first complete record; the index must not be
// one, or the edition record is never asked.
let record =
open_library_search("9780140447934", "isbn", WAR_AND_PEACE_INDEX).expect("a record");
assert!(!record.complete());
}
#[test]
fn the_index_is_not_asked_for_what_its_editions_differ_in() {
let url = Source::OpenLibrarySearch.url("9780140447934");
for field in [
"publisher",
"language",
"first_publish_year",
"number_of_pages_median",
] {
assert!(
!url.split(['=', ',', '&']).any(|f| f == field),
"{field} in {url}"
);
}
}
#[test] #[test]
fn an_answer_that_is_not_what_it_should_be_says_so() { fn an_answer_that_is_not_what_it_should_be_says_so() {
assert!(matches!( assert!(matches!(
crates/bindings/python/catena/__init__.py
+6
View File
@@ -21,6 +21,8 @@ from ._catena import (
compose_subjects, compose_subjects,
count_tokens, count_tokens,
cue_windows, cue_windows,
Limited,
fetch_cover,
fetch_identifier, fetch_identifier,
find_citations, find_citations,
fuse_hits, fuse_hits,
@@ -41,6 +43,7 @@ from ._catena import (
record_severity, record_severity,
same_work, same_work,
scrub, scrub,
search_title,
segment_cues, segment_cues,
series_of, series_of,
shelve, shelve,
@@ -53,6 +56,7 @@ from ._catena import (
__all__ = [ __all__ = [
"__version__", "__version__",
"Limited",
"canonical_subject", "canonical_subject",
"chunk_markdown", "chunk_markdown",
"citation_prompt", "citation_prompt",
@@ -62,6 +66,7 @@ __all__ = [
"compose_subjects", "compose_subjects",
"count_tokens", "count_tokens",
"cue_windows", "cue_windows",
"fetch_cover",
"fetch_identifier", "fetch_identifier",
"find_citations", "find_citations",
"fuse_hits", "fuse_hits",
@@ -82,6 +87,7 @@ __all__ = [
"record_severity", "record_severity",
"same_work", "same_work",
"scrub", "scrub",
"search_title",
"segment_cues", "segment_cues",
"series_of", "series_of",
"shelve", "shelve",
crates/bindings/python/catena/__init__.pyi
+34
View File
@@ -103,6 +103,40 @@ def fetch_identifier(
""" """
... ...
class Limited(Exception):
"""A register refused because it has been asked too often; ask again later."""
def fetch_cover(
isbn: str,
cover_id: int | None = ...,
size: str = ...,
polite: str | None = ...,
timeout: int = ...,
deadline: int = ...,
) -> dict[str, Any] | None:
"""The cover Open Library has for a book, by ISBN, then by cover id.
A dict with `bytes`, `media_type`, `extension`, `source` and `url`, or
`None` when there is none. Raises `Limited` when Open Library refused for
being asked too often. Blocks on the network.
"""
...
def search_title(
title: str,
author: str | None = ...,
limit: int = ...,
polite: str | None = ...,
timeout: int = ...,
attempts: int = ...,
) -> list[dict[str, Any]]:
"""The editions the registers list for a title and author, in their order.
Records shaped like `fetch_identifier`'s. Use `same_work` to decide which
one is the book. Blocks on the network.
"""
...
def colophon_prompt( def colophon_prompt(
record: dict[str, Any], text: str, pages: int = ... record: dict[str, Any], text: str, pages: int = ...
) -> str | None: ) -> str | None:
crates/bindings/src/python.rs
+104 -2
View File
@@ -10,7 +10,7 @@
use pyo3::exceptions::PyValueError; use pyo3::exceptions::PyValueError;
use pyo3::prelude::*; use pyo3::prelude::*;
use pyo3::types::{PyDict, PyList}; use pyo3::types::{PyBytes, PyDict, PyList};
use crate::{ChunkBudgets, CueBudgets}; use crate::{ChunkBudgets, CueBudgets};
@@ -434,6 +434,9 @@ fn _catena(module: &Bound<'_, PyModule>) -> PyResult<()> {
module.add_function(wrap_pyfunction!(read_colophon, module)?)?; module.add_function(wrap_pyfunction!(read_colophon, module)?)?;
module.add_function(wrap_pyfunction!(shelve, module)?)?; module.add_function(wrap_pyfunction!(shelve, module)?)?;
module.add_function(wrap_pyfunction!(fetch_identifier, module)?)?; module.add_function(wrap_pyfunction!(fetch_identifier, module)?)?;
module.add_function(wrap_pyfunction!(fetch_cover, module)?)?;
module.add_function(wrap_pyfunction!(search_title, module)?)?;
module.add("Limited", module.py().get_type::<Limited>())?;
module.add_function(wrap_pyfunction!(colophon_prompt, module)?)?; module.add_function(wrap_pyfunction!(colophon_prompt, module)?)?;
module.add_function(wrap_pyfunction!(merge_colophon, module)?)?; module.add_function(wrap_pyfunction!(merge_colophon, module)?)?;
Ok(()) Ok(())
@@ -512,13 +515,112 @@ fn fetch_identifier<'py>(
// would stop every other thread in the interpreter for the whole request. // would stop every other thread in the interpreter for the whole request.
let found = py let found = py
.detach(|| bibfetch::ask::lookup(&agent, id, kind, ask.attempts)) .detach(|| bibfetch::ask::lookup(&agent, id, kind, ask.attempts))
.map_err(|e| PyValueError::new_err(e.to_string()))?; .map_err(register_error)?;
match found { match found {
Some(record) => Ok(Some(to_py(py, &record)?)), Some(record) => Ok(Some(to_py(py, &record)?)),
None => Ok(None), None => Ok(None),
} }
} }
pyo3::create_exception!(
_catena,
Limited,
pyo3::exceptions::PyException,
"A register refused because it has been asked too often; ask again later."
);
/// A register's error as the Python exception that says what to do about it.
fn register_error(error: bibfetch::Error) -> PyErr {
match error {
bibfetch::Error::Limited(why) => Limited::new_err(why),
other => PyValueError::new_err(other.to_string()),
}
}
/// The cover Open Library has for a book, by its ISBN, then by the cover id
/// a record named.
///
/// Returns a dict with `bytes`, `media_type` (`image/jpeg` or `image/png`),
/// `extension`, `source` and `url`, or `None` when there is no cover. Covers
/// asked by ISBN are kept three seconds apart across the process, and the
/// whole fetch stops after `deadline` seconds. Raises `catena.Limited` when
/// Open Library refused for being asked too often, and `ValueError` when the
/// ISBN is not one or the store could not be reached. Blocks on the network.
#[pyfunction]
#[pyo3(signature = (isbn, cover_id = None, size = "large", polite = None, timeout = 20, deadline = 120))]
fn fetch_cover<'py>(
py: Python<'py>,
isbn: &str,
cover_id: Option<u64>,
size: &str,
polite: Option<String>,
timeout: u64,
deadline: u64,
) -> PyResult<Option<Bound<'py, PyDict>>> {
let size = match size {
"small" => bibfetch::CoverSize::Small,
"medium" => bibfetch::CoverSize::Medium,
"large" => bibfetch::CoverSize::Large,
other => {
return Err(PyValueError::new_err(format!(
"size is small, medium or large, not {other:?}"
)));
}
};
let agent = bibfetch::ask::Ask {
polite,
timeout: std::time::Duration::from_secs(timeout.max(1)),
..Default::default()
}
.agent();
let covers = bibfetch::cover::Covers {
deadline: std::time::Duration::from_secs(deadline.max(1)),
..Default::default()
};
let found = py
.detach(|| covers.fetch(&agent, isbn, cover_id, size))
.map_err(register_error)?;
let Some(cover) = found else {
return Ok(None);
};
let out = PyDict::new(py);
out.set_item("bytes", PyBytes::new(py, &cover.bytes))?;
out.set_item("media_type", cover.media_type)?;
out.set_item("extension", cover.extension())?;
out.set_item("source", cover.source)?;
out.set_item("url", cover.url)?;
Ok(Some(out))
}
/// The editions the registers list for a title, and an author if one is known.
///
/// Open Library is asked first and Google Books only when it has nothing.
/// The candidates come back in the register's own order, as records shaped
/// like `fetch_identifier`'s; nothing here decides which is the book, which is
/// what `same_work` is for. Blocks on the network.
#[pyfunction]
#[pyo3(signature = (title, author = None, limit = 10, polite = None, timeout = 20, attempts = 3))]
fn search_title<'py>(
py: Python<'py>,
title: &str,
author: Option<&str>,
limit: usize,
polite: Option<String>,
timeout: u64,
attempts: u32,
) -> PyResult<Bound<'py, PyList>> {
let ask = bibfetch::ask::Ask {
polite,
timeout: std::time::Duration::from_secs(timeout.max(1)),
attempts: attempts.max(1),
};
let agent = ask.agent();
let found = py
.detach(|| bibfetch::search(&agent, title, author, limit, ask.attempts))
.map_err(register_error)?;
list_to_py(py, &found)
}
/// The question to put to a model about the fields a book's pages did not /// The question to put to a model about the fields a book's pages did not
/// settle, or `None` when they settled everything. /// settle, or `None` when they settled everything.
/// ///
crates/bookmeta/README.md
+53 -6
View File
@@ -99,8 +99,10 @@ A Kindle book's title is EXTH 503 where it is there, then the MOBI header's
full name, then the Palm database name (cut to 31 bytes, with a note). Authors full name, then the Palm database name (cut to 31 bytes, with a note). Authors
(100) and subjects (105) are every record of their kind, in order; publisher (100) and subjects (105) are every record of their kind, in order; publisher
(101), description (103), date (106) and language (524) the first. EXTH 104 is (101), description (103), date (106) and language (524) the first. EXTH 104 is
an `isbn` when it is shaped like one and `other` when not, and 113 is an an `isbn` when it is shaped like one and `other` when not. 113 is filed by
`asin`. A contributor (108) is not an author, the same as an EPUB's its shape: a `uuid` when it is one (calibre writes its book id there), an
`asin` when it is ten letters and digits, and `other` when it is neither. A
contributor (108) is not an author, the same as an EPUB's
`dc:contributor`, and is not reported. With no EXTH language, the language `dc:contributor`, and is not reported. With no EXTH language, the language
comes from the header's locale. Strings are decoded as the header says, comes from the header's locale. Strings are decoded as the header says,
UTF-8 or Windows-1252. A book whose text is encrypted still has its header UTF-8 or Windows-1252. A book whose text is encrypted still has its header
@@ -112,10 +114,55 @@ an EXTH block that breaks off keeps the records before the break.
## Covers ## Covers
`Metadata::cover` is the cover image a Kindle book carries (EXTH 201, or the `Metadata::cover` is the cover image the file carries, with its media type
thumbnail, 202, when no cover is named), with its media type read off its read off its first bytes: JPEG, PNG, GIF, WebP or BMP. It is for library
first bytes. It is for library callers and is not part of the JSON record, callers and is not part of the JSON record, which is a line of text.
which is a line of text.
A Kindle book's cover is EXTH 201, or the thumbnail, 202, when no cover is
named. With neither, it is the record at the header's first image index, as
calibre does, and only that record: when it is not a JPEG, PNG or GIF there is
no cover, because the records after it can be illustrations, and kindlegen
sometimes puts a font there first.
An EPUB's is the first of these that reaches an image:
1. The manifest item whose `properties` include `cover-image` (EPUB 3).
2. The item `<meta name="cover" content="…">` names (EPUB 2). Some
converters write the image's address there rather than its id, and that is
followed too.
3. The guide's `<reference type="cover">`, a page, and the `<img src>` or
SVG `<image href>` on it. Where the page has several, the one whose file
name, `alt`, `id` or `class` says `cover` is taken; where none does, the
first is, with a note that there were others. A guide that points straight
at an image is taken at its word.
4. Any image in the manifest whose id or file name says `cover`, one named
`cover` outright first and the rest in manifest order.
A name says `cover` when it has it as a word of its own, in any case. Words
break at anything not a letter or digit and where a lower case letter meets a
capital, so `CoverImage`, `Book_COVER` and `cover-thumb` say it, and
`discover-ch1` and `recovery-map` do not.
A route that names something unusable — not in the archive, not an image —
leaves a note, and the next route is asked. Addresses are resolved against the
document they are written in, with their fragment off and their
percent-escapes undone, and one that climbs out of the archive or names a
scheme (`http:`, `data:`) is not followed.
What the bytes say beats what the manifest says: a JPEG listed as
`image/png` comes back as `image/jpeg`, with a note. A cover larger than
`bookmeta::MAX_COVER` (16 MiB) is left out with a note, in either format, and
an EPUB's is never decompressed further than that, whatever size the archive
claims for it. Nor are more than eight members read looking for one,
however many candidates the manifest lists: the search stops there, with a
note.
`Cover::guessed` says the file left the choice open: a Kindle book's first
image record, or an EPUB image picked because its name says `cover`. Either
can be an illustration from inside the book, so a caller that can get a cover
another way, from a register by ISBN, should prefer that one. A cover the file
names is never a guess. `Cover::extension` gives the usual file extension for
its media type.
## Asking a model ## Asking a model
crates/bookmeta/src/epub.rs
+992 -1
View File
File diff suppressed because it is too large. Load diff
crates/bookmeta/src/lib.rs
+55 -2
View File
@@ -106,10 +106,63 @@ pub struct Metadata {
#[derive(Debug, Clone, PartialEq, Eq)] #[derive(Debug, Clone, PartialEq, Eq)]
pub struct Cover { pub struct Cover {
/// What kind of image it is, read off its first bytes: `image/jpeg`, /// What kind of image it is, read off its first bytes: `image/jpeg`,
/// `image/png`, `image/gif` or `image/bmp`. /// `image/png`, `image/gif`, `image/webp` or `image/bmp`.
pub media_type: String, pub media_type: String,
/// The image itself. /// The image itself, never more than [`MAX_COVER`] bytes.
pub bytes: Vec<u8>, pub bytes: Vec<u8>,
/// Whether the file left the choice open. A MOBI that names no cover, or
/// an EPUB whose only cover is an image with "cover" in its name, has one
/// picked by position or by name, and that pick can be an illustration
/// from inside the book. A caller with a better source should prefer it.
pub guessed: bool,
}
impl Cover {
/// The usual file extension for the image, without the dot.
pub fn extension(&self) -> &'static str {
match self.media_type.as_str() {
"image/png" => "png",
"image/gif" => "gif",
"image/webp" => "webp",
"image/bmp" => "bmp",
_ => "jpg",
}
}
}
/// How large a cover is worth keeping.
///
/// A real one is a few hundred kilobytes. An EPUB's cover has to be
/// decompressed to be read, and this is as far as that goes: a larger one is
/// left out with a note, in every format, so a hostile file cannot make a
/// reader allocate whatever it likes.
pub const MAX_COVER: usize = 16 * 1024 * 1024;
/// A size, as a note says it.
pub(crate) fn size(bytes: usize) -> String {
const MIB: usize = 1024 * 1024;
if bytes >= MIB && bytes % MIB == 0 {
format!("{} MiB", bytes / MIB)
} else {
format!("{bytes} bytes")
}
}
/// What kind of image these bytes are, by their magic.
pub(crate) fn image_type(data: &[u8]) -> Option<&'static str> {
if data.starts_with(&[0xff, 0xd8, 0xff]) {
Some("image/jpeg")
} else if data.starts_with(b"\x89PNG\r\n\x1a\n") {
Some("image/png")
} else if data.starts_with(b"GIF87a") || data.starts_with(b"GIF89a") {
Some("image/gif")
} else if data.starts_with(b"RIFF") && data.get(8..12) == Some(b"WEBP") {
Some("image/webp")
} else if data.starts_with(b"BM") && data.len() > 14 {
Some("image/bmp")
} else {
None
}
} }
fn version() -> u32 { fn version() -> u32 {
crates/bookmeta/src/mobi.rs
+13 -16
View File
@@ -13,7 +13,7 @@
//! panics or reads past the end. //! panics or reads past the end.
use crate::epub::{digits_of, is_isbn, is_uuid}; use crate::epub::{digits_of, is_isbn, is_uuid};
use crate::{Cover, Metadata, Source}; use crate::{Cover, MAX_COVER, Metadata, Source, image_type};
/// Where a Palm database says what it holds. /// Where a Palm database says what it holds.
const TYPE_AT: usize = 60; const TYPE_AT: usize = 60;
@@ -332,9 +332,17 @@ fn cover(
meta.note("the cover record is not an image"); meta.note("the cover record is not an image");
return None; return None;
}; };
if data.len() > MAX_COVER {
meta.note(format!(
"the cover record is larger than {}",
crate::size(MAX_COVER)
));
return None;
}
Some(Cover { Some(Cover {
media_type: media_type.to_string(), media_type: media_type.to_string(),
bytes: data.to_vec(), bytes: data.to_vec(),
guessed: false,
}) })
} }
@@ -347,10 +355,14 @@ fn cover(
fn first_image_record(bytes: &[u8], records: &[usize], first_image: Option<u32>) -> Option<Cover> { fn first_image_record(bytes: &[u8], records: &[usize], first_image: Option<u32>) -> Option<Cover> {
let first = first_image.filter(|&index| index != NONE)?; let first = first_image.filter(|&index| index != NONE)?;
let data = record(bytes, records, first as usize)?; let data = record(bytes, records, first as usize)?;
if data.len() > MAX_COVER {
return None;
}
let media_type = image_type(data).filter(|kind| *kind != "image/bmp")?; let media_type = image_type(data).filter(|kind| *kind != "image/bmp")?;
Some(Cover { Some(Cover {
media_type: media_type.to_string(), media_type: media_type.to_string(),
bytes: data.to_vec(), bytes: data.to_vec(),
guessed: true,
}) })
} }
@@ -359,21 +371,6 @@ fn is_asin(value: &str) -> bool {
value.len() == 10 && value.chars().all(|c| c.is_ascii_alphanumeric()) value.len() == 10 && value.chars().all(|c| c.is_ascii_alphanumeric())
} }
/// What kind of image these bytes are, by their magic.
fn image_type(data: &[u8]) -> Option<&'static str> {
if data.starts_with(&[0xff, 0xd8, 0xff]) {
Some("image/jpeg")
} else if data.starts_with(b"\x89PNG\r\n\x1a\n") {
Some("image/png")
} else if data.starts_with(b"GIF87a") || data.starts_with(b"GIF89a") {
Some("image/gif")
} else if data.starts_with(b"BM") && data.len() > 14 {
Some("image/bmp")
} else {
None
}
}
/// The Palm database's name, the last place a title can come from. /// The Palm database's name, the last place a title can come from.
/// ///
/// Converters write the title there with spaces made underscores; it is /// Converters write the title there with spaces made underscores; it is
crates/bookmeta/tests/mobi.rs
+2
View File
@@ -31,6 +31,7 @@ fn says_what_it_was_given(meta: &bookmeta::Metadata) {
let cover = meta.cover.as_ref().expect("a cover"); let cover = meta.cover.as_ref().expect("a cover");
assert_eq!(cover.media_type, "image/jpeg"); assert_eq!(cover.media_type, "image/jpeg");
assert!(!cover.guessed, "EXTH 201 names the cover");
let original = std::fs::read(fixture("lantern-cover.jpg")).expect("the cover fixture"); let original = std::fs::read(fixture("lantern-cover.jpg")).expect("the cover fixture");
assert_eq!(cover.bytes, original); assert_eq!(cover.bytes, original);
} }
@@ -84,6 +85,7 @@ fn a_file_naming_no_cover_gets_its_first_image_record() {
let meta = bookmeta::from_bytes("x.mobi", &lantern_naming_no_cover()); let meta = bookmeta::from_bytes("x.mobi", &lantern_naming_no_cover());
let cover = meta.cover.expect("the first image record"); let cover = meta.cover.expect("the first image record");
assert_eq!(cover.media_type, "image/jpeg"); assert_eq!(cover.media_type, "image/jpeg");
assert!(cover.guessed, "a cover taken by position is a guess");
let original = std::fs::read(fixture("lantern-cover.jpg")).expect("the cover fixture"); let original = std::fs::read(fixture("lantern-cover.jpg")).expect("the cover fixture");
assert_eq!(cover.bytes, original); assert_eq!(cover.bytes, original);
} }
crates/booksniff/README.md
+30 -2
View File
@@ -26,8 +26,8 @@ file too small to be a book, a name that says "sample".
cover images that ended up in a folder of PDFs. cover images that ended up in a folder of PDFs.
A library accumulates files that lie: a DjVu named `.pdf`, a Word document A library accumulates files that lie: a DjVu named `.pdf`, a Word document
named `.epub`, an EPUB saved as `.zip`, a single PDF that somebody zipped and named `.epub`, an RTF saved as `.doc`, an EPUB saved as `.zip`, a single PDF
renamed back to `.pdf`. that somebody zipped and renamed back to `.pdf`.
## Install ## Install
@@ -89,7 +89,12 @@ one bad file does not stop a sweep.
| `ole2` | The compound-file header every binary Word document starts with | | `ole2` | The compound-file header every binary Word document starts with |
| `epub` | A zip holding `mimetype` and `META-INF/container.xml` | | `epub` | A zip holding `mimetype` and `META-INF/container.xml` |
| `zip-wrapped-pdf` | A zip whose only real member is one PDF | | `zip-wrapped-pdf` | A zip whose only real member is one PDF |
| `zip-wrapped-fb2` | A zip whose only real member is named `.fb2` — the `.fb2.zip` most FictionBooks are passed around as |
| `cbz` | A zip whose every real member is a page image (JPEG, PNG, GIF or WebP), and which carries a `ComicInfo.xml` or is itself named `.cbz` or `.cbr` |
| `azw3`, `mobi` | A Palm database tagged `BOOKMOBI`, told apart by the format version inside the first record | | `azw3`, `mobi` | A Palm database tagged `BOOKMOBI`, told apart by the format version inside the first record |
| `lit` | `ITOLITLS`, Microsoft Reader's signature |
| `rtf` | `{\rtf` at the front, after any byte-order mark and blank space, followed by the version digit, a space or the next control word |
| `fb2` | XML whose root element is `FictionBook`, found by stepping over the declaration, comments and doctype in the first four kilobytes; UTF-16 with a byte-order mark is decoded first |
| `html` | Markup at the front, after any byte-order mark | | `html` | Markup at the front, after any byte-order mark |
| `txt` | Text that is none of the above | | `txt` | Text that is none of the above |
| `unknown` | Anything else, with a note saying what was seen | | `unknown` | Anything else, with a note saying what was seen |
@@ -98,6 +103,29 @@ An archive holding several PDFs is a collection, not a wrapped book: choosing
among them would be a guess, and naming one chapter as the whole work is worse among them would be a guess, and naming one chapter as the whole work is worse
than refusing. than refusing.
A zipped FictionBook is reported as wrapped rather than as `fb2`, the same way
a zipped PDF is: a converter handed it as FictionBook is handed a zip, and the
kind is there to stop exactly that.
A comic is the pages and nothing else. Directories, the `ComicInfo.xml` a
comic reader writes into the archive, and the desktop clutter every archive
picks up are passed over; anything more — one PDF among the pages, a readme —
and the zip is not a comic, and says what it holds instead.
A zip of pictures is as often a holiday as a comic, so images alone are not
enough. It is a comic when it carries a `ComicInfo.xml`, or when its own name
ends `.cbz` or `.cbr`. Otherwise it is `unknown`, with a note saying it is a
zip of so many images and would be a comic book if its name said `.cbz`.
Members are judged by name. Only the archive's directory is read, so no
member is ever unpacked and a page called `001.jpg` is taken to be an image
without its bytes being looked at. The same goes for a wrapped FictionBook:
the one member is taken to be a FictionBook because its name ends `.fb2`.
FictionBook is read as bytes rather than as text. A great many are written in
windows-1251, which spells its markup in ASCII but is not UTF-8, and would
otherwise be called binary.
## As a library ## As a library
```rust ```rust
crates/booksniff/src/fb2.rs
+261
View File
@@ -0,0 +1,261 @@
//! Telling a FictionBook from any other XML.
//!
//! A FictionBook is an XML document whose root element is `FictionBook`.
//! The root comes after the prolog — a declaration, comments, perhaps a
//! doctype — so it is found by stepping over those rather than by matching
//! at a fixed offset.
//!
//! The walk is over bytes, not text. A great many FictionBooks are written in
//! windows-1251, which is not UTF-8 but spells its markup in ASCII all the
//! same, and a reader that insisted on UTF-8 would call every one of them
//! binary.
//!
//! A FictionBook saved as UTF-16 is the one case where the markup is not
//! ASCII bytes. Its byte-order mark says so, and the front is decoded before
//! the same walk is made over it.
use crate::BOM;
/// The UTF-16 byte-order mark, little-endian.
const UTF16_LE: &[u8] = b"\xff\xfe";
/// The UTF-16 byte-order mark, big-endian.
const UTF16_BE: &[u8] = b"\xfe\xff";
/// Most UTF-16 code units decoded from the front: the same window, in
/// characters, as the bytes read for every other encoding.
const UTF16_UNITS: usize = 2048;
/// What the root element of a FictionBook is called.
const ROOT: &[u8] = b"FictionBook";
/// Whether the front of a file is the start of a FictionBook.
///
/// `head` is however much of the front the caller read. A root element that
/// does not begin inside it is not found, so the answer is no rather than a
/// guess.
pub fn is_fictionbook(head: &[u8]) -> bool {
match utf16(head) {
Some(decoded) => is_fictionbook_root(decoded.as_bytes()),
None => is_fictionbook_root(head),
}
}
fn is_fictionbook_root(head: &[u8]) -> bool {
root_element(head).is_some_and(|name| local_name(name) == ROOT)
}
/// The front of a UTF-16 file as UTF-8, if a byte-order mark says it is one.
///
/// Only a bounded prefix is decoded. A half character at the end, or a
/// surrogate without its pair, becomes a replacement character, which the
/// walk takes as part of a name and so never as the end of one.
fn utf16(head: &[u8]) -> Option<String> {
let (body, big_endian) = match head.strip_prefix(UTF16_LE) {
Some(body) => (body, false),
None => (head.strip_prefix(UTF16_BE)?, true),
};
let units = body.chunks_exact(2).take(UTF16_UNITS).map(|pair| {
let pair = [pair[0], pair[1]];
if big_endian {
u16::from_be_bytes(pair)
} else {
u16::from_le_bytes(pair)
}
});
Some(
char::decode_utf16(units)
.map(|c| c.unwrap_or(char::REPLACEMENT_CHARACTER))
.collect(),
)
}
/// The name of the first element, after the prolog.
fn root_element(head: &[u8]) -> Option<&[u8]> {
let mut rest = head.strip_prefix(BOM).unwrap_or(head);
loop {
rest = rest.trim_ascii_start();
if let Some(after) = rest.strip_prefix(b"<?") {
rest = after_marker(after, b"?>")?;
} else if let Some(after) = rest.strip_prefix(b"<!--") {
rest = after_marker(after, b"-->")?;
} else if let Some(after) = rest.strip_prefix(b"<!") {
rest = after_declaration(after)?;
} else {
// Anything but an element here is text before the root, which
// XML does not allow.
let after = rest.strip_prefix(b"<")?;
// A name that runs to the end of what was read may be the front
// of a longer one, so it is not taken.
let end = after.iter().position(|byte| !is_name_byte(*byte))?;
let ends_properly =
matches!(after[end], b'>' | b'/') || after[end].is_ascii_whitespace();
return (end > 0 && ends_properly).then_some(&after[..end]);
}
}
}
/// What follows the first `marker`, if it is there at all.
fn after_marker<'a>(rest: &'a [u8], marker: &[u8]) -> Option<&'a [u8]> {
rest.windows(marker.len())
.position(|window| window == marker)
.map(|at| &rest[at + marker.len()..])
}
/// What follows a doctype, whose internal subset may hold `>` of its own.
fn after_declaration(rest: &[u8]) -> Option<&[u8]> {
let mut depth = 0usize;
let mut quote = None;
for (at, &byte) in rest.iter().enumerate() {
match quote {
Some(open) if byte == open => quote = None,
Some(_) => {}
None => match byte {
b'"' | b'\'' => quote = Some(byte),
b'[' => depth += 1,
b']' => depth = depth.saturating_sub(1),
b'>' if depth == 0 => return Some(&rest[at + 1..]),
_ => {}
},
}
}
None
}
/// The name without its namespace prefix.
fn local_name(name: &[u8]) -> &[u8] {
match name.iter().rposition(|byte| *byte == b':') {
Some(at) => &name[at + 1..],
None => name,
}
}
fn is_name_byte(byte: u8) -> bool {
byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'-' | b'.' | b':') || byte >= 0x80
}
#[cfg(test)]
mod tests {
use super::*;
const BOOK: &[u8] = b"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n\
<FictionBook xmlns=\"http://www.gribuser.ru/xml/fictionbook/2.0\">\n\
<description><title-info><book-title>The Mill</book-title></title-info></description>";
#[test]
fn a_fictionbook_is_recognised() {
assert!(is_fictionbook(BOOK));
assert!(is_fictionbook(b"<FictionBook>"));
}
#[test]
fn the_prolog_is_stepped_over() {
let head = b"\xef\xbb\xbf <?xml version=\"1.0\"?>\r\n\
<!-- converted by hand -->\n\
<!DOCTYPE FictionBook [ <!ENTITY mill \"<b>the mill</b>\"> ]>\n\
<?xml-stylesheet href=\"fb2.css\"?>\n\
<FictionBook\n xmlns=\"http://www.gribuser.ru/xml/fictionbook/2.0\">";
assert!(is_fictionbook(head));
}
#[test]
fn a_prefixed_root_is_still_a_fictionbook() {
assert!(is_fictionbook(
b"<fb:FictionBook xmlns:fb=\"http://www.gribuser.ru/xml/fictionbook/2.0\">"
));
}
#[test]
fn markup_need_not_be_utf8() {
// windows-1251 for "Мельница" in the title.
let head = b"<?xml version=\"1.0\" encoding=\"windows-1251\"?>\
<FictionBook><description><book-title>\xcc\xe5\xeb\xfc\xed\xe8\xf6\xe0</book-title>";
assert!(is_fictionbook(head));
}
#[test]
fn other_xml_is_not_a_fictionbook() {
assert!(!is_fictionbook(
b"<?xml version=\"1.0\"?>\n<rss version=\"2.0\">"
));
assert!(!is_fictionbook(b"<?xml version=\"1.0\"?><FictionBookish>"));
assert!(!is_fictionbook(b"<html><FictionBook>"));
assert!(!is_fictionbook(b"text before <FictionBook>"));
assert!(!is_fictionbook(b""));
}
#[test]
fn a_root_the_window_cut_short_is_not_taken() {
assert!(!is_fictionbook(b"<?xml version=\"1.0\"?><FictionBook"));
assert!(!is_fictionbook(b"<?xml version=\"1.0\"?><Fiction"));
assert!(!is_fictionbook(b"<?xml version=\"1.0\""));
assert!(!is_fictionbook(b"<!-- never closed <FictionBook>"));
}
fn utf16(text: &str, big_endian: bool) -> Vec<u8> {
let mut out = if big_endian {
UTF16_BE.to_vec()
} else {
UTF16_LE.to_vec()
};
for unit in text.encode_utf16() {
out.extend_from_slice(&if big_endian {
unit.to_be_bytes()
} else {
unit.to_le_bytes()
});
}
out
}
const UTF16_BOOK: &str = "<?xml version=\"1.0\" encoding=\"UTF-16\"?>\r\n\
<FictionBook xmlns=\"http://www.gribuser.ru/xml/fictionbook/2.0\">\
<description><book-title>Мельница</book-title>";
#[test]
fn a_little_endian_utf16_fictionbook_is_recognised() {
assert!(is_fictionbook(&utf16(UTF16_BOOK, false)));
assert!(!is_fictionbook(&utf16(
"<?xml version=\"1.0\"?><rss>",
false
)));
}
#[test]
fn a_big_endian_utf16_fictionbook_is_recognised() {
assert!(is_fictionbook(&utf16(UTF16_BOOK, true)));
assert!(!is_fictionbook(&utf16(
"<?xml version=\"1.0\"?><rss>",
true
)));
}
#[test]
fn utf16_read_in_the_wrong_order_is_not_a_fictionbook() {
let mut swapped = utf16(UTF16_BOOK, false);
swapped[..2].copy_from_slice(UTF16_BE);
assert!(!is_fictionbook(&swapped));
}
#[test]
fn a_utf16_root_the_window_cut_short_is_not_taken() {
let head = utf16("<FictionBook>", false);
for cut in 0..head.len() {
assert!(!is_fictionbook(&head[..cut]), "cut at {cut}");
}
assert!(is_fictionbook(&head));
}
#[test]
fn a_truncated_or_mangled_prolog_does_not_panic() {
let head = b"\xef\xbb\xbf<?xml?><!--c--><!DOCTYPE x [ <!ENTITY a '>'> ] \"]\">\
<FictionBook>";
assert!(is_fictionbook(head));
for cut in 0..head.len() {
let _ = is_fictionbook(&head[..cut]);
}
for byte in 0..=255u8 {
let _ = is_fictionbook(&[b'<', byte]);
let _ = is_fictionbook(&[b'<', b'!', byte, b'>']);
}
}
}
crates/booksniff/src/lib.rs
+378 -13
View File
@@ -9,6 +9,7 @@
//! The containers are unambiguous, so they are read rather than guessed at. //! The containers are unambiguous, so they are read rather than guessed at.
pub mod bytes; pub mod bytes;
pub mod fb2;
pub mod mobi; pub mod mobi;
pub mod zip; pub mod zip;
@@ -22,17 +23,35 @@ const DJVU: &[u8] = b"AT&TFORM";
const OLE2: &[u8] = &[0xd0, 0xcf, 0x11, 0xe0, 0xa1, 0xb1, 0x1a, 0xe1]; const OLE2: &[u8] = &[0xd0, 0xcf, 0x11, 0xe0, 0xa1, 0xb1, 0x1a, 0xe1];
/// A PDF announces its version in its first bytes. /// A PDF announces its version in its first bytes.
const PDF: &[u8] = b"%PDF-"; const PDF: &[u8] = b"%PDF-";
/// Microsoft Reader's signature, ahead of everything else in the file.
const LIT: &[u8] = b"ITOLITLS";
/// The group every RTF document is, opened with the control word that names
/// the format.
const RTF: &[u8] = br"{\rtf";
/// The UTF-8 byte-order mark.
const BOM: &[u8] = b"\xef\xbb\xbf";
/// Archive members that carry no content: resource forks and desktop clutter. /// Archive members that carry no content: resource forks and desktop clutter.
const ARCHIVE_NOISE: &[&str] = &["__macosx/", ".ds_store", "thumbs.db"]; const ARCHIVE_NOISE: &[&str] = &["__macosx/", ".ds_store", "thumbs.db"];
/// What a comic book archive carries besides its pages: the catalogue entry
/// comic readers write into it.
const COMIC_CATALOGUE: &str = "comicinfo.xml";
/// The extensions that say an archive is a comic book.
const COMIC_EXTENSIONS: &[&str] = &["cbz", "cbr"];
/// The extensions of the images a comic book archive is made of.
const PAGE_IMAGES: &[&str] = &[".jpg", ".jpeg", ".png", ".gif", ".webp"];
/// Substrings in a name that mark a file as a sample rather than a book. /// Substrings in a name that mark a file as a sample rather than a book.
const SAMPLE_MARKERS: &[&str] = &["sample", "readme", "read me"]; const SAMPLE_MARKERS: &[&str] = &["sample", "readme", "read me"];
/// Below this, a file is too small to be a book whatever it claims to be. /// Below this, a file is too small to be a book whatever it claims to be.
const SMALLEST_BOOK: u64 = 4096; const SMALLEST_BOOK: u64 = 4096;
/// How much of the front is read to tell text from markup. /// How much of the front is read to tell RTF, FictionBook, markup and text
/// apart.
const TEXT_WINDOW: usize = 4096; const TEXT_WINDOW: usize = 4096;
/// Identify a file. /// Identify a file.
@@ -49,12 +68,17 @@ pub fn sniff(source: &impl Bytes, name: &str) -> Doc {
Kind::Pdf Kind::Pdf
} else if source.magic_at(0, OLE2) { } else if source.magic_at(0, OLE2) {
Kind::Ole2 Kind::Ole2
} else if source.magic_at(0, LIT) {
Kind::Lit
} else if source.magic_at(0, zip::LOCAL_HEADER) { } else if source.magic_at(0, zip::LOCAL_HEADER) {
archive(source, &mut doc) archive(source, &mut doc)
} else if let Some(kind) = mobi::kind(source) { } else if let Some(kind) = mobi::kind(source) {
kind kind
} else { } else {
text(source, &mut doc) match source.slice(0, TEXT_WINDOW) {
Ok(head) => front(&head, &mut doc),
Err(_) => Kind::Unknown,
}
}; };
if doc.kind == Kind::Unknown { if doc.kind == Kind::Unknown {
@@ -80,7 +104,8 @@ pub fn sniff(source: &impl Bytes, name: &str) -> Doc {
doc doc
} }
/// What is inside a zip: an EPUB, a wrapped PDF, or a collection of pages. /// What is inside a zip: an EPUB, a wrapped book, a comic, or a collection of
/// pages.
fn archive(source: &impl Bytes, doc: &mut Doc) -> Kind { fn archive(source: &impl Bytes, doc: &mut Doc) -> Kind {
let Some(names) = zip::members(source) else { let Some(names) = zip::members(source) else {
doc.note("a zip whose directory could not be read"); doc.note("a zip whose directory could not be read");
@@ -99,6 +124,28 @@ fn archive(source: &impl Bytes, doc: &mut Doc) -> Kind {
doc.note(format!("a zip holding one PDF, {pdf:?}")); doc.note(format!("a zip holding one PDF, {pdf:?}"));
return Kind::ZipWrappedPdf; return Kind::ZipWrappedPdf;
} }
if let Some(fb2) = sole_member(&names, ".fb2") {
doc.note(format!("a zip holding one FictionBook, {fb2:?}"));
return Kind::ZipWrappedFb2;
}
if let Some(pages) = comic_pages(&names) {
// A zip of pictures is as often a holiday as a comic, and only the
// names have been read. It is a comic when something says so: the
// catalogue entry a comic reader writes, or a name that claims it.
let named_comic = doc
.declared_ext
.as_deref()
.is_some_and(|ext| COMIC_EXTENSIONS.contains(&ext));
if pages.catalogued || named_comic {
return Kind::Cbz;
}
doc.note(format!(
"a zip of {} image{}, which would be a comic book if its name said .cbz",
pages.count,
if pages.count == 1 { "" } else { "s" }
));
return Kind::Unknown;
}
if lower if lower
.iter() .iter()
.any(|name| name.ends_with(".htm") || name.ends_with(".html") || name.ends_with(".xhtml")) .any(|name| name.ends_with(".htm") || name.ends_with(".html") || name.ends_with(".xhtml"))
@@ -119,25 +166,94 @@ fn archive(source: &impl Bytes, doc: &mut Doc) -> Kind {
/// a collection, and choosing among them would be a guess — better to say it /// a collection, and choosing among them would be a guess — better to say it
/// is not a book than to name one chapter as the whole. /// is not a book than to name one chapter as the whole.
pub fn sole_pdf(names: &[String]) -> Option<&str> { pub fn sole_pdf(names: &[String]) -> Option<&str> {
let real: Vec<&String> = names sole_member(names, ".pdf")
}
/// The one member an archive exists to carry, if it is the only real member
/// and its name ends in `extension`.
fn sole_member<'a>(names: &'a [String], extension: &str) -> Option<&'a str> {
match real_members(names).as_slice() {
[only] if only.to_lowercase().ends_with(extension) => Some(only.as_str()),
_ => None,
}
}
/// The pages of an archive that is images and nothing else.
struct Pages {
/// How many images there are.
count: usize,
/// Whether a comic reader's catalogue entry is there beside them.
catalogued: bool,
}
/// The pages, if an archive is images and nothing else.
///
/// Every real member has to be an image. One PDF or one text file among the
/// pages makes it something else, and saying what would be a guess. Images
/// are known by their names alone: only the archive's directory is read.
fn comic_pages(names: &[String]) -> Option<Pages> {
let mut catalogued = false;
let pages: Vec<String> = real_members(names)
.into_iter()
.map(|name| name.to_lowercase())
.filter(|name| {
let file = name.rsplit('/').next().unwrap_or(name);
let catalogue = file == COMIC_CATALOGUE;
catalogued |= catalogue;
!catalogue
})
.collect();
let all_images = !pages.is_empty()
&& pages
.iter()
.all(|name| PAGE_IMAGES.iter().any(|image| name.ends_with(image)));
all_images.then_some(Pages {
count: pages.len(),
catalogued,
})
}
/// The members that carry content: no directories, no desktop clutter.
fn real_members(names: &[String]) -> Vec<&String> {
names
.iter() .iter()
.filter(|name| !name.ends_with('/')) .filter(|name| !name.ends_with('/'))
.filter(|name| { .filter(|name| {
let lower = name.to_lowercase(); let lower = name.to_lowercase();
!ARCHIVE_NOISE.iter().any(|noise| lower.contains(noise)) !ARCHIVE_NOISE.iter().any(|noise| lower.contains(noise))
}) })
.collect(); .collect()
match real.as_slice() {
[only] if only.to_lowercase().ends_with(".pdf") => Some(only.as_str()),
_ => None,
} }
/// What the front of a file says when it is not a container: RTF, a
/// FictionBook, markup, text, or nothing this knows.
fn front(head: &[u8], doc: &mut Doc) -> Kind {
// Both are recognised on bytes rather than text. RTF is seven-bit by
// design but not always in practice, and a FictionBook is as often
// windows-1251 as UTF-8.
let lead = head.strip_prefix(BOM).unwrap_or(head).trim_ascii_start();
if is_rtf(lead) {
return Kind::Rtf;
}
if fb2::is_fictionbook(head) {
return Kind::Fb2;
}
text(head, doc)
}
/// Whether `lead` opens an RTF document.
///
/// `{\rtf` has to be the whole control word: what follows it is the version,
/// normally `1`, or the space or next control word that ends it. `{\rtfoo` is
/// some other word that happens to start the same way.
fn is_rtf(lead: &[u8]) -> bool {
lead.strip_prefix(RTF)
.and_then(|rest| rest.first())
.is_some_and(|next| next.is_ascii_digit() || next.is_ascii_whitespace() || *next == b'\\')
} }
/// Text, markup, or nothing this knows. /// Text, markup, or nothing this knows.
fn text(source: &impl Bytes, doc: &mut Doc) -> Kind { fn text(head: &[u8], doc: &mut Doc) -> Kind {
let Ok(head) = source.slice(0, TEXT_WINDOW) else {
return Kind::Unknown;
};
if head.is_empty() { if head.is_empty() {
doc.note("the file is empty"); doc.note("the file is empty");
return Kind::Unknown; return Kind::Unknown;
@@ -149,7 +265,7 @@ fn text(source: &impl Bytes, doc: &mut Doc) -> Kind {
} }
// The window may have cut a character in half, which is not the same // The window may have cut a character in half, which is not the same
// thing as the file not being text. // thing as the file not being text.
let head = match std::str::from_utf8(&head) { let head = match std::str::from_utf8(head) {
Ok(text) => text, Ok(text) => text,
Err(e) if e.error_len().is_none() => { Err(e) if e.error_len().is_none() => {
std::str::from_utf8(&head[..e.valid_up_to()]).expect("a valid prefix") std::str::from_utf8(&head[..e.valid_up_to()]).expect("a valid prefix")
@@ -178,10 +294,15 @@ fn names_the_same_thing(extension: &str, kind: Kind) -> bool {
Kind::Djvu => matches!(extension, "djvu" | "djv"), Kind::Djvu => matches!(extension, "djvu" | "djv"),
Kind::Azw3 => matches!(extension, "azw3" | "azw" | "kfx"), Kind::Azw3 => matches!(extension, "azw3" | "azw" | "kfx"),
Kind::Mobi => matches!(extension, "mobi" | "prc" | "azw"), Kind::Mobi => matches!(extension, "mobi" | "prc" | "azw"),
Kind::Lit => extension == "lit",
Kind::Fb2 => extension == "fb2",
Kind::Rtf => extension == "rtf",
Kind::Txt => matches!(extension, "txt" | "text" | "md" | "markdown"), Kind::Txt => matches!(extension, "txt" | "text" | "md" | "markdown"),
Kind::Html => matches!(extension, "html" | "htm" | "xhtml" | "xml"), Kind::Html => matches!(extension, "html" | "htm" | "xhtml" | "xml"),
Kind::Ole2 => matches!(extension, "doc" | "xls" | "ppt"), Kind::Ole2 => matches!(extension, "doc" | "xls" | "ppt"),
Kind::ZipWrappedPdf => extension == "zip", Kind::ZipWrappedPdf => extension == "zip",
Kind::ZipWrappedFb2 => matches!(extension, "zip" | "fbz"),
Kind::Cbz => extension == "cbz",
// Nothing was recognised, so there is nothing to disagree with. // Nothing was recognised, so there is nothing to disagree with.
Kind::Unknown => true, Kind::Unknown => true,
} }
@@ -194,10 +315,15 @@ fn spelling(kind: Kind) -> &'static str {
Kind::Djvu => "DjVu", Kind::Djvu => "DjVu",
Kind::Azw3 => "AZW3", Kind::Azw3 => "AZW3",
Kind::Mobi => "MOBI", Kind::Mobi => "MOBI",
Kind::Lit => "Microsoft Reader",
Kind::Fb2 => "FictionBook",
Kind::Rtf => "RTF",
Kind::Txt => "plain text", Kind::Txt => "plain text",
Kind::Html => "HTML", Kind::Html => "HTML",
Kind::Ole2 => "an OLE2 document", Kind::Ole2 => "an OLE2 document",
Kind::ZipWrappedPdf => "a zip holding one PDF", Kind::ZipWrappedPdf => "a zip holding one PDF",
Kind::ZipWrappedFb2 => "a zip holding one FictionBook",
Kind::Cbz => "a comic book archive",
Kind::Unknown => "nothing recognisable", Kind::Unknown => "nothing recognisable",
} }
} }
@@ -332,4 +458,243 @@ mod tests {
head.truncate(TEXT_WINDOW + 1); head.truncate(TEXT_WINDOW + 1);
assert_eq!(sniff(&head.as_slice(), "a.txt").kind, Kind::Txt); assert_eq!(sniff(&head.as_slice(), "a.txt").kind, Kind::Txt);
} }
fn archive_of(names: &[&str], name: &str) -> Doc {
sniff(&zip::tests::zip(names).as_slice(), name)
}
#[test]
fn rtf_is_recognised_by_its_opening_group() {
assert_eq!(kind_of(br"{\rtf1\ansi\deff0 {\fonttbl", "a.rtf"), Kind::Rtf);
assert_eq!(
kind_of(b"\xef\xbb\xbf\r\n {\\rtf1\\ansi", "a.rtf"),
Kind::Rtf
);
// Eight-bit text in the body does not stop it being RTF.
assert_eq!(kind_of(b"{\\rtf1 caf\xe9 \x00", "a.rtf"), Kind::Rtf);
}
#[test]
fn rtf_cut_short_of_its_control_word_is_not_rtf() {
assert_eq!(sniff(&br"{\rt".as_slice(), "a.rtf").kind, Kind::Txt);
assert_eq!(sniff(&br"{\rtf".as_slice(), "a.rtf").kind, Kind::Txt);
assert_eq!(kind_of(br"{\rtx1", "a.rtf"), Kind::Txt);
assert_eq!(kind_of(br"text {\rtf1", "a.rtf"), Kind::Txt);
}
#[test]
fn rtf_is_the_whole_control_word() {
assert_eq!(kind_of(br"{\rtf\ansi\deff0", "a.rtf"), Kind::Rtf);
assert_eq!(kind_of(b"{\\rtf \\ansi", "a.rtf"), Kind::Rtf);
assert_eq!(kind_of(b"{\\rtf\r\n\\ansi", "a.rtf"), Kind::Rtf);
assert_eq!(kind_of(br"{\rtf2", "a.rtf"), Kind::Rtf);
assert_eq!(kind_of(br"{\rtfoo bar}", "a.rtf"), Kind::Txt);
assert_eq!(kind_of(br"{\rtf}", "a.rtf"), Kind::Txt);
assert_eq!(kind_of(br"{\rtf-1", "a.rtf"), Kind::Txt);
}
#[test]
fn rtf_named_as_a_word_document_is_remarked_on() {
let doc = sniff(&padded(br"{\rtf1\ansi").as_slice(), "letter.doc");
assert_eq!(doc.kind, Kind::Rtf);
assert!(
doc.notes
.iter()
.any(|note| note == "named .doc, but the bytes say RTF"),
"{:?}",
doc.notes
);
}
#[test]
fn microsoft_reader_is_recognised_by_its_signature() {
let doc = sniff(&padded(b"ITOLITLS\x01\x00\x00\x00").as_slice(), "book.lit");
assert_eq!(doc.kind, Kind::Lit);
assert!(doc.notes.is_empty(), "{:?}", doc.notes);
assert_eq!(kind_of(b"ITOLITL", "book.lit"), Kind::Txt);
assert_eq!(kind_of(b" ITOLITLS", "book.lit"), Kind::Txt);
}
#[test]
fn a_fictionbook_is_recognised_and_other_xml_is_not() {
let book = b"<?xml version=\"1.0\" encoding=\"windows-1251\"?>\n\
<FictionBook xmlns=\"http://www.gribuser.ru/xml/fictionbook/2.0\">\
<book-title>\xcc\xe5\xeb\xfc\xed\xe8\xf6\xe0</book-title>";
let doc = sniff(&padded(book).as_slice(), "mill.fb2");
assert_eq!(doc.kind, Kind::Fb2);
assert!(doc.notes.is_empty(), "{:?}", doc.notes);
let feed = b"<?xml version=\"1.0\"?>\n<rss version=\"2.0\"><channel>";
assert_eq!(kind_of(feed, "feed.xml"), Kind::Html);
}
#[test]
fn a_utf16_fictionbook_is_recognised_in_either_byte_order() {
let book = "<?xml version=\"1.0\" encoding=\"UTF-16\"?>\n<FictionBook>";
for big_endian in [false, true] {
let mut bytes = if big_endian {
vec![0xfe, 0xff]
} else {
vec![0xff, 0xfe]
};
for unit in book.encode_utf16() {
bytes.extend_from_slice(&if big_endian {
unit.to_be_bytes()
} else {
unit.to_le_bytes()
});
}
let doc = sniff(&padded(&bytes).as_slice(), "mill.fb2");
assert_eq!(doc.kind, Kind::Fb2, "big-endian: {big_endian}");
assert!(doc.notes.is_empty(), "{:?}", doc.notes);
}
}
#[test]
fn a_fictionbook_named_as_something_else_is_remarked_on() {
let doc = sniff(&padded(b"<FictionBook>").as_slice(), "mill.txt");
assert_eq!(doc.kind, Kind::Fb2);
assert!(
doc.notes.iter().any(|note| note.contains("FictionBook")),
"{:?}",
doc.notes
);
}
#[test]
fn a_zip_holding_one_fictionbook_is_a_wrapped_one() {
let doc = archive_of(&["__MACOSX/._mill.fb2", "mill.fb2"], "mill.fb2.zip");
assert_eq!(doc.kind, Kind::ZipWrappedFb2);
assert_eq!(doc.declared_ext.as_deref(), Some("zip"));
assert!(
doc.notes
.iter()
.any(|note| note.contains("one FictionBook")),
"{:?}",
doc.notes
);
assert!(!doc.notes.iter().any(|note| note.contains("named .")));
let two = archive_of(&["one.fb2", "two.fb2"], "pair.zip");
assert_eq!(two.kind, Kind::Unknown);
}
#[test]
fn a_zip_of_page_images_is_a_comic() {
let doc = archive_of(
&[
"Issue 1/",
"Issue 1/001.jpg",
"Issue 1/002.JPEG",
"Issue 1/003.png",
"Issue 1/004.gif",
"Issue 1/005.webp",
"ComicInfo.xml",
"__MACOSX/Issue 1/._001.jpg",
"Issue 1/Thumbs.db",
".DS_Store",
],
"issue-1.cbz",
);
assert_eq!(doc.kind, Kind::Cbz);
assert!(
!doc.notes.iter().any(|note| note.contains("named .")),
"{:?}",
doc.notes
);
}
#[test]
fn a_zip_of_photographs_is_not_a_comic() {
let doc = archive_of(&["IMG_0001.jpg", "IMG_0002.jpg"], "holiday.zip");
assert_eq!(doc.kind, Kind::Unknown);
assert!(
doc.notes.iter().any(|note| note
== "a zip of 2 images, which would be a comic book if its name said .cbz"),
"{:?}",
doc.notes
);
let one = archive_of(&["IMG_0001.jpg"], "holiday.zip");
assert!(
one.notes
.iter()
.any(|note| note.starts_with("a zip of 1 image,")),
"{:?}",
one.notes
);
}
#[test]
fn the_same_images_named_as_a_comic_are_one() {
let doc = archive_of(&["IMG_0001.jpg", "IMG_0002.jpg"], "holiday.cbz");
assert_eq!(doc.kind, Kind::Cbz);
assert!(
!doc.notes.iter().any(|note| note.contains("named .")),
"{:?}",
doc.notes
);
// A comic zipped and named as if it were RAR is still a comic, and
// the name is remarked on.
let rar = archive_of(&["001.jpg", "002.jpg"], "issue-1.cbr");
assert_eq!(rar.kind, Kind::Cbz);
assert!(
rar.notes.iter().any(|note| note.contains("named .cbr")),
"{:?}",
rar.notes
);
}
#[test]
fn a_comic_saved_as_a_zip_is_known_by_its_catalogue_and_remarked_on() {
let doc = archive_of(&["001.jpg", "002.jpg", "ComicInfo.xml"], "issue-1.zip");
assert_eq!(doc.kind, Kind::Cbz);
assert!(
doc.notes
.iter()
.any(|note| note == "named .zip, but the bytes say a comic book archive"),
"{:?}",
doc.notes
);
}
#[test]
fn a_zip_with_anything_but_images_is_not_a_comic() {
let one_pdf = archive_of(&["001.jpg", "002.jpg", "extras.pdf"], "issue-1.cbz");
assert_eq!(one_pdf.kind, Kind::Unknown);
assert!(
one_pdf
.notes
.iter()
.any(|note| note.contains("none of them a book")),
"{:?}",
one_pdf.notes
);
let readme = archive_of(&["001.jpg", "readme.txt"], "issue-1.cbz");
assert_eq!(readme.kind, Kind::Unknown);
// A name that ends like an image without being one.
let lookalike = archive_of(&["001.jpg", "notes.jpg.txt"], "issue-1.cbz");
assert_eq!(lookalike.kind, Kind::Unknown);
}
#[test]
fn a_zip_with_no_pages_is_not_a_comic() {
let catalogue_only = archive_of(&["ComicInfo.xml", "pages/"], "issue-1.cbz");
assert_eq!(catalogue_only.kind, Kind::Unknown);
let empty = archive_of(&[], "issue-1.cbz");
assert_eq!(empty.kind, Kind::Unknown);
}
#[test]
fn a_truncated_comic_does_not_panic() {
let whole = zip::tests::zip(&["001.jpg", "002.png", "ComicInfo.xml"]);
for cut in 0..whole.len() {
let _ = sniff(&&whole[..cut], "issue-1.cbz");
}
}
} }
crates/booksniff/src/zip.rs
+2 -2
View File
@@ -78,11 +78,11 @@ fn directory_start(source: &impl Bytes) -> Option<(usize, u64)> {
} }
#[cfg(test)] #[cfg(test)]
mod tests { pub(crate) mod tests {
use super::*; use super::*;
/// Build a zip holding `names`, each with empty stored content. /// Build a zip holding `names`, each with empty stored content.
fn zip(names: &[&str]) -> Vec<u8> { pub(crate) fn zip(names: &[&str]) -> Vec<u8> {
let mut out: Vec<u8> = Vec::new(); let mut out: Vec<u8> = Vec::new();
let mut offsets = Vec::new(); let mut offsets = Vec::new();
for name in names { for name in names {
crates/booksniff/tests/cli.rs
+24
View File
@@ -64,6 +64,30 @@ fn several_files_come_back_as_several_records() {
assert_eq!(kinds, ["\"pdf\"", "\"djvu\""]); assert_eq!(kinds, ["\"pdf\"", "\"djvu\""]);
} }
#[test]
fn the_formats_read_from_the_front_come_back_by_their_wire_names() {
let rtf = written("letter.rtf", &padded(br"{\rtf1\ansi"));
let lit = written("book.lit", &padded(b"ITOLITLS"));
let fb2 = written(
"mill.fb2",
&padded(b"<?xml version=\"1.0\"?>\n<FictionBook xmlns=\"http://www.gribuser.ru/xml/fictionbook/2.0\">"),
);
let out = run(&[
rtf.to_str().expect("utf-8 path"),
lit.to_str().expect("utf-8 path"),
fb2.to_str().expect("utf-8 path"),
]);
assert_eq!(out.status.code(), Some(0));
let kinds: Vec<String> = String::from_utf8(out.stdout)
.expect("utf-8")
.lines()
.map(|line| {
serde_json::from_str::<serde_json::Value>(line).expect("valid JSON")["kind"].to_string()
})
.collect();
assert_eq!(kinds, ["\"rtf\"", "\"lit\"", "\"fb2\""]);
}
#[test] #[test]
fn a_file_that_is_not_there_is_an_io_error_and_the_rest_still_run() { fn a_file_that_is_not_there_is_an_io_error_and_the_rest_still_run() {
let path = written("kept.pdf", &padded(b"%PDF-1.7")); let path = written("kept.pdf", &padded(b"%PDF-1.7"));
crates/corpus-core/src/doc.rs
+15
View File
@@ -77,6 +77,12 @@ pub enum Kind {
Azw3, Azw3,
/// Amazon MOBI. /// Amazon MOBI.
Mobi, Mobi,
/// Microsoft Reader.
Lit,
/// FictionBook, the XML ebook format.
Fb2,
/// Rich Text Format.
Rtf,
/// Plain text. /// Plain text.
Txt, Txt,
/// HTML or XHTML. /// HTML or XHTML.
@@ -85,6 +91,10 @@ pub enum Kind {
Ole2, Ole2,
/// A zip archive whose only real content is a single PDF. /// A zip archive whose only real content is a single PDF.
ZipWrappedPdf, ZipWrappedPdf,
/// A zip archive whose only real content is a single FictionBook.
ZipWrappedFb2,
/// A comic book archive: a zip of page images.
Cbz,
/// Recognised as none of the above. /// Recognised as none of the above.
Unknown, Unknown,
} }
@@ -118,6 +128,11 @@ mod tests {
(Kind::Ole2, "\"ole2\""), (Kind::Ole2, "\"ole2\""),
(Kind::Azw3, "\"azw3\""), (Kind::Azw3, "\"azw3\""),
(Kind::ZipWrappedPdf, "\"zip-wrapped-pdf\""), (Kind::ZipWrappedPdf, "\"zip-wrapped-pdf\""),
(Kind::ZipWrappedFb2, "\"zip-wrapped-fb2\""),
(Kind::Fb2, "\"fb2\""),
(Kind::Lit, "\"lit\""),
(Kind::Rtf, "\"rtf\""),
(Kind::Cbz, "\"cbz\""),
(Kind::Unknown, "\"unknown\""), (Kind::Unknown, "\"unknown\""),
]; ];
for (kind, wire) in cases { for (kind, wire) in cases {
crates/corpus-core/src/exit.rs
+2
View File
@@ -30,6 +30,8 @@ pub const UNAVAILABLE: u8 = 69;
pub const SOFTWARE: u8 = 70; pub const SOFTWARE: u8 = 70;
/// Reading or writing failed. /// Reading or writing failed.
pub const IO: u8 = 74; pub const IO: u8 = 74;
/// A service refused for now, and asking again later will work.
pub const TEMPFAIL: u8 = 75;
/// The answer a predicate tool gives. /// The answer a predicate tool gives.
#[derive(Debug, Clone, Copy, PartialEq, Eq)] #[derive(Debug, Clone, Copy, PartialEq, Eq)]
docs/contract.md
+1
View File
@@ -47,6 +47,7 @@ that was concatenated or written on Windows still reads.
| `69` | A program it needs is not installed | | `69` | A program it needs is not installed |
| `70` | Its own fault | | `70` | Its own fault |
| `74` | Reading or writing failed | | `74` | Reading or writing failed |
| `75` | A service refused for now; try again later |
Failures start where `sysexits.h` puts them, so "no" and "broke" are never the Failures start where `sysexits.h` puts them, so "no" and "broke" are never the
same number. That is what makes this work: same number. That is what makes this work:
docs/decisions.md
+18
View File
@@ -20,6 +20,7 @@ to any of them is a change to a decision rather than to an accident.
- [pdfgutter — a page tree that does not add up is refused](#pdfgutter--a-page-tree-that-does-not-add-up-is-refused) - [pdfgutter — a page tree that does not add up is refused](#pdfgutter--a-page-tree-that-does-not-add-up-is-refused)
- [pdfgutter — a sheet on its side is cut the way it is read](#pdfgutter--a-sheet-on-its-side-is-cut-the-way-it-is-read) - [pdfgutter — a sheet on its side is cut the way it is read](#pdfgutter--a-sheet-on-its-side-is-cut-the-way-it-is-read)
- [pdfgutter — an unreadable file is an answer, not a false](#pdfgutter--an-unreadable-file-is-an-answer-not-a-false) - [pdfgutter — an unreadable file is an answer, not a false](#pdfgutter--an-unreadable-file-is-an-answer-not-a-false)
- [bookmeta — a cover is what its bytes say it is](#bookmeta--a-cover-is-what-its-bytes-say-it-is)
- [Every pattern — two regex engines, two Unicode tables](#every-pattern--two-regex-engines-two-unicode-tables) - [Every pattern — two regex engines, two Unicode tables](#every-pattern--two-regex-engines-two-unicode-tables)
## cuenorm — settings after the end time are the format, not a malformation ## cuenorm — settings after the end time are the format, not a malformation
@@ -243,6 +244,23 @@ answers, and the [contract](contract.md) gives them three different numbers. A
caller wanting them collapsed treats 1 and 2 alike; a caller wanting to know caller wanting them collapsed treats 1 and 2 alike; a caller wanting to know
which files it failed on can now find out. which files it failed on can now find out.
## bookmeta — a cover is what its bytes say it is
An EPUB's manifest gives every image a media type, and the cover's is not
used: the type comes from the image's first bytes, and a disagreement is a
note. Elsewhere `bookmeta` reports what the file says rather than what is
true, and this is the one place it does not.
The difference is that a title nobody believes is still a title, while a
media type that is wrong is an image that does not open. A caller handed
`image/png` for a JPEG writes `cover.png`, serves it as PNG, or hands it to a
decoder that refuses it. An image can be replaced by a converter or by hand
without its manifest entry changing, and the note keeps what the file said,
so nothing is lost by correcting it.
The same reasoning goes the other way: bytes that are not an image are not a
cover, whatever the manifest calls them.
## Every pattern — two regex engines, two Unicode tables ## Every pattern — two regex engines, two Unicode tables
Patterns here spell their character classes explicitly rather than leaning on Patterns here spell their character classes explicitly rather than leaning on
docs/records.md
+1 -1
View File
@@ -19,7 +19,7 @@ Part of [catena](../README.md).
| Field | Meaning | | Field | Meaning |
|---|---| |---|---|
| `path` | Where the file was read from, as given | | `path` | Where the file was read from, as given |
| `kind` | `pdf`, `epub`, `djvu`, `azw3`, `mobi`, `txt`, `html`, `ole2`, `zip-wrapped-pdf`, `unknown` | | `kind` | `pdf`, `epub`, `djvu`, `azw3`, `mobi`, `lit`, `fb2`, `rtf`, `txt`, `html`, `ole2`, `cbz`, `zip-wrapped-pdf`, `zip-wrapped-fb2`, `unknown` |
| `declared_ext` | The extension the name claimed, lowercased, without the dot | | `declared_ext` | The extension the name claimed, lowercased, without the dot |
| `bytes` | Size on disk | | `bytes` | Size on disk |
| `pages` | Page count, for paginated formats | | `pages` | Page count, for paginated formats |