Author SHA1 Message Date
funkpower 65fd4adaad Merge remote-tracking branch 'origin/main' into fixes-040 2026-11-13 16:04:00 -05:00
funkpower 58573399e7 Merge pull request 'bibfetch: the books a register has under a title' (#9) from bibfetch-search into main 2026-11-13 15:59:31 -05:00
funkpower d06e13e2f3 Merge pull request 'bookmeta: read an EPUB's cover, by the four routes a package can name one' (#8) from bookmeta-epub-cover into main 2026-11-13 15:59:25 -05:00
funkpower 8ecc4766ee Merge pull request 'booksniff: RTF, Microsoft Reader, FictionBook and comic archives' (#7) from booksniff-more-formats into main 2026-11-13 15:59:20 -05:00
funkpower 3bd3ea11ae 0.5.0 — covers a caller can trust, more formats known, and books found by title
bookmeta gives an EPUB's cover by the routes its package can name one, and
marks a cover the file left open (a MOBI's first image, an EPUB image
found by its name) as guessed. booksniff knows RTF, Microsoft Reader,
FictionBook and comic book zips. bibfetch searches the registers by title
and author for a book with no identifier; covers by ISBN keep under Open
Library's rate limit, and a refusal for it is Error::Limited, not an
outage. The Python package gains fetch_cover, search_title and
catena.Limited.

Breaking: corpus_core::Kind has five new kinds; bibfetch::Error has
Limited; bookmeta::Cover has guessed; bibfetch::Covers has pace.
2026-11-13 15:58:49 -05:00
funkpower b7feefb704 Merge remote-tracking branch 'origin/main' into fixes-040 2026-11-13 15:58:00 -05:00
funkpower f43b9d4900 bibfetch: a title Open Library will not find as written is asked again
Open Library's title search, given an author as well, finds nothing for
The Ladder of Divine Ascent and four editions of Ladder of Divine Ascent.
When the title as given finds nothing, the title without a leading The, A
or An is asked, then the title and author as one free-text query, each
only when the one before found nothing. Google Books, whose anonymous
quota runs out, is asked only after all three.
2026-11-13 15:56:49 -05:00
funkpower 6844f10e0e Merge remote-tracking branch 'origin/fixes-040-sniff' into fixes-040 2026-11-13 15:51:01 -05:00
funkpower 4aae5b6ac0 Merge remote-tracking branch 'origin/fixes-040-search' into fixes-040 2026-11-13 15:51:01 -05:00
funkpower 3fd48d4999 bibfetch: a search candidate is one edition, and a refused fallback says so
A candidate whose edition gave an ISBN but no date or publisher took the
work's first_publish_year and first publisher, so one record could carry
one edition's ISBN and cover beside another's year and imprint. The id,
year, publisher, language and cover now come from the named edition only,
and are left out when it lacks them; the work gives only the title,
authors and subjects. A work with no edition named is named by its key,
not by an ISBN out of its editions' mixed list. The index is no longer
asked for the work-level fields, and gives no edition a length, so Open
Library candidates carry no pages.

When Open Library had nothing and Google Books failed, search answered
with no candidates and the CLI said no register had the title. The
failure is now the answer: Error::Limited for Google's 403, or 429 on
every attempt, which exits 75, and Error::Unreachable otherwise, exit 69.
2026-11-13 15:50:46 -05:00
funkpower 0e83c6ffc1 bookmeta: an EPUB cover by the word, not the letters
The last route took any image whose id or address contained "cover", so
images/discover-ch1.png listed before images/cover.jpg won, and
recovery-map.png alone became the cover. A name now says cover only with
it as a word of its own: words break at anything not a letter or digit
and where a lower case letter meets a capital, so CoverImage and
Book_COVER still do. An image named cover outright comes first, then
manifest order. It is still a guess.

The guide's cover page took its first image, which can be a logo. Where
the page has several, the one whose file name, alt, id or class says
cover is taken; where none does, the first is, with a note saying how
many there were.

Each member was read no further than MAX_COVER, but nothing bounded how
many: two hundred cover-named images of 16 MiB took seven seconds to
inflate and left two hundred notes. One search now reads at most eight
members and stops with one note.

A percent-escape takes two hex digits and nothing else. from_str_radix
also takes a sign, so a%+fb decoded to a, 0x0f, b.
2026-11-13 15:49:18 -05:00
funkpower 5f2a471f58 Merge pull request 'Release 0.4.0' (#10) from release-0.4.0 into main 2026-11-13 15:37:47 -05:00
funkpower bd104834ed 0.4.0 — MOBI and AZW books read without calibre, and a cover from Open Library
bookmeta reads MOBI 6, AZW and AZW3 from the MOBI header and its EXTH
records: title, authors, publisher, description, ISBN, subjects, date and
language, and the cover image the book carries (#1). EXTH 113 is filed by
its shape, an ASIN under asin and a calibre UUID under uuid. The EPUB
reader now keeps dc:description as well.

bibfetch fetches a book's cover from Open Library by ISBN, falling back to
the ISBN-10 form and then to the edition's own cover id, which a Record now
carries; `bibfetch cover` does the same from a terminal (#2). idnorm gains
ISBN-10 and ISBN-13 conversion.

Breaking:
- bookmeta::Metadata has two new public fields, description and cover, so
  a struct literal without `..` no longer compiles.
- bibfetch::Record has a new public field, cover_id.
- bibfetch::Error has a new variant, Invalid, so an exhaustive match
  without it no longer compiles.
- A bibfetch input file named `cover` is now read as the subcommand; pass
  it as ./cover.
2026-11-13 15:27:46 -05:00
7 changed files with 704 additions and 189 deletions

No files matched your search

Generated
+26 -26
View File
@@ -101,7 +101,7 @@ checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
[[package]] [[package]]
name = "avplan" name = "avplan"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -126,7 +126,7 @@ checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5"
[[package]] [[package]]
name = "bibclean" name = "bibclean"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -143,7 +143,7 @@ dependencies = [
[[package]] [[package]]
name = "bibfetch" name = "bibfetch"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -196,7 +196,7 @@ dependencies = [
[[package]] [[package]]
name = "bookmeta" name = "bookmeta"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -210,7 +210,7 @@ dependencies = [
[[package]] [[package]]
name = "booksniff" name = "booksniff"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -250,7 +250,7 @@ checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04"
[[package]] [[package]]
name = "catena" name = "catena"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"bibclean", "bibclean",
"bibfetch", "bibfetch",
@@ -277,7 +277,7 @@ dependencies = [
[[package]] [[package]]
name = "catena-cli" name = "catena-cli"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"clap", "clap",
"clap_mangen", "clap_mangen",
@@ -312,7 +312,7 @@ checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
[[package]] [[package]]
name = "chunkmd" name = "chunkmd"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -334,7 +334,7 @@ dependencies = [
[[package]] [[package]]
name = "citefind" name = "citefind"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"chunkmd", "chunkmd",
@@ -399,7 +399,7 @@ dependencies = [
[[package]] [[package]]
name = "colophon" name = "colophon"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"bibclean", "bibclean",
"catena-cli", "catena-cli",
@@ -448,7 +448,7 @@ dependencies = [
[[package]] [[package]]
name = "corpus-core" name = "corpus-core"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"schemars", "schemars",
"serde", "serde",
@@ -492,7 +492,7 @@ dependencies = [
[[package]] [[package]]
name = "cuekit" name = "cuekit"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"corpus-core", "corpus-core",
"regex", "regex",
@@ -504,7 +504,7 @@ dependencies = [
[[package]] [[package]]
name = "cuenorm" name = "cuenorm"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -514,7 +514,7 @@ dependencies = [
[[package]] [[package]]
name = "cuepack" name = "cuepack"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -525,7 +525,7 @@ dependencies = [
[[package]] [[package]]
name = "cueseg" name = "cueseg"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -612,7 +612,7 @@ dependencies = [
[[package]] [[package]]
name = "emb" name = "emb"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -876,7 +876,7 @@ dependencies = [
[[package]] [[package]]
name = "idnorm" name = "idnorm"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"bibclean", "bibclean",
"catena-cli", "catena-cli",
@@ -942,7 +942,7 @@ dependencies = [
[[package]] [[package]]
name = "langid" name = "langid"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -1022,7 +1022,7 @@ dependencies = [
[[package]] [[package]]
name = "mdscrub" name = "mdscrub"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -1034,7 +1034,7 @@ dependencies = [
[[package]] [[package]]
name = "mdsplit" name = "mdsplit"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -1113,7 +1113,7 @@ checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
[[package]] [[package]]
name = "pdfgutter" name = "pdfgutter"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -1126,7 +1126,7 @@ dependencies = [
[[package]] [[package]]
name = "pdftriage" name = "pdftriage"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -1370,7 +1370,7 @@ checksum = "323c417e1d9665a65b263ec744ba09030cfb277e9daa0b018a4ab62e57bc8189"
[[package]] [[package]]
name = "rrf" name = "rrf"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -1517,7 +1517,7 @@ dependencies = [
[[package]] [[package]]
name = "seriesof" name = "seriesof"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
@@ -1634,7 +1634,7 @@ checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca"
[[package]] [[package]]
name = "textkit" name = "textkit"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"regex", "regex",
"tiktoken-rs", "tiktoken-rs",
@@ -1724,7 +1724,7 @@ checksum = "fd3ca314f692efd6c868f8408f53fe444634a845f96c028b97d35f6a1f79f0ee"
[[package]] [[package]]
name = "tocfind" name = "tocfind"
version = "0.3.1" version = "0.5.0"
dependencies = [ dependencies = [
"catena-cli", "catena-cli",
"clap", "clap",
+27 -27
View File
@@ -3,39 +3,39 @@ resolver = "3"
members = ["crates/*"] members = ["crates/*"]
[workspace.package] [workspace.package]
version = "0.3.1" version = "0.5.0"
edition = "2024" edition = "2024"
rust-version = "1.85" rust-version = "1.85"
license = "MPL-2.0" license = "MPL-2.0"
repository = "https://git.pumpin.online/funkpower/catena" repository = "https://git.pumpin.online/funkpower/catena"
[workspace.dependencies] [workspace.dependencies]
catena-cli = { path = "crates/catena-cli", version = "0.3.1" } catena-cli = { path = "crates/catena-cli", version = "0.5.0" }
corpus-core = { path = "crates/corpus-core", version = "0.3.1" } corpus-core = { path = "crates/corpus-core", version = "0.5.0" }
cuekit = { path = "crates/cuekit", version = "0.3.1" } cuekit = { path = "crates/cuekit", version = "0.5.0" }
textkit = { path = "crates/textkit", version = "0.3.1" } textkit = { path = "crates/textkit", version = "0.5.0" }
chunkmd = { path = "crates/chunkmd", version = "0.3.1" } chunkmd = { path = "crates/chunkmd", version = "0.5.0" }
mdscrub = { path = "crates/mdscrub", version = "0.3.1" } mdscrub = { path = "crates/mdscrub", version = "0.5.0" }
mdsplit = { path = "crates/mdsplit", version = "0.3.1" } mdsplit = { path = "crates/mdsplit", version = "0.5.0" }
pdftriage = { path = "crates/pdftriage", version = "0.3.1" } pdftriage = { path = "crates/pdftriage", version = "0.5.0" }
cuenorm = { path = "crates/cuenorm", version = "0.3.1" } cuenorm = { path = "crates/cuenorm", version = "0.5.0" }
cueseg = { path = "crates/cueseg", version = "0.3.1" } cueseg = { path = "crates/cueseg", version = "0.5.0" }
rrf = { path = "crates/rrf", version = "0.3.1" } rrf = { path = "crates/rrf", version = "0.5.0" }
cuepack = { path = "crates/cuepack", version = "0.3.1" } cuepack = { path = "crates/cuepack", version = "0.5.0" }
bibclean = { path = "crates/bibclean", version = "0.3.1" } bibclean = { path = "crates/bibclean", version = "0.5.0" }
avplan = { path = "crates/avplan", version = "0.3.1" } avplan = { path = "crates/avplan", version = "0.5.0" }
pdfgutter = { path = "crates/pdfgutter", version = "0.3.1" } pdfgutter = { path = "crates/pdfgutter", version = "0.5.0" }
catena = { path = "crates/bindings", version = "0.3.1" } catena = { path = "crates/bindings", version = "0.5.0" }
seriesof = { path = "crates/seriesof", version = "0.3.1" } seriesof = { path = "crates/seriesof", version = "0.5.0" }
idnorm = { path = "crates/idnorm", version = "0.3.1" } idnorm = { path = "crates/idnorm", version = "0.5.0" }
bibfetch = { path = "crates/bibfetch", version = "0.3.1" } bibfetch = { path = "crates/bibfetch", version = "0.5.0" }
bookmeta = { path = "crates/bookmeta", version = "0.3.1" } bookmeta = { path = "crates/bookmeta", version = "0.5.0" }
langid = { path = "crates/langid", version = "0.3.1" } langid = { path = "crates/langid", version = "0.5.0" }
tocfind = { path = "crates/tocfind", version = "0.3.1" } tocfind = { path = "crates/tocfind", version = "0.5.0" }
booksniff = { path = "crates/booksniff", version = "0.3.1" } booksniff = { path = "crates/booksniff", version = "0.5.0" }
citefind = { path = "crates/citefind", version = "0.3.1" } citefind = { path = "crates/citefind", version = "0.5.0" }
colophon = { path = "crates/colophon", version = "0.3.1" } colophon = { path = "crates/colophon", version = "0.5.0" }
emb = { path = "crates/emb", version = "0.3.1" } emb = { path = "crates/emb", version = "0.5.0" }
clap = { version = "4", features = ["derive"] } clap = { version = "4", features = ["derive"] }
clap_mangen = "0.3" clap_mangen = "0.3"
+35 -16
View File
@@ -163,7 +163,7 @@ bytes, the media type and the address asked when there is.
``` ```
$ bibfetch search --title 'The First World War' --author Howard --limit 2 $ bibfetch search --title 'The First World War' --author Howard --limit 2
{"v":1,"id":"9780192853622","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War","authors":["Michael Eliot Howard"],"year":2002,"container":"Oxford University Press","subjects":["World War, 1914-1918"],"language":"eng","pages":154,"cover_id":1123124} {"v":1,"id":"9780192853622","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War","authors":["Michael Eliot Howard"],"year":2002,"container":"Oxford University Press","subjects":["World War, 1914-1918"],"language":"eng","cover_id":1123124}
{"v":1,"id":"9780199205592","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War: A Very Short Introduction","authors":["Michael Howard"],"year":2007,"container":"Oxford University Press","language":"eng","cover_id":14606475} {"v":1,"id":"9780199205592","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War: A Very Short Introduction","authors":["Michael Howard"],"year":2007,"container":"Oxford University Press","language":"eng","cover_id":14606475}
``` ```
@@ -172,18 +172,33 @@ title and perhaps a name. This asks which books a register has under those,
and prints one record per candidate: Open Library's index first, and Google and prints one record per candidate: Open Library's index first, and Google
Books only when Open Library has nothing. Books only when Open Library has nothing.
Open Library's title search is strict. Given an author as well, it finds
nothing for *The Ladder of Divine Ascent* and four editions of *Ladder of
Divine Ascent*. So when the title as given finds nothing, the title without a
leading "The", "A" or "An" is asked next, and then the title and author as one
free-text query. Each is asked only when the one before found nothing, so a
title that matches costs one request. The looser questions can bring back
books that only share words with the title, which is one more reason the
candidates are for the caller to judge.
Each candidate is one edition. Open Library's index answers with works, and Each candidate is one edition. Open Library's index answers with works, and
names the edition of each that best matches the search; what belongs to an names the edition of each that best matches the search. The ISBN, publisher,
edition — its ISBN, publisher, language, year and cover — is that edition's, language, year and cover are that edition's and no other's: a field the
and what belongs to the work — the title, the authors, the subjects and the edition does not give is left out rather than filled from the work, whose
length — is the work's. Google Books answers with volumes, which are editions publishers, years and ISBNs belong to all its editions at once. Only the
already. title, the authors and the subjects, which describe the work, are the work's.
Open Library's index gives no length for an edition, so its candidates carry
no `pages`. Google Books answers with volumes, which are editions already.
| Field | When the candidate has an ISBN | When it has none | | Field | When the candidate has an ISBN | When it has none |
|---|---|---| |---|---|---|
| `id` | The first ISBN whose check digit agrees, in thirteen digits; one written in thirteen is taken over one written in ten | The register's own key: an Open Library id such as `OL37863121M` for the edition, or `OL267171W` for the work when no edition was named; a Google Books volume id | | `id` | The first ISBN of the edition whose check digit agrees, in thirteen digits; one written in thirteen is taken over one written in ten | The register's own key: an Open Library id such as `OL37863121M` for the edition, or `OL267171W` for the work when no edition was named; a Google Books volume id |
| `kind` | `isbn` | `olid` for Open Library, `googlebooks` for Google Books | | `kind` | `isbn` | `olid` for Open Library, `googlebooks` for Google Books |
A work the index names no edition of, or names one it gives neither an ISBN
nor a key for, is a candidate named by the work's key, with its title,
authors and subjects and nothing an edition would say.
An ISBN whose check digit is wrong is passed over rather than corrected, and An ISBN whose check digit is wrong is passed over rather than corrected, and
Google's shelfmarks, typed `OTHER`, are not read as ISBNs even when their Google's shelfmarks, typed `OTHER`, are not read as ISBNs even when their
digits would pass for one. A record of kind `isbn` can be handed straight back digits would pass for one. A record of kind `isbn` can be handed straight back
@@ -194,9 +209,8 @@ to `bibfetch` for everything the registers say about that edition; `olid` and
Open Library's record of the edition the record names, by that edition's ISBN Open Library's record of the edition the record names, by that edition's ISBN
or its key, so `bibfetch cover --cover-id` fetches that edition's cover and no or its key, so `bibfetch cover --cover-id` fetches that edition's cover and no
other. The cover the index picked for the whole work is never asked for, other. The cover the index picked for the whole work is never asked for,
because it is often another edition's. A work the index names no edition of because it is often another edition's. Google Books knows nothing of Open
gives the first ISBN it lists that reads, which may be any edition's, and no Library's covers and gives none.
cover id; Google Books knows nothing of Open Library's covers and gives none.
The candidates are what the register said, in the order it said them. This The candidates are what the register said, in the order it said them. This
makes no judgement about which one is the book in hand, and keeps one that is makes no judgement about which one is the book in hand, and keeps one that is
@@ -210,10 +224,14 @@ bibfetch search --title 'Anathem' --author Stephenson \
| jq -r 'select(.kind == "isbn") | .id' | head -1 | jq -r 'select(.kind == "isbn") | .id' | head -1
``` ```
Open Library answering that it has nothing is an answer, and it stands when Open Library having nothing is an answer only once Google Books has been
Google Books cannot be reached; a register being down is a failure only when asked and has nothing either. When Google Books cannot be reached, the search
both are. Each is asked the same way a lookup asks, retried with the same fails rather than say no register has the title, and when it refuses for its
waits. Open Library is sent the title and the author as written; Google Books quota (`403`, or `429` on every attempt) the failure says so and the command
exits `75`. Open Library being down is a failure only when Google Books has
nothing to offer in its place. Each is asked the same way a lookup asks,
retried with the same waits, except that a `403` from Google Books is not
asked again. Open Library is sent the title and the author as written; Google Books
is sent each word as its own `intitle:` or `inauthor:` term, so nothing in a is sent each word as its own `intitle:` or `inauthor:` term, so nothing in a
title is read as search syntax. title is read as search syntax.
@@ -224,8 +242,9 @@ title is read as search syntax.
| `--limit <COUNT>` | The most candidates to print, `1` to `40`. Default `10` | | `--limit <COUNT>` | The most candidates to print, `1` to `40`. Default `10` |
Exit `0` at least one candidate was printed, `1` no register has the title, Exit `0` at least one candidate was printed, `1` no register has the title,
`65` the title was empty, `69` no register could be reached, `74` the output `65` the title was empty, `69` a register that had to answer could not be
could not be written. reached, `74` the output could not be written, `75` Google Books refused for
its quota.
From Rust, `bibfetch::search(&agent, title, author, limit, attempts)` returns From Rust, `bibfetch::search(&agent, title, author, limit, attempts)` returns
the candidates, empty when no register has the title. the candidates, empty when no register has the title.
+1
View File
@@ -256,6 +256,7 @@ fn search(
eprintln!("bibfetch: {why}"); eprintln!("bibfetch: {why}");
ExitCode::from(match why { ExitCode::from(match why {
bibfetch::Error::Invalid(_) => exit::DATA, bibfetch::Error::Invalid(_) => exit::DATA,
bibfetch::Error::Limited(_) => exit::TEMPFAIL,
_ => exit::UNAVAILABLE, _ => exit::UNAVAILABLE,
}) })
} }
+329 -105
View File
@@ -23,22 +23,34 @@ pub const MAX_LIMIT: usize = 40;
/// What Open Library's index is asked to say about each work, and about the /// What Open Library's index is asked to say about each work, and about the
/// one edition of it that best matches the search. /// one edition of it that best matches the search.
/// ///
/// Not the work's own `cover_i` or `cover_edition_key`: the cover the index /// Of the work, only what describes the work: its publishers, ISBNs,
/// chose for a work is often another edition's. The edition's `cover_i` is /// languages, years and page counts are every edition's at once, and a
/// the first cover in that edition's record, which is what a lookup by ISBN /// record made from them would describe no edition. Not the work's own
/// gives as `cover_id`. /// `cover_i` or `cover_edition_key` either: the cover the index chose for a
const OPEN_LIBRARY_FIELDS: &str = "key,title,subtitle,author_name,first_publish_year,publisher,\ /// work is often another edition's. The edition's `cover_i` is the first
isbn,language,subject,number_of_pages_median,editions,editions.key,editions.isbn,\ /// cover in that edition's record, which is what a lookup by ISBN gives as
editions.publisher,editions.language,editions.publish_date,editions.cover_i"; /// `cover_id`.
const OPEN_LIBRARY_FIELDS: &str = "key,title,subtitle,author_name,subject,editions,editions.key,\
editions.isbn,editions.publisher,editions.language,editions.publish_date,editions.cover_i";
/// Where Google Books is asked.
const GOOGLE_BOOKS: &str = "https://www.googleapis.com/books/v1/volumes";
/// Ask which books a register has under a title, and by an author if one is /// Ask which books a register has under a title, and by an author if one is
/// named. /// named.
/// ///
/// Open Library's index first, and Google Books only when that answered with /// Open Library's index first, and Google Books only when that answered with
/// nothing. A register being down is a failure only when every register /// nothing. Open Library's title search is strict: given an author too, it
/// asked is: Open Library answering that it has nothing is an answer, and it /// finds nothing for "The Ladder of Divine Ascent" that it finds for "Ladder
/// stands when Google Books cannot be reached. Asks nothing when `limit` is /// of Divine Ascent". So when the title as given finds nothing, the title
/// zero; a `limit` over [`MAX_LIMIT`] is taken as that. /// without a leading article is asked, and then the title and author as one
/// free-text query, before Google Books is. Each later question is asked only
/// when the one before it found nothing, and a looser one can return books
/// that only share words with the title, which the caller judges. Once Google Books has been asked, its failure is the answer:
/// [`Error::Limited`] when it refused for its quota, [`Error::Unreachable`]
/// when it could not be reached, since saying no register has the title
/// would be a claim no register made. Asks nothing when `limit` is zero; a
/// `limit` over [`MAX_LIMIT`] is taken as that.
pub fn search( pub fn search(
agent: &ureq::Agent, agent: &ureq::Agent,
title: &str, title: &str,
@@ -46,9 +58,7 @@ pub fn search(
limit: usize, limit: usize,
attempts: u32, attempts: u32,
) -> Result<Vec<Record>, Error> { ) -> Result<Vec<Record>, Error> {
search_with(title, author, limit, |url| { search_with(title, author, limit, |url| get(agent, url, attempts))
crate::ask::get(agent, url, attempts)
})
} }
/// [`search`], with how to fetch an address passed in. /// [`search`], with how to fetch an address passed in.
@@ -75,25 +85,104 @@ fn search_with(
None => Err(Error::Unreachable(format!("{url}: answered 404"))), None => Err(Error::Unreachable(format!("{url}: answered 404"))),
}; };
let open_library = let mut from_open_library = Ok(Vec::new());
fetch(open_library_url(title, author, limit)).and_then(|body| open_library(&body, limit)); for url in open_library_urls(title, author, limit) {
from_open_library = fetch(url).and_then(|body| open_library(&body, limit));
if !from_open_library.as_ref().is_ok_and(Vec::is_empty) {
break;
}
}
let open_library = from_open_library;
if open_library.as_ref().is_ok_and(|found| !found.is_empty()) { if open_library.as_ref().is_ok_and(|found| !found.is_empty()) {
return open_library; return open_library;
} }
let google = match google_url(title, author, limit) { let Some(url) = google_url(title, author, limit) else {
Some(url) => fetch(url).and_then(|body| google_books(&body, limit)),
// Nothing in the title Google could be asked by. // Nothing in the title Google could be asked by.
None => Ok(Vec::new()), return open_library;
}; };
match (open_library, google) { let google = fetch(url).and_then(|body| google_books(&body, limit))?;
(_, Ok(found)) if !found.is_empty() => Ok(found), if google.is_empty() {
(Ok(none), _) => Ok(none), // Open Library's nothing, or its failure, which Google Books having
(Err(why), _) => Err(why), // nothing either does not excuse.
return open_library;
} }
Ok(google)
} }
/// Where to ask Open Library. /// The body at an address, asked as a lookup asks, or `None` when the
/// register says there is no such thing.
///
/// Except that Google Books refusing for its quota is [`Error::Limited`]: a
/// `403` at once, since it is the day's quota and asking again within
/// seconds will not move it, and a `429` once every attempt has been refused.
fn get(agent: &ureq::Agent, url: &str, attempts: u32) -> Result<Option<String>, Error> {
let attempts = attempts.max(1);
let mut last = Error::Unreachable("no attempt was made".into());
for attempt in 0..attempts {
match agent.get(url).call() {
Ok(mut response) => {
return response
.body_mut()
.read_to_string()
.map(Some)
.map_err(|e| Error::Unreachable(e.to_string()));
}
Err(ureq::Error::StatusCode(404)) => return Ok(None),
Err(ureq::Error::StatusCode(status)) => match refused(url, status) {
Some(why) if status == 403 => return Err(why),
Some(why) => last = why,
None => last = Error::Unreachable(format!("{url}: answered {status}")),
},
Err(e) => last = Error::Unreachable(e.to_string()),
}
if attempt + 1 < attempts {
std::thread::sleep(std::time::Duration::from_secs(1 << attempt.min(5)));
}
}
Err(last)
}
/// The refusal a status from a register is, when it is one for asking too
/// often.
fn refused(url: &str, status: u16) -> Option<Error> {
(url.starts_with(GOOGLE_BOOKS) && matches!(status, 403 | 429)).then(|| {
Error::Limited(format!(
"Google Books refused ({status}): its quota for this address is spent for now"
))
})
}
/// Every question to put to Open Library, strictest first.
fn open_library_urls(title: &str, author: Option<&str>, limit: usize) -> Vec<String> {
let mut urls = vec![open_library_url(title, author, limit)];
if let Some(rest) = without_article(title) {
urls.push(open_library_url(rest, author, limit));
}
let words = match author {
Some(author) => format!("{title} {author}"),
None => title.to_string(),
};
urls.push(format!(
"https://openlibrary.org/search.json?q={}&fields={OPEN_LIBRARY_FIELDS}&limit={limit}",
source::encode(&words),
));
urls
}
/// A title without its leading English article, when it has one and
/// something follows it.
fn without_article(title: &str) -> Option<&str> {
let (first, rest) = title.split_once(char::is_whitespace)?;
let rest = rest.trim_start();
(["the", "a", "an"]
.iter()
.any(|article| first.eq_ignore_ascii_case(article))
&& !rest.is_empty())
.then_some(rest)
}
/// Where to ask Open Library by title and author.
fn open_library_url(title: &str, author: Option<&str>, limit: usize) -> String { fn open_library_url(title: &str, author: Option<&str>, limit: usize) -> String {
let author = author let author = author
.map(|author| format!("&author={}", source::encode(author))) .map(|author| format!("&author={}", source::encode(author)))
@@ -124,7 +213,7 @@ fn google_url(title: &str, author: Option<&str>, limit: usize) -> Option<String>
return None; return None;
} }
Some(format!( Some(format!(
"https://www.googleapis.com/books/v1/volumes?q={}&maxResults={limit}", "{GOOGLE_BOOKS}?q={}&maxResults={limit}",
source::encode(&terms.join(" ")) source::encode(&terms.join(" "))
)) ))
} }
@@ -162,18 +251,8 @@ struct Work {
#[serde(default)] #[serde(default)]
author_name: Vec<String>, author_name: Vec<String>,
#[serde(default)] #[serde(default)]
first_publish_year: Option<i64>,
#[serde(default)]
publisher: Vec<String>,
#[serde(default)]
isbn: Vec<String>,
#[serde(default)]
language: Vec<String>,
#[serde(default)]
subject: Vec<String>, subject: Vec<String>,
#[serde(default)] #[serde(default)]
number_of_pages_median: Option<u32>,
#[serde(default)]
editions: Option<Editions>, editions: Option<Editions>,
} }
@@ -212,10 +291,11 @@ fn olid(key: &str) -> Option<String> {
/// Read Open Library's answer into one candidate per work, in its order. /// Read Open Library's answer into one candidate per work, in its order.
/// ///
/// What belongs to an edition — its ISBN, publisher, language, year and /// What belongs to an edition — its ISBN, publisher, language, year and
/// cover — is taken from the edition the index chose as matching best; what /// cover — is taken from the edition the index chose as matching best, and
/// belongs to the work — its title, authors, subjects and length — from the /// left out when that edition does not give it; only the title, authors and
/// work. A work the index names no edition of gives its first ISBN that reads, /// subjects, which describe the work, are the work's. A work the index names
/// which may be any edition's, and no cover id. /// no edition of is named by the work's key and says nothing an edition
/// would.
fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> { fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
let reply: OpenLibraryReply = let reply: OpenLibraryReply =
serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?; serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?;
@@ -228,26 +308,47 @@ fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
} }
fn candidate(work: Work) -> Option<Record> { fn candidate(work: Work) -> Option<Record> {
let edition = work.editions.and_then(|e| e.docs.into_iter().next()); // The edition, and what it is named by: its ISBN, or its key when it has
// none. One with neither cannot be named, and is as good as none.
// The cover is kept only when the record names the edition it is the let edition = work
// cover of, by that edition's ISBN or its key. .editions
let (id, kind, edition_named) = match &edition { .and_then(|e| e.docs.into_iter().next())
Some(edition) => match first_isbn(edition.isbn.iter().map(String::as_str)) { .and_then(|edition| {
Some(isbn) => (isbn, "isbn", true), match first_isbn(edition.isbn.iter().map(String::as_str)) {
None => match edition.key.as_deref().and_then(olid) { Some(isbn) => Some((isbn, "isbn")),
Some(key) => (key, "olid", true), None => edition
None => (work.key.as_deref().and_then(olid)?, "olid", false), .key
}, .as_deref()
}, .and_then(olid)
None => match first_isbn(work.isbn.iter().map(String::as_str)) { .map(|key| (key, "olid")),
Some(isbn) => (isbn, "isbn", false), }
// A work with no key cannot be named, so it is not a candidate. .map(|named| (named, edition))
None => (work.key.as_deref().and_then(olid)?, "olid", false), });
},
};
let mut record = match edition {
Some(((id, kind), edition)) => {
let mut record = source::blank(&id, kind, Source::OpenLibrarySearch); let mut record = source::blank(&id, kind, Source::OpenLibrarySearch);
record.container = edition.publisher.into_iter().next();
record.language = edition.language.into_iter().next();
record.year = edition
.publish_date
.iter()
.find_map(|date| source::year_in(date));
record.cover_id = edition
.cover_i
.and_then(|id| u64::try_from(id).ok())
.filter(|id| *id > 0);
record
}
// The work's own ISBNs, publishers and years are its editions', and
// which belongs with which the index does not say.
None => source::blank(
// A work with no key cannot be named, so it is not a candidate.
&work.key.as_deref().and_then(olid)?,
"olid",
Source::OpenLibrarySearch,
),
};
record.title = match (work.title, work.subtitle) { record.title = match (work.title, work.subtitle) {
(Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")), (Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")),
(title, _) => title, (title, _) => title,
@@ -255,30 +356,6 @@ fn candidate(work: Work) -> Option<Record> {
record.authors = work.author_name; record.authors = work.author_name;
record.subjects = work.subject; record.subjects = work.subject;
record.subjects.truncate(SUBJECT_LIMIT); record.subjects.truncate(SUBJECT_LIMIT);
record.pages = work.number_of_pages_median;
let (publisher, language, year) = match edition {
Some(edition) => {
if edition_named {
record.cover_id = edition
.cover_i
.and_then(|id| u64::try_from(id).ok())
.filter(|id| *id > 0);
}
(
edition.publisher.into_iter().next(),
edition.language.into_iter().next(),
edition
.publish_date
.iter()
.find_map(|date| source::year_in(date)),
)
}
None => (None, None, None),
};
record.container = publisher.or_else(|| work.publisher.into_iter().next());
record.language = language.or_else(|| work.language.into_iter().next());
record.year = year.or(work.first_publish_year);
Some(record) Some(record)
} }
@@ -312,8 +389,9 @@ mod tests {
use super::*; use super::*;
/// Open Library's answer to `title=the first world war&author=howard`, /// Open Library's answer to `title=the first world war&author=howard`,
/// recorded 2026-11-13 with the fields above and `limit=2`, the ISBN and /// recorded 2026-11-13 with `limit=2`, the ISBN and subject lists
/// subject lists shortened. /// shortened, and with the work's own publishers, ISBNs, years and length,
/// which are not asked for and are read past when they come.
const HOWARD: &str = r#"{"numFound": 9, "start": 0, "numFoundExact": true, "num_found": 9, const HOWARD: &str = r#"{"numFound": 9, "start": 0, "numFoundExact": true, "num_found": 9,
"documentation_url": "https://openlibrary.org/dev/docs/api/search", "q": "", "offset": null, "documentation_url": "https://openlibrary.org/dev/docs/api/search", "q": "", "offset": null,
"docs": [ "docs": [
@@ -420,7 +498,8 @@ mod tests {
assert_eq!(first.container.as_deref(), Some("Oxford University Press")); assert_eq!(first.container.as_deref(), Some("Oxford University Press"));
assert_eq!(first.language.as_deref(), Some("eng")); assert_eq!(first.language.as_deref(), Some("eng"));
assert_eq!(first.subjects, ["World War, 1914-1918"]); assert_eq!(first.subjects, ["World War, 1914-1918"]);
assert_eq!(first.pages, Some(154)); // The work's median length is no edition's.
assert_eq!(first.pages, None);
assert_eq!(first.cover_id, Some(1123124)); assert_eq!(first.cover_id, Some(1123124));
let second = &found[1]; let second = &found[1];
@@ -474,17 +553,64 @@ mod tests {
} }
#[test] #[test]
fn a_work_with_no_edition_named_gives_its_isbn_but_no_cover() { fn what_the_edition_does_not_give_is_left_out_not_taken_from_the_work() {
let body = r#"{"docs": [{"key": "/works/OL1W", "title": "A Book", let body = r#"{"docs": [{"key": "/works/OL267171W", "title": "War and Peace",
"isbn": ["0-306-40615-2"], "publisher": ["Plenum"], "first_publish_year": 1981}]}"#; "author_name": ["Leo Tolstoy"], "subject": ["Napoleonic Wars"],
"first_publish_year": 1867, "publisher": ["Crowell"], "language": ["eng"],
"number_of_pages_median": 1296,
"editions": {"docs": [{"key": "/books/OL1M", "isbn": ["9780199232765"],
"cover_i": 5}]}}]}"#;
let found = open_library(body, 5).expect("candidates"); let found = open_library(body, 5).expect("candidates");
let only = &found[0];
assert_eq!( assert_eq!(
(found[0].id.as_str(), found[0].kind.as_str()), (only.id.as_str(), only.kind.as_str()),
("9780306406157", "isbn") ("9780199232765", "isbn")
); );
assert_eq!(found[0].container.as_deref(), Some("Plenum")); assert_eq!(only.cover_id, Some(5));
assert_eq!(found[0].year, Some(1981)); // 1867 and Crowell are other editions'.
assert_eq!(found[0].cover_id, None); assert_eq!(only.year, None);
assert_eq!(only.container, None);
assert_eq!(only.language, None);
assert_eq!(only.pages, None);
// What describes the work is the work's.
assert_eq!(only.title.as_deref(), Some("War and Peace"));
assert_eq!(only.authors, ["Leo Tolstoy"]);
assert_eq!(only.subjects, ["Napoleonic Wars"]);
}
#[test]
fn a_work_with_no_edition_named_is_named_by_its_key_and_says_nothing_an_edition_would() {
let work = r#""key": "/works/OL1W", "title": "A Book", "author_name": ["A. Writer"],
"isbn": ["0-306-40615-2"], "publisher": ["Plenum"], "first_publish_year": 1981,
"language": ["eng"], "number_of_pages_median": 300"#;
// No edition, and an edition with neither an ISBN nor a key.
for editions in [
"",
r#", "editions": {"docs": [{"publisher": ["Plenum"], "cover_i": 7}]}"#,
] {
let body = format!(r#"{{"docs": [{{{work}{editions}}}]}}"#);
let found = open_library(&body, 5).expect("candidates");
let only = &found[0];
// Not 9780306406157: which edition that is, the index does not say.
assert_eq!(
(only.id.as_str(), only.kind.as_str()),
("OL1W", "olid"),
"{body}"
);
assert_eq!(only.title.as_deref(), Some("A Book"));
assert_eq!(only.authors, ["A. Writer"]);
assert_eq!(
(
only.year,
only.container.as_deref(),
only.language.as_deref(),
only.pages,
only.cover_id
),
(None, None, None, None, None),
"{body}"
);
}
} }
#[test] #[test]
@@ -545,8 +671,19 @@ mod tests {
recorded(&mut asked, &bodies), recorded(&mut asked, &bodies),
) )
.expect("candidates"); .expect("candidates");
assert_eq!(asked.len(), 2); // The title as given, without its article, as free text, then Google.
assert!(asked[1].starts_with("https://www.googleapis.com/books/v1/volumes?q=")); assert_eq!(asked.len(), 4, "{asked:?}");
assert!(
asked[1].contains("?title=Google%20Story&author=Vise&"),
"{}",
asked[1]
);
assert!(
asked[2].contains("?q=The%20Google%20Story%20Vise&"),
"{}",
asked[2]
);
assert!(asked[3].starts_with("https://www.googleapis.com/books/v1/volumes?q="));
assert_eq!(found.len(), 2); assert_eq!(found.len(), 2);
let first = &found[0]; let first = &found[0];
@@ -593,15 +730,46 @@ mod tests {
} }
#[test] #[test]
fn open_library_having_nothing_stands_when_google_books_is_down() { fn google_books_down_after_open_library_had_nothing_is_a_failure() {
let mut asked = Vec::new(); let mut asked = Vec::new();
let bodies = [ let bodies = [
("openlibrary.org", Ok(NOTHING)), ("openlibrary.org", Ok(NOTHING)),
("googleapis.com", Err(down())), ("googleapis.com", Err(down())),
]; ];
let found = let failed = search_with("Nothing", None, 5, recorded(&mut asked, &bodies));
search_with("Nothing", None, 5, recorded(&mut asked, &bodies)).expect("an answer"); assert!(matches!(failed, Err(Error::Unreachable(_))), "{failed:?}");
assert!(found.is_empty()); // By title, as free text, then Google.
assert_eq!(asked.len(), 3, "{asked:?}");
}
#[test]
fn google_books_refusing_for_its_quota_is_said_to_be_that() {
let mut asked = Vec::new();
let limited = refused(&google_url("Nothing", None, 5).expect("a url"), 429);
let bodies = [
("openlibrary.org", Ok(NOTHING)),
("googleapis.com", Err(limited.expect("a refusal"))),
];
let failed = search_with("Nothing", None, 5, recorded(&mut asked, &bodies));
assert!(matches!(failed, Err(Error::Limited(_))), "{failed:?}");
}
#[test]
fn only_google_books_quota_statuses_are_refusals() {
let google = google_url("Anathem", None, 5).expect("a url");
for status in [403, 429] {
assert!(
matches!(refused(&google, status), Some(Error::Limited(_))),
"{status}"
);
}
for status in [400, 500, 503] {
assert!(refused(&google, status).is_none(), "{status}");
}
let open_library = open_library_url("Anathem", None, 5);
for status in [403, 429] {
assert!(refused(&open_library, status).is_none(), "{status}");
}
} }
#[test] #[test]
@@ -650,8 +818,50 @@ mod tests {
]; ];
search_with("Anathem", Some(" "), 5, recorded(&mut asked, &bodies)).expect("ok"); search_with("Anathem", Some(" "), 5, recorded(&mut asked, &bodies)).expect("ok");
assert!(!asked[0].contains("author="), "{}", asked[0]); assert!(!asked[0].contains("author="), "{}", asked[0]);
assert!(!asked[1].contains("inauthor"), "{}", asked[1]); assert!(asked[1].contains("?q=Anathem&"), "{}", asked[1]);
assert!(asked[1].contains("q=intitle%3AAnathem&"), "{}", asked[1]); assert!(!asked[2].contains("inauthor"), "{}", asked[2]);
assert!(asked[2].contains("q=intitle%3AAnathem&"), "{}", asked[2]);
}
#[test]
fn a_leading_article_is_dropped_when_the_title_as_given_finds_nothing() {
let mut asked = Vec::new();
let mut answered = 0;
let found = search_with(
"The Ladder of Divine Ascent",
Some("John Climacus"),
5,
|url: &str| {
asked.push(url.to_string());
answered += 1;
Ok(Some(
if answered == 2 { TOLSTOY } else { NOTHING }.to_string(),
))
},
)
.expect("candidates");
assert!(!found.is_empty());
assert_eq!(
asked.len(),
2,
"nothing is asked after a question finds something"
);
assert!(
asked[1].contains("?title=Ladder%20of%20Divine%20Ascent&author=John%20Climacus&"),
"{}",
asked[1]
);
}
#[test]
fn only_a_leading_article_with_words_after_it_is_dropped() {
assert_eq!(without_article("The Republic"), Some("Republic"));
assert_eq!(without_article("an Essay"), Some("Essay"));
assert_eq!(without_article("A"), None);
assert_eq!(without_article("The "), None);
assert_eq!(without_article("Theology"), None);
assert_eq!(without_article("Anathem"), None);
assert_eq!(open_library_urls("Anathem", None, 5).len(), 2);
} }
#[test] #[test]
@@ -723,7 +933,9 @@ intitle:y intitle:z inauthor:O'Brien inauthor:Sons";
let found = search_with("?!", Some("Someone"), 5, recorded(&mut asked, &bodies)) let found = search_with("?!", Some("Someone"), 5, recorded(&mut asked, &bodies))
.expect("an answer"); .expect("an answer");
assert!(found.is_empty()); assert!(found.is_empty());
assert_eq!(asked.len(), 1); // By title and as free text; never Google.
assert_eq!(asked.len(), 2, "{asked:?}");
assert!(asked.iter().all(|url| url.contains("openlibrary.org")));
} }
#[test] #[test]
@@ -744,13 +956,25 @@ intitle:y intitle:z inauthor:O'Brien inauthor:Sons";
let found = let found =
search_with("The Google Story", None, 1, recorded(&mut asked, &bodies)).expect("ok"); search_with("The Google Story", None, 1, recorded(&mut asked, &bodies)).expect("ok");
assert_eq!(found.len(), 1); assert_eq!(found.len(), 1);
assert!(asked[0].ends_with("&limit=1"), "{}", asked[0]); let google = asked.last().expect("Google was asked");
assert!(asked[1].ends_with("&maxResults=1"), "{}", asked[1]); assert!(
asked[..asked.len() - 1]
.iter()
.all(|url| url.ends_with("&limit=1")),
"{asked:?}"
);
assert!(google.ends_with("&maxResults=1"), "{google}");
let mut asked = Vec::new(); let mut asked = Vec::new();
search_with("The Google Story", None, 500, recorded(&mut asked, &bodies)).expect("ok"); search_with("The Google Story", None, 500, recorded(&mut asked, &bodies)).expect("ok");
assert!(asked[0].ends_with("&limit=40"), "{}", asked[0]); let google = asked.last().expect("Google was asked");
assert!(asked[1].ends_with("&maxResults=40"), "{}", asked[1]); assert!(
asked[..asked.len() - 1]
.iter()
.all(|url| url.ends_with("&limit=40")),
"{asked:?}"
);
assert!(google.ends_with("&maxResults=40"), "{google}");
let mut asked = Vec::new(); let mut asked = Vec::new();
let found = let found =
+15 -5
View File
@@ -130,10 +130,18 @@ An EPUB's is the first of these that reaches an image:
2. The item `<meta name="cover" content="…">` names (EPUB 2). Some 2. The item `<meta name="cover" content="…">` names (EPUB 2). Some
converters write the image's address there rather than its id, and that is converters write the image's address there rather than its id, and that is
followed too. followed too.
3. The guide's `<reference type="cover">`, a page, and the first `<img src>` 3. The guide's `<reference type="cover">`, a page, and the `<img src>` or
or SVG `<image href>` on it. A guide that points straight at an image is SVG `<image href>` on it. Where the page has several, the one whose file
taken at its word. name, `alt`, `id` or `class` says `cover` is taken; where none does, the
4. Any image in the manifest whose id or address says `cover`. first is, with a note that there were others. A guide that points straight
at an image is taken at its word.
4. Any image in the manifest whose id or file name says `cover`, one named
`cover` outright first and the rest in manifest order.
A name says `cover` when it has it as a word of its own, in any case. Words
break at anything not a letter or digit and where a lower case letter meets a
capital, so `CoverImage`, `Book_COVER` and `cover-thumb` say it, and
`discover-ch1` and `recovery-map` do not.
A route that names something unusable — not in the archive, not an image — A route that names something unusable — not in the archive, not an image —
leaves a note, and the next route is asked. Addresses are resolved against the leaves a note, and the next route is asked. Addresses are resolved against the
@@ -145,7 +153,9 @@ What the bytes say beats what the manifest says: a JPEG listed as
`image/png` comes back as `image/jpeg`, with a note. A cover larger than `image/png` comes back as `image/jpeg`, with a note. A cover larger than
`bookmeta::MAX_COVER` (16 MiB) is left out with a note, in either format, and `bookmeta::MAX_COVER` (16 MiB) is left out with a note, in either format, and
an EPUB's is never decompressed further than that, whatever size the archive an EPUB's is never decompressed further than that, whatever size the archive
claims for it. claims for it. Nor are more than eight members read looking for one,
however many candidates the manifest lists: the search stops there, with a
note.
`Cover::guessed` says the file left the choice open: a Kindle book's first `Cover::guessed` says the file left the choice open: a Kindle book's first
image record, or an EPUB image picked because its name says `cover`. Either image record, or an EPUB image picked because its name says `cover`. Either
+271 -10
View File
@@ -22,6 +22,13 @@ use crate::{Cover, MAX_COVER, Metadata, Source, image_type};
/// document or is trying to be a decompression bomb. /// document or is trying to be a decompression bomb.
const MAX_PACKAGE: u64 = 4 * 1024 * 1024; const MAX_PACKAGE: u64 = 4 * 1024 * 1024;
/// How many members one search for a cover will decompress.
///
/// Each is bounded by [`MAX_COVER`], but a manifest can list as many
/// cover-named images as it likes, and reading every one of them is time a
/// single book is not owed. A real cover is found in one or two reads.
const MAX_READS: usize = 8;
/// Read what an EPUB says about itself. /// Read what an EPUB says about itself.
pub fn read(path: &str, bytes: &[u8]) -> Metadata { pub fn read(path: &str, bytes: &[u8]) -> Metadata {
let mut meta = Metadata::empty(path, Source::Opf); let mut meta = Metadata::empty(path, Source::Opf);
@@ -295,6 +302,8 @@ fn cover<R: std::io::Read + std::io::Seek>(
archive, archive,
items: &items, items: &items,
tried: Vec::new(), tried: Vec::new(),
reads: 0,
spent: false,
meta, meta,
}; };
@@ -344,12 +353,21 @@ fn cover<R: std::io::Read + std::io::Seek>(
} }
} }
// Last: an image whose id or address says cover. // Last: an image whose id or file name has the word cover in it, one
for item in items.iter().filter(|item| { // named `cover` outright first, then in manifest order.
let file_stem = |item: &Item| stem(item.path.as_deref().unwrap_or(&item.href)).to_string();
let mut named: Vec<&Item> = items
.iter()
.filter(|item| {
item.media_type.starts_with("image/") item.media_type.starts_with("image/")
&& (item.id.to_lowercase().contains("cover") && (says_cover(&item.id) || says_cover(&file_stem(item)))
|| item.href.to_lowercase().contains("cover")) })
}) { .collect();
named.sort_by_key(|item| !file_stem(item).eq_ignore_ascii_case("cover"));
for item in named {
if look.spent {
break;
}
if let Some(mut found) = look.image(base, &item.href) { if let Some(mut found) = look.image(base, &item.href) {
found.guessed = true; found.guessed = true;
return Some(found); return Some(found);
@@ -364,12 +382,19 @@ struct Look<'a, R> {
archive: &'a mut zip::ZipArchive<R>, archive: &'a mut zip::ZipArchive<R>,
items: &'a [Item], items: &'a [Item],
tried: Vec<String>, tried: Vec<String>,
/// How many members have been decompressed, against [`MAX_READS`].
reads: usize,
/// Whether the reads have run out, and said so.
spent: bool,
meta: &'a mut Metadata, meta: &'a mut Metadata,
} }
impl<R: std::io::Read + std::io::Seek> Look<'_, R> { impl<R: std::io::Read + std::io::Seek> Look<'_, R> {
/// Where an address leads, or a note saying it leads nowhere it may. /// Where an address leads, or a note saying it leads nowhere it may.
fn locate(&mut self, base: &str, href: &str) -> Option<String> { fn locate(&mut self, base: &str, href: &str) -> Option<String> {
if self.spent {
return None;
}
let Some(path) = resolve(base, href) else { let Some(path) = resolve(base, href) else {
self.meta.note(format!( self.meta.note(format!(
"the cover's address, {href}, is not a path inside the archive" "the cover's address, {href}, is not a path inside the archive"
@@ -392,6 +417,11 @@ impl<R: std::io::Read + std::io::Seek> Look<'_, R> {
/// The image a cover page shows, or the page itself where the guide /// The image a cover page shows, or the page itself where the guide
/// points straight at an image. /// points straight at an image.
///
/// A page with several images may lead with a logo or an ornament, so
/// the one whose file name, `alt`, `id` or `class` says cover is taken
/// over the first; with none of them saying so, the first is taken, and
/// a note says the page had others.
fn page(&mut self, base: &str, href: &str) -> Option<Cover> { fn page(&mut self, base: &str, href: &str) -> Option<Cover> {
let path = self.locate(base, href)?; let path = self.locate(base, href)?;
let bytes = self.fetch(&path, MAX_PACKAGE as usize, "cover page")?; let bytes = self.fetch(&path, MAX_PACKAGE as usize, "cover page")?;
@@ -399,23 +429,55 @@ impl<R: std::io::Read + std::io::Seek> Look<'_, R> {
return self.check(&path, bytes); return self.check(&path, bytes);
} }
let text = String::from_utf8_lossy(&bytes); let text = String::from_utf8_lossy(&bytes);
let Some(src) = tags(&text).find_map(|(name, tag)| match name { let images: Vec<(String, bool)> = tags(&text)
.filter_map(|(name, tag)| {
let src = match name {
"img" => attr(tag, "src"), "img" => attr(tag, "src"),
"image" => attr(tag, "xlink:href").or_else(|| attr(tag, "href")), "image" => attr(tag, "xlink:href").or_else(|| attr(tag, "href")),
_ => None, _ => None,
}) else { }?;
let named = says_cover(stem(src.split('#').next().unwrap_or("")))
|| ["alt", "id", "class"]
.iter()
.any(|name| attr(tag, name).is_some_and(|value| says_cover(&value)));
Some((src, named))
})
.collect();
let src = if let Some((src, _)) = images.iter().find(|(_, named)| *named) {
src
} else if let Some((first, _)) = images.first() {
if images.len() > 1 {
self.meta.note(format!(
"the cover page, {path}, shows {} images, and the first was taken",
images.len()
));
}
first
} else {
self.meta self.meta
.note(format!("the cover page, {path}, shows no image")); .note(format!("the cover page, {path}, shows no image"));
return None; return None;
}; };
self.image(directory_of(&path), &src) self.image(directory_of(&path), src)
} }
/// One member's bytes, read no further than `cap`. /// One member's bytes, read no further than `cap`.
/// ///
/// The size the archive declares is checked first and the read is /// The size the archive declares is checked first and the read is
/// bounded anyway, because the declaration may be a lie. /// bounded anyway, because the declaration may be a lie. No more than
/// [`MAX_READS`] members are read in all; the first asked for after
/// that ends the search, with a note.
fn fetch(&mut self, path: &str, cap: usize, what: &str) -> Option<Vec<u8>> { fn fetch(&mut self, path: &str, cap: usize, what: &str) -> Option<Vec<u8>> {
if self.spent {
return None;
}
if self.reads == MAX_READS {
self.spent = true;
self.meta.note(format!(
"the search for a cover stopped after reading {MAX_READS} members"
));
return None;
}
let file = match self.archive.by_name(path) { let file = match self.archive.by_name(path) {
Ok(file) => file, Ok(file) => file,
Err(zip::result::ZipError::FileNotFound) => { Err(zip::result::ZipError::FileNotFound) => {
@@ -434,6 +496,7 @@ impl<R: std::io::Read + std::io::Seek> Look<'_, R> {
self.meta.note(too_large); self.meta.note(too_large);
return None; return None;
} }
self.reads += 1;
let mut bytes = Vec::new(); let mut bytes = Vec::new();
if file.take(cap as u64 + 1).read_to_end(&mut bytes).is_err() { if file.take(cap as u64 + 1).read_to_end(&mut bytes).is_err() {
self.meta self.meta
@@ -480,6 +543,50 @@ fn directory_of(path: &str) -> &str {
path.rsplit_once('/').map_or("", |(dir, _)| dir) path.rsplit_once('/').map_or("", |(dir, _)| dir)
} }
/// A path's file name without its extension.
fn stem(path: &str) -> &str {
let name = path.rsplit_once('/').map_or(path, |(_, name)| name);
match name.rsplit_once('.') {
Some((stem, _)) if !stem.is_empty() => stem,
_ => name,
}
}
/// Whether a name has `cover` in it as a word of its own, in any case.
///
/// Words are split at anything not a letter or a digit and where a lower
/// case letter meets a capital, so `CoverImage`, `Book_COVER` and
/// `cover-thumb` say cover, and `discover` and `recovery` do not.
fn says_cover(name: &str) -> bool {
words(name)
.iter()
.any(|word| word.eq_ignore_ascii_case("cover"))
}
/// The words of a name, as [`says_cover`] splits them.
fn words(name: &str) -> Vec<&str> {
let mut out = Vec::new();
let mut start: Option<usize> = None;
let mut prev = ' ';
for (at, c) in name.char_indices() {
let begins =
c.is_alphanumeric() && (start.is_none() || (prev.is_lowercase() && c.is_uppercase()));
if !c.is_alphanumeric() || begins {
if let Some(from) = start.take() {
out.push(&name[from..at]);
}
}
if begins {
start = Some(at);
}
prev = c;
}
if let Some(from) = start {
out.push(&name[from..]);
}
out
}
/// Where an address written in a member in `base` leads inside the archive. /// Where an address written in a member in `base` leads inside the archive.
/// ///
/// The fragment comes off and percent-escapes are undone. An address with a /// The fragment comes off and percent-escapes are undone. An address with a
@@ -523,7 +630,13 @@ fn percent_decode(text: &str) -> Option<String> {
let mut at = 0; let mut at = 0;
while at < raw.len() { while at < raw.len() {
if raw[at] == b'%' { if raw[at] == b'%' {
let hex = std::str::from_utf8(raw.get(at + 1..at + 3)?).ok()?; // Two hex digits and nothing else: `from_str_radix` would also
// take a sign, and read `%+f` as 15.
let hex = raw.get(at + 1..at + 3)?;
if !hex.iter().all(u8::is_ascii_hexdigit) {
return None;
}
let hex = std::str::from_utf8(hex).ok()?;
out.push(u8::from_str_radix(hex, 16).ok()?); out.push(u8::from_str_radix(hex, 16).ok()?);
at += 3; at += 3;
} else { } else {
@@ -878,6 +991,147 @@ mod tests {
assert_eq!(meta.cover.expect("a cover").bytes, JPEG); assert_eq!(meta.cover.expect("a cover").bytes, JPEG);
} }
#[test]
fn a_name_says_cover_only_as_a_word_of_its_own() {
for yes in [
"cover",
"CoverImage",
"Book_COVER",
"cover-thumb",
"frontCover",
"x.cover",
] {
assert!(says_cover(yes), "{yes}");
}
for no in [
"discover-ch1",
"recovery-map",
"covers",
"Coverage",
"uncover",
"",
] {
assert!(!says_cover(no), "{no}");
}
assert_eq!(words("CoverImage_v2"), ["Cover", "Image", "v2"]);
}
#[test]
fn a_name_that_only_contains_cover_is_not_a_cover() {
let package = opf(
r#"<item id="ch1" href="images/discover-ch1.png" media-type="image/png"/>
<item id="i2" href="images/cover.jpg" media-type="image/jpeg"/>"#,
"",
);
let meta = book(
&package,
&[
("OEBPS/images/discover-ch1.png", PNG),
("OEBPS/images/cover.jpg", JPEG),
],
);
let cover = meta.cover.expect("a cover");
assert_eq!(cover.bytes, JPEG);
assert!(cover.guessed);
let alone = opf(
r#"<item id="map" href="images/recovery-map.png" media-type="image/png"/>"#,
"",
);
let meta = book(&alone, &[("OEBPS/images/recovery-map.png", PNG)]);
assert_eq!(meta.cover, None);
assert!(meta.notes.is_empty(), "{:?}", meta.notes);
}
#[test]
fn an_image_named_cover_outright_beats_one_listed_before_it() {
let package = opf(
r#"<item id="cover-thumb" href="images/thumb.png" media-type="image/png"/>
<item id="i2" href="images/Cover.jpg" media-type="image/jpeg"/>"#,
"",
);
let meta = book(
&package,
&[
("OEBPS/images/thumb.png", PNG),
("OEBPS/images/Cover.jpg", JPEG),
],
);
let cover = meta.cover.expect("a cover");
assert_eq!(cover.bytes, JPEG);
assert!(cover.guessed);
}
#[test]
fn the_guide_cover_page_prefers_the_image_that_says_cover() {
let package = opf(
"",
r#"<guide><reference type="cover" href="text/cover.xhtml"/></guide>"#,
);
for art in [
r#"<img src="../images/front-cover.jpg"/>"#,
r#"<img alt="Cover" src="../images/front-cover.jpg"/>"#,
r#"<img id="coverImage" src="../images/front-cover.jpg"/>"#,
r#"<img class="full cover" src="../images/front-cover.jpg"/>"#,
] {
let page = format!(r#"<html><body><img src="../images/logo.png"/>{art}</body></html>"#);
let meta = book(
&package,
&[
("OEBPS/text/cover.xhtml", page.as_bytes()),
("OEBPS/images/logo.png", PNG),
("OEBPS/images/front-cover.jpg", JPEG),
],
);
assert_eq!(meta.cover.expect(art).bytes, JPEG, "{art}");
assert!(meta.notes.is_empty(), "{art}: {:?}", meta.notes);
}
}
#[test]
fn a_guide_cover_page_with_several_unnamed_images_takes_the_first_and_says_so() {
let package = opf(
"",
r#"<guide><reference type="cover" href="text/cover.xhtml"/></guide>"#,
);
let page = br#"<html><body><img src="../images/logo.png"/><img src="../images/art.jpg"/></body></html>"#;
let meta = book(
&package,
&[
("OEBPS/text/cover.xhtml", page),
("OEBPS/images/logo.png", PNG),
("OEBPS/images/art.jpg", JPEG),
],
);
assert_eq!(meta.cover.expect("a cover").bytes, PNG);
assert_eq!(
meta.notes,
["the cover page, OEBPS/text/cover.xhtml, shows 2 images, and the first was taken"]
);
}
#[test]
fn a_search_for_a_cover_reads_no_more_than_its_budget() {
let count = MAX_READS + 4;
let manifest: String = (0..count)
.map(|n| format!(r#"<item id="cover-{n}" href="c{n}.jpg" media-type="image/jpeg"/>"#))
.collect();
let names: Vec<String> = (0..count).map(|n| format!("OEBPS/c{n}.jpg")).collect();
let members: Vec<(&str, &[u8])> = names
.iter()
.map(|name| (name.as_str(), &b"<html>not an image</html>"[..]))
.collect();
let meta = book(&opf(&manifest, ""), &members);
assert_eq!(meta.cover, None);
let mut expected: Vec<String> = (0..MAX_READS)
.map(|n| format!("the cover, OEBPS/c{n}.jpg, is not an image"))
.collect();
expected.push(format!(
"the search for a cover stopped after reading {MAX_READS} members"
));
assert_eq!(meta.notes, expected);
}
#[test] #[test]
fn a_book_that_names_no_cover_has_none_and_says_nothing() { fn a_book_that_names_no_cover_has_none_and_says_nothing() {
let package = opf( let package = opf(
@@ -938,6 +1192,8 @@ mod tests {
archive: &mut archive, archive: &mut archive,
items: &[], items: &[],
tried: Vec::new(), tried: Vec::new(),
reads: 0,
spent: false,
meta: &mut meta, meta: &mut meta,
}; };
let got = look.fetch("big.jpg", cap, "cover").map(|bytes| bytes.len()); let got = look.fetch("big.jpg", cap, "cover").map(|bytes| bytes.len());
@@ -1030,6 +1286,8 @@ mod tests {
Some("OEBPS/text/b.jpg") Some("OEBPS/text/b.jpg")
); );
assert_eq!(resolve("", "a%2Fb.jpg").as_deref(), Some("a/b.jpg")); assert_eq!(resolve("", "a%2Fb.jpg").as_deref(), Some("a/b.jpg"));
assert_eq!(percent_decode("a%2fb%2F").as_deref(), Some("a/b/"));
assert_eq!(percent_decode("a%+fb"), None);
for hostile in [ for hostile in [
"../../c.jpg", "../../c.jpg",
"../../../etc/passwd", "../../../etc/passwd",
@@ -1038,6 +1296,9 @@ mod tests {
"file:///etc/passwd", "file:///etc/passwd",
"%ZZ.jpg", "%ZZ.jpg",
"%ff.jpg", "%ff.jpg",
"a%+fb",
"a%-1b",
"a% fb",
"c%", "c%",
"#only-a-fragment", "#only-a-fragment",
"", "",