Author SHA1 Message Date
funkpower 0e83c6ffc1 bookmeta: an EPUB cover by the word, not the letters
The last route took any image whose id or address contained "cover", so
images/discover-ch1.png listed before images/cover.jpg won, and
recovery-map.png alone became the cover. A name now says cover only with
it as a word of its own: words break at anything not a letter or digit
and where a lower case letter meets a capital, so CoverImage and
Book_COVER still do. An image named cover outright comes first, then
manifest order. It is still a guess.

The guide's cover page took its first image, which can be a logo. Where
the page has several, the one whose file name, alt, id or class says
cover is taken; where none does, the first is, with a note saying how
many there were.

Each member was read no further than MAX_COVER, but nothing bounded how
many: two hundred cover-named images of 16 MiB took seven seconds to
inflate and left two hundred notes. One search now reads at most eight
members and stops with one note.

A percent-escape takes two hex digits and nothing else. from_str_radix
also takes a sign, so a%+fb decoded to a, 0x0f, b.
2026-11-13 15:49:18 -05:00
5 changed files with 398 additions and 253 deletions

No files matched your search

+16 -26
View File
@@ -163,7 +163,7 @@ bytes, the media type and the address asked when there is.
```
$ bibfetch search --title 'The First World War' --author Howard --limit 2
{"v":1,"id":"9780192853622","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War","authors":["Michael Eliot Howard"],"year":2002,"container":"Oxford University Press","subjects":["World War, 1914-1918"],"language":"eng","cover_id":1123124}
{"v":1,"id":"9780192853622","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War","authors":["Michael Eliot Howard"],"year":2002,"container":"Oxford University Press","subjects":["World War, 1914-1918"],"language":"eng","pages":154,"cover_id":1123124}
{"v":1,"id":"9780199205592","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War: A Very Short Introduction","authors":["Michael Howard"],"year":2007,"container":"Oxford University Press","language":"eng","cover_id":14606475}
```
@@ -173,23 +173,17 @@ and prints one record per candidate: Open Library's index first, and Google
Books only when Open Library has nothing.
Each candidate is one edition. Open Library's index answers with works, and
names the edition of each that best matches the search. The ISBN, publisher,
language, year and cover are that edition's and no other's: a field the
edition does not give is left out rather than filled from the work, whose
publishers, years and ISBNs belong to all its editions at once. Only the
title, the authors and the subjects, which describe the work, are the work's.
Open Library's index gives no length for an edition, so its candidates carry
no `pages`. Google Books answers with volumes, which are editions already.
names the edition of each that best matches the search; what belongs to an
edition — its ISBN, publisher, language, year and cover — is that edition's,
and what belongs to the work — the title, the authors, the subjects and the
length — is the work's. Google Books answers with volumes, which are editions
already.
| Field | When the candidate has an ISBN | When it has none |
|---|---|---|
| `id` | The first ISBN of the edition whose check digit agrees, in thirteen digits; one written in thirteen is taken over one written in ten | The register's own key: an Open Library id such as `OL37863121M` for the edition, or `OL267171W` for the work when no edition was named; a Google Books volume id |
| `id` | The first ISBN whose check digit agrees, in thirteen digits; one written in thirteen is taken over one written in ten | The register's own key: an Open Library id such as `OL37863121M` for the edition, or `OL267171W` for the work when no edition was named; a Google Books volume id |
| `kind` | `isbn` | `olid` for Open Library, `googlebooks` for Google Books |
A work the index names no edition of, or names one it gives neither an ISBN
nor a key for, is a candidate named by the work's key, with its title,
authors and subjects and nothing an edition would say.
An ISBN whose check digit is wrong is passed over rather than corrected, and
Google's shelfmarks, typed `OTHER`, are not read as ISBNs even when their
digits would pass for one. A record of kind `isbn` can be handed straight back
@@ -200,8 +194,9 @@ to `bibfetch` for everything the registers say about that edition; `olid` and
Open Library's record of the edition the record names, by that edition's ISBN
or its key, so `bibfetch cover --cover-id` fetches that edition's cover and no
other. The cover the index picked for the whole work is never asked for,
because it is often another edition's. Google Books knows nothing of Open
Library's covers and gives none.
because it is often another edition's. A work the index names no edition of
gives the first ISBN it lists that reads, which may be any edition's, and no
cover id; Google Books knows nothing of Open Library's covers and gives none.
The candidates are what the register said, in the order it said them. This
makes no judgement about which one is the book in hand, and keeps one that is
@@ -215,14 +210,10 @@ bibfetch search --title 'Anathem' --author Stephenson \
| jq -r 'select(.kind == "isbn") | .id' | head -1
```
Open Library having nothing is an answer only once Google Books has been
asked and has nothing either. When Google Books cannot be reached, the search
fails rather than say no register has the title, and when it refuses for its
quota (`403`, or `429` on every attempt) the failure says so and the command
exits `75`. Open Library being down is a failure only when Google Books has
nothing to offer in its place. Each is asked the same way a lookup asks,
retried with the same waits, except that a `403` from Google Books is not
asked again. Open Library is sent the title and the author as written; Google Books
Open Library answering that it has nothing is an answer, and it stands when
Google Books cannot be reached; a register being down is a failure only when
both are. Each is asked the same way a lookup asks, retried with the same
waits. Open Library is sent the title and the author as written; Google Books
is sent each word as its own `intitle:` or `inauthor:` term, so nothing in a
title is read as search syntax.
@@ -233,9 +224,8 @@ title is read as search syntax.
| `--limit <COUNT>` | The most candidates to print, `1` to `40`. Default `10` |
Exit `0` at least one candidate was printed, `1` no register has the title,
`65` the title was empty, `69` a register that had to answer could not be
reached, `74` the output could not be written, `75` Google Books refused for
its quota.
`65` the title was empty, `69` no register could be reached, `74` the output
could not be written.
From Rust, `bibfetch::search(&agent, title, author, limit, attempts)` returns
the candidates, empty when no register has the title.
-1
View File
@@ -256,7 +256,6 @@ fn search(
eprintln!("bibfetch: {why}");
ExitCode::from(match why {
bibfetch::Error::Invalid(_) => exit::DATA,
bibfetch::Error::Limited(_) => exit::TEMPFAIL,
_ => exit::UNAVAILABLE,
})
}
+92 -207
View File
@@ -23,28 +23,22 @@ pub const MAX_LIMIT: usize = 40;
/// What Open Library's index is asked to say about each work, and about the
/// one edition of it that best matches the search.
///
/// Of the work, only what describes the work: its publishers, ISBNs,
/// languages, years and page counts are every edition's at once, and a
/// record made from them would describe no edition. Not the work's own
/// `cover_i` or `cover_edition_key` either: the cover the index chose for a
/// work is often another edition's. The edition's `cover_i` is the first
/// cover in that edition's record, which is what a lookup by ISBN gives as
/// `cover_id`.
const OPEN_LIBRARY_FIELDS: &str = "key,title,subtitle,author_name,subject,editions,editions.key,\
editions.isbn,editions.publisher,editions.language,editions.publish_date,editions.cover_i";
/// Where Google Books is asked.
const GOOGLE_BOOKS: &str = "https://www.googleapis.com/books/v1/volumes";
/// Not the work's own `cover_i` or `cover_edition_key`: the cover the index
/// chose for a work is often another edition's. The edition's `cover_i` is
/// the first cover in that edition's record, which is what a lookup by ISBN
/// gives as `cover_id`.
const OPEN_LIBRARY_FIELDS: &str = "key,title,subtitle,author_name,first_publish_year,publisher,\
isbn,language,subject,number_of_pages_median,editions,editions.key,editions.isbn,\
editions.publisher,editions.language,editions.publish_date,editions.cover_i";
/// Ask which books a register has under a title, and by an author if one is
/// named.
///
/// Open Library's index first, and Google Books only when that answered with
/// nothing. Once Google Books has been asked, its failure is the answer:
/// [`Error::Limited`] when it refused for its quota, [`Error::Unreachable`]
/// when it could not be reached, since saying no register has the title
/// would be a claim no register made. Asks nothing when `limit` is zero; a
/// `limit` over [`MAX_LIMIT`] is taken as that.
/// nothing. A register being down is a failure only when every register
/// asked is: Open Library answering that it has nothing is an answer, and it
/// stands when Google Books cannot be reached. Asks nothing when `limit` is
/// zero; a `limit` over [`MAX_LIMIT`] is taken as that.
pub fn search(
agent: &ureq::Agent,
title: &str,
@@ -52,7 +46,9 @@ pub fn search(
limit: usize,
attempts: u32,
) -> Result<Vec<Record>, Error> {
search_with(title, author, limit, |url| get(agent, url, attempts))
search_with(title, author, limit, |url| {
crate::ask::get(agent, url, attempts)
})
}
/// [`search`], with how to fetch an address passed in.
@@ -85,60 +81,16 @@ fn search_with(
return open_library;
}
let Some(url) = google_url(title, author, limit) else {
let google = match google_url(title, author, limit) {
Some(url) => fetch(url).and_then(|body| google_books(&body, limit)),
// Nothing in the title Google could be asked by.
return open_library;
None => Ok(Vec::new()),
};
let google = fetch(url).and_then(|body| google_books(&body, limit))?;
if google.is_empty() {
// Open Library's nothing, or its failure, which Google Books having
// nothing either does not excuse.
return open_library;
match (open_library, google) {
(_, Ok(found)) if !found.is_empty() => Ok(found),
(Ok(none), _) => Ok(none),
(Err(why), _) => Err(why),
}
Ok(google)
}
/// The body at an address, asked as a lookup asks, or `None` when the
/// register says there is no such thing.
///
/// Except that Google Books refusing for its quota is [`Error::Limited`]: a
/// `403` at once, since it is the day's quota and asking again within
/// seconds will not move it, and a `429` once every attempt has been refused.
fn get(agent: &ureq::Agent, url: &str, attempts: u32) -> Result<Option<String>, Error> {
let attempts = attempts.max(1);
let mut last = Error::Unreachable("no attempt was made".into());
for attempt in 0..attempts {
match agent.get(url).call() {
Ok(mut response) => {
return response
.body_mut()
.read_to_string()
.map(Some)
.map_err(|e| Error::Unreachable(e.to_string()));
}
Err(ureq::Error::StatusCode(404)) => return Ok(None),
Err(ureq::Error::StatusCode(status)) => match refused(url, status) {
Some(why) if status == 403 => return Err(why),
Some(why) => last = why,
None => last = Error::Unreachable(format!("{url}: answered {status}")),
},
Err(e) => last = Error::Unreachable(e.to_string()),
}
if attempt + 1 < attempts {
std::thread::sleep(std::time::Duration::from_secs(1 << attempt.min(5)));
}
}
Err(last)
}
/// The refusal a status from a register is, when it is one for asking too
/// often.
fn refused(url: &str, status: u16) -> Option<Error> {
(url.starts_with(GOOGLE_BOOKS) && matches!(status, 403 | 429)).then(|| {
Error::Limited(format!(
"Google Books refused ({status}): its quota for this address is spent for now"
))
})
}
/// Where to ask Open Library.
@@ -172,7 +124,7 @@ fn google_url(title: &str, author: Option<&str>, limit: usize) -> Option<String>
return None;
}
Some(format!(
"{GOOGLE_BOOKS}?q={}&maxResults={limit}",
"https://www.googleapis.com/books/v1/volumes?q={}&maxResults={limit}",
source::encode(&terms.join(" "))
))
}
@@ -210,8 +162,18 @@ struct Work {
#[serde(default)]
author_name: Vec<String>,
#[serde(default)]
first_publish_year: Option<i64>,
#[serde(default)]
publisher: Vec<String>,
#[serde(default)]
isbn: Vec<String>,
#[serde(default)]
language: Vec<String>,
#[serde(default)]
subject: Vec<String>,
#[serde(default)]
number_of_pages_median: Option<u32>,
#[serde(default)]
editions: Option<Editions>,
}
@@ -250,11 +212,10 @@ fn olid(key: &str) -> Option<String> {
/// Read Open Library's answer into one candidate per work, in its order.
///
/// What belongs to an edition — its ISBN, publisher, language, year and
/// cover — is taken from the edition the index chose as matching best, and
/// left out when that edition does not give it; only the title, authors and
/// subjects, which describe the work, are the work's. A work the index names
/// no edition of is named by the work's key and says nothing an edition
/// would.
/// cover — is taken from the edition the index chose as matching best; what
/// belongs to the work — its title, authors, subjects and length — from the
/// work. A work the index names no edition of gives its first ISBN that reads,
/// which may be any edition's, and no cover id.
fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
let reply: OpenLibraryReply =
serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?;
@@ -267,47 +228,26 @@ fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
}
fn candidate(work: Work) -> Option<Record> {
// The edition, and what it is named by: its ISBN, or its key when it has
// none. One with neither cannot be named, and is as good as none.
let edition = work
.editions
.and_then(|e| e.docs.into_iter().next())
.and_then(|edition| {
match first_isbn(edition.isbn.iter().map(String::as_str)) {
Some(isbn) => Some((isbn, "isbn")),
None => edition
.key
.as_deref()
.and_then(olid)
.map(|key| (key, "olid")),
}
.map(|named| (named, edition))
});
let edition = work.editions.and_then(|e| e.docs.into_iter().next());
let mut record = match edition {
Some(((id, kind), edition)) => {
let mut record = source::blank(&id, kind, Source::OpenLibrarySearch);
record.container = edition.publisher.into_iter().next();
record.language = edition.language.into_iter().next();
record.year = edition
.publish_date
.iter()
.find_map(|date| source::year_in(date));
record.cover_id = edition
.cover_i
.and_then(|id| u64::try_from(id).ok())
.filter(|id| *id > 0);
record
}
// The work's own ISBNs, publishers and years are its editions', and
// which belongs with which the index does not say.
None => source::blank(
// The cover is kept only when the record names the edition it is the
// cover of, by that edition's ISBN or its key.
let (id, kind, edition_named) = match &edition {
Some(edition) => match first_isbn(edition.isbn.iter().map(String::as_str)) {
Some(isbn) => (isbn, "isbn", true),
None => match edition.key.as_deref().and_then(olid) {
Some(key) => (key, "olid", true),
None => (work.key.as_deref().and_then(olid)?, "olid", false),
},
},
None => match first_isbn(work.isbn.iter().map(String::as_str)) {
Some(isbn) => (isbn, "isbn", false),
// A work with no key cannot be named, so it is not a candidate.
&work.key.as_deref().and_then(olid)?,
"olid",
Source::OpenLibrarySearch,
),
None => (work.key.as_deref().and_then(olid)?, "olid", false),
},
};
let mut record = source::blank(&id, kind, Source::OpenLibrarySearch);
record.title = match (work.title, work.subtitle) {
(Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")),
(title, _) => title,
@@ -315,6 +255,30 @@ fn candidate(work: Work) -> Option<Record> {
record.authors = work.author_name;
record.subjects = work.subject;
record.subjects.truncate(SUBJECT_LIMIT);
record.pages = work.number_of_pages_median;
let (publisher, language, year) = match edition {
Some(edition) => {
if edition_named {
record.cover_id = edition
.cover_i
.and_then(|id| u64::try_from(id).ok())
.filter(|id| *id > 0);
}
(
edition.publisher.into_iter().next(),
edition.language.into_iter().next(),
edition
.publish_date
.iter()
.find_map(|date| source::year_in(date)),
)
}
None => (None, None, None),
};
record.container = publisher.or_else(|| work.publisher.into_iter().next());
record.language = language.or_else(|| work.language.into_iter().next());
record.year = year.or(work.first_publish_year);
Some(record)
}
@@ -348,9 +312,8 @@ mod tests {
use super::*;
/// Open Library's answer to `title=the first world war&author=howard`,
/// recorded 2026-11-13 with `limit=2`, the ISBN and subject lists
/// shortened, and with the work's own publishers, ISBNs, years and length,
/// which are not asked for and are read past when they come.
/// recorded 2026-11-13 with the fields above and `limit=2`, the ISBN and
/// subject lists shortened.
const HOWARD: &str = r#"{"numFound": 9, "start": 0, "numFoundExact": true, "num_found": 9,
"documentation_url": "https://openlibrary.org/dev/docs/api/search", "q": "", "offset": null,
"docs": [
@@ -457,8 +420,7 @@ mod tests {
assert_eq!(first.container.as_deref(), Some("Oxford University Press"));
assert_eq!(first.language.as_deref(), Some("eng"));
assert_eq!(first.subjects, ["World War, 1914-1918"]);
// The work's median length is no edition's.
assert_eq!(first.pages, None);
assert_eq!(first.pages, Some(154));
assert_eq!(first.cover_id, Some(1123124));
let second = &found[1];
@@ -512,64 +474,17 @@ mod tests {
}
#[test]
fn what_the_edition_does_not_give_is_left_out_not_taken_from_the_work() {
let body = r#"{"docs": [{"key": "/works/OL267171W", "title": "War and Peace",
"author_name": ["Leo Tolstoy"], "subject": ["Napoleonic Wars"],
"first_publish_year": 1867, "publisher": ["Crowell"], "language": ["eng"],
"number_of_pages_median": 1296,
"editions": {"docs": [{"key": "/books/OL1M", "isbn": ["9780199232765"],
"cover_i": 5}]}}]}"#;
fn a_work_with_no_edition_named_gives_its_isbn_but_no_cover() {
let body = r#"{"docs": [{"key": "/works/OL1W", "title": "A Book",
"isbn": ["0-306-40615-2"], "publisher": ["Plenum"], "first_publish_year": 1981}]}"#;
let found = open_library(body, 5).expect("candidates");
let only = &found[0];
assert_eq!(
(only.id.as_str(), only.kind.as_str()),
("9780199232765", "isbn")
(found[0].id.as_str(), found[0].kind.as_str()),
("9780306406157", "isbn")
);
assert_eq!(only.cover_id, Some(5));
// 1867 and Crowell are other editions'.
assert_eq!(only.year, None);
assert_eq!(only.container, None);
assert_eq!(only.language, None);
assert_eq!(only.pages, None);
// What describes the work is the work's.
assert_eq!(only.title.as_deref(), Some("War and Peace"));
assert_eq!(only.authors, ["Leo Tolstoy"]);
assert_eq!(only.subjects, ["Napoleonic Wars"]);
}
#[test]
fn a_work_with_no_edition_named_is_named_by_its_key_and_says_nothing_an_edition_would() {
let work = r#""key": "/works/OL1W", "title": "A Book", "author_name": ["A. Writer"],
"isbn": ["0-306-40615-2"], "publisher": ["Plenum"], "first_publish_year": 1981,
"language": ["eng"], "number_of_pages_median": 300"#;
// No edition, and an edition with neither an ISBN nor a key.
for editions in [
"",
r#", "editions": {"docs": [{"publisher": ["Plenum"], "cover_i": 7}]}"#,
] {
let body = format!(r#"{{"docs": [{{{work}{editions}}}]}}"#);
let found = open_library(&body, 5).expect("candidates");
let only = &found[0];
// Not 9780306406157: which edition that is, the index does not say.
assert_eq!(
(only.id.as_str(), only.kind.as_str()),
("OL1W", "olid"),
"{body}"
);
assert_eq!(only.title.as_deref(), Some("A Book"));
assert_eq!(only.authors, ["A. Writer"]);
assert_eq!(
(
only.year,
only.container.as_deref(),
only.language.as_deref(),
only.pages,
only.cover_id
),
(None, None, None, None, None),
"{body}"
);
}
assert_eq!(found[0].container.as_deref(), Some("Plenum"));
assert_eq!(found[0].year, Some(1981));
assert_eq!(found[0].cover_id, None);
}
#[test]
@@ -678,45 +593,15 @@ mod tests {
}
#[test]
fn google_books_down_after_open_library_had_nothing_is_a_failure() {
fn open_library_having_nothing_stands_when_google_books_is_down() {
let mut asked = Vec::new();
let bodies = [
("openlibrary.org", Ok(NOTHING)),
("googleapis.com", Err(down())),
];
let failed = search_with("Nothing", None, 5, recorded(&mut asked, &bodies));
assert!(matches!(failed, Err(Error::Unreachable(_))), "{failed:?}");
assert_eq!(asked.len(), 2);
}
#[test]
fn google_books_refusing_for_its_quota_is_said_to_be_that() {
let mut asked = Vec::new();
let limited = refused(&google_url("Nothing", None, 5).expect("a url"), 429);
let bodies = [
("openlibrary.org", Ok(NOTHING)),
("googleapis.com", Err(limited.expect("a refusal"))),
];
let failed = search_with("Nothing", None, 5, recorded(&mut asked, &bodies));
assert!(matches!(failed, Err(Error::Limited(_))), "{failed:?}");
}
#[test]
fn only_google_books_quota_statuses_are_refusals() {
let google = google_url("Anathem", None, 5).expect("a url");
for status in [403, 429] {
assert!(
matches!(refused(&google, status), Some(Error::Limited(_))),
"{status}"
);
}
for status in [400, 500, 503] {
assert!(refused(&google, status).is_none(), "{status}");
}
let open_library = open_library_url("Anathem", None, 5);
for status in [403, 429] {
assert!(refused(&open_library, status).is_none(), "{status}");
}
let found =
search_with("Nothing", None, 5, recorded(&mut asked, &bodies)).expect("an answer");
assert!(found.is_empty());
}
#[test]
+15 -5
View File
@@ -130,10 +130,18 @@ An EPUB's is the first of these that reaches an image:
2. The item `<meta name="cover" content="…">` names (EPUB 2). Some
converters write the image's address there rather than its id, and that is
followed too.
3. The guide's `<reference type="cover">`, a page, and the first `<img src>`
or SVG `<image href>` on it. A guide that points straight at an image is
taken at its word.
4. Any image in the manifest whose id or address says `cover`.
3. The guide's `<reference type="cover">`, a page, and the `<img src>` or
SVG `<image href>` on it. Where the page has several, the one whose file
name, `alt`, `id` or `class` says `cover` is taken; where none does, the
first is, with a note that there were others. A guide that points straight
at an image is taken at its word.
4. Any image in the manifest whose id or file name says `cover`, one named
`cover` outright first and the rest in manifest order.
A name says `cover` when it has it as a word of its own, in any case. Words
break at anything not a letter or digit and where a lower case letter meets a
capital, so `CoverImage`, `Book_COVER` and `cover-thumb` say it, and
`discover-ch1` and `recovery-map` do not.
A route that names something unusable — not in the archive, not an image —
leaves a note, and the next route is asked. Addresses are resolved against the
@@ -145,7 +153,9 @@ What the bytes say beats what the manifest says: a JPEG listed as
`image/png` comes back as `image/jpeg`, with a note. A cover larger than
`bookmeta::MAX_COVER` (16 MiB) is left out with a note, in either format, and
an EPUB's is never decompressed further than that, whatever size the archive
claims for it.
claims for it. Nor are more than eight members read looking for one,
however many candidates the manifest lists: the search stops there, with a
note.
`Cover::guessed` says the file left the choice open: a Kindle book's first
image record, or an EPUB image picked because its name says `cover`. Either
+275 -14
View File
@@ -22,6 +22,13 @@ use crate::{Cover, MAX_COVER, Metadata, Source, image_type};
/// document or is trying to be a decompression bomb.
const MAX_PACKAGE: u64 = 4 * 1024 * 1024;
/// How many members one search for a cover will decompress.
///
/// Each is bounded by [`MAX_COVER`], but a manifest can list as many
/// cover-named images as it likes, and reading every one of them is time a
/// single book is not owed. A real cover is found in one or two reads.
const MAX_READS: usize = 8;
/// Read what an EPUB says about itself.
pub fn read(path: &str, bytes: &[u8]) -> Metadata {
let mut meta = Metadata::empty(path, Source::Opf);
@@ -295,6 +302,8 @@ fn cover<R: std::io::Read + std::io::Seek>(
archive,
items: &items,
tried: Vec::new(),
reads: 0,
spent: false,
meta,
};
@@ -344,12 +353,21 @@ fn cover<R: std::io::Read + std::io::Seek>(
}
}
// Last: an image whose id or address says cover.
for item in items.iter().filter(|item| {
item.media_type.starts_with("image/")
&& (item.id.to_lowercase().contains("cover")
|| item.href.to_lowercase().contains("cover"))
}) {
// Last: an image whose id or file name has the word cover in it, one
// named `cover` outright first, then in manifest order.
let file_stem = |item: &Item| stem(item.path.as_deref().unwrap_or(&item.href)).to_string();
let mut named: Vec<&Item> = items
.iter()
.filter(|item| {
item.media_type.starts_with("image/")
&& (says_cover(&item.id) || says_cover(&file_stem(item)))
})
.collect();
named.sort_by_key(|item| !file_stem(item).eq_ignore_ascii_case("cover"));
for item in named {
if look.spent {
break;
}
if let Some(mut found) = look.image(base, &item.href) {
found.guessed = true;
return Some(found);
@@ -364,12 +382,19 @@ struct Look<'a, R> {
archive: &'a mut zip::ZipArchive<R>,
items: &'a [Item],
tried: Vec<String>,
/// How many members have been decompressed, against [`MAX_READS`].
reads: usize,
/// Whether the reads have run out, and said so.
spent: bool,
meta: &'a mut Metadata,
}
impl<R: std::io::Read + std::io::Seek> Look<'_, R> {
/// Where an address leads, or a note saying it leads nowhere it may.
fn locate(&mut self, base: &str, href: &str) -> Option<String> {
if self.spent {
return None;
}
let Some(path) = resolve(base, href) else {
self.meta.note(format!(
"the cover's address, {href}, is not a path inside the archive"
@@ -392,6 +417,11 @@ impl<R: std::io::Read + std::io::Seek> Look<'_, R> {
/// The image a cover page shows, or the page itself where the guide
/// points straight at an image.
///
/// A page with several images may lead with a logo or an ornament, so
/// the one whose file name, `alt`, `id` or `class` says cover is taken
/// over the first; with none of them saying so, the first is taken, and
/// a note says the page had others.
fn page(&mut self, base: &str, href: &str) -> Option<Cover> {
let path = self.locate(base, href)?;
let bytes = self.fetch(&path, MAX_PACKAGE as usize, "cover page")?;
@@ -399,23 +429,55 @@ impl<R: std::io::Read + std::io::Seek> Look<'_, R> {
return self.check(&path, bytes);
}
let text = String::from_utf8_lossy(&bytes);
let Some(src) = tags(&text).find_map(|(name, tag)| match name {
"img" => attr(tag, "src"),
"image" => attr(tag, "xlink:href").or_else(|| attr(tag, "href")),
_ => None,
}) else {
let images: Vec<(String, bool)> = tags(&text)
.filter_map(|(name, tag)| {
let src = match name {
"img" => attr(tag, "src"),
"image" => attr(tag, "xlink:href").or_else(|| attr(tag, "href")),
_ => None,
}?;
let named = says_cover(stem(src.split('#').next().unwrap_or("")))
|| ["alt", "id", "class"]
.iter()
.any(|name| attr(tag, name).is_some_and(|value| says_cover(&value)));
Some((src, named))
})
.collect();
let src = if let Some((src, _)) = images.iter().find(|(_, named)| *named) {
src
} else if let Some((first, _)) = images.first() {
if images.len() > 1 {
self.meta.note(format!(
"the cover page, {path}, shows {} images, and the first was taken",
images.len()
));
}
first
} else {
self.meta
.note(format!("the cover page, {path}, shows no image"));
return None;
};
self.image(directory_of(&path), &src)
self.image(directory_of(&path), src)
}
/// One member's bytes, read no further than `cap`.
///
/// The size the archive declares is checked first and the read is
/// bounded anyway, because the declaration may be a lie.
/// bounded anyway, because the declaration may be a lie. No more than
/// [`MAX_READS`] members are read in all; the first asked for after
/// that ends the search, with a note.
fn fetch(&mut self, path: &str, cap: usize, what: &str) -> Option<Vec<u8>> {
if self.spent {
return None;
}
if self.reads == MAX_READS {
self.spent = true;
self.meta.note(format!(
"the search for a cover stopped after reading {MAX_READS} members"
));
return None;
}
let file = match self.archive.by_name(path) {
Ok(file) => file,
Err(zip::result::ZipError::FileNotFound) => {
@@ -434,6 +496,7 @@ impl<R: std::io::Read + std::io::Seek> Look<'_, R> {
self.meta.note(too_large);
return None;
}
self.reads += 1;
let mut bytes = Vec::new();
if file.take(cap as u64 + 1).read_to_end(&mut bytes).is_err() {
self.meta
@@ -480,6 +543,50 @@ fn directory_of(path: &str) -> &str {
path.rsplit_once('/').map_or("", |(dir, _)| dir)
}
/// A path's file name without its extension.
fn stem(path: &str) -> &str {
let name = path.rsplit_once('/').map_or(path, |(_, name)| name);
match name.rsplit_once('.') {
Some((stem, _)) if !stem.is_empty() => stem,
_ => name,
}
}
/// Whether a name has `cover` in it as a word of its own, in any case.
///
/// Words are split at anything not a letter or a digit and where a lower
/// case letter meets a capital, so `CoverImage`, `Book_COVER` and
/// `cover-thumb` say cover, and `discover` and `recovery` do not.
fn says_cover(name: &str) -> bool {
words(name)
.iter()
.any(|word| word.eq_ignore_ascii_case("cover"))
}
/// The words of a name, as [`says_cover`] splits them.
fn words(name: &str) -> Vec<&str> {
let mut out = Vec::new();
let mut start: Option<usize> = None;
let mut prev = ' ';
for (at, c) in name.char_indices() {
let begins =
c.is_alphanumeric() && (start.is_none() || (prev.is_lowercase() && c.is_uppercase()));
if !c.is_alphanumeric() || begins {
if let Some(from) = start.take() {
out.push(&name[from..at]);
}
}
if begins {
start = Some(at);
}
prev = c;
}
if let Some(from) = start {
out.push(&name[from..]);
}
out
}
/// Where an address written in a member in `base` leads inside the archive.
///
/// The fragment comes off and percent-escapes are undone. An address with a
@@ -523,7 +630,13 @@ fn percent_decode(text: &str) -> Option<String> {
let mut at = 0;
while at < raw.len() {
if raw[at] == b'%' {
let hex = std::str::from_utf8(raw.get(at + 1..at + 3)?).ok()?;
// Two hex digits and nothing else: `from_str_radix` would also
// take a sign, and read `%+f` as 15.
let hex = raw.get(at + 1..at + 3)?;
if !hex.iter().all(u8::is_ascii_hexdigit) {
return None;
}
let hex = std::str::from_utf8(hex).ok()?;
out.push(u8::from_str_radix(hex, 16).ok()?);
at += 3;
} else {
@@ -878,6 +991,147 @@ mod tests {
assert_eq!(meta.cover.expect("a cover").bytes, JPEG);
}
#[test]
fn a_name_says_cover_only_as_a_word_of_its_own() {
for yes in [
"cover",
"CoverImage",
"Book_COVER",
"cover-thumb",
"frontCover",
"x.cover",
] {
assert!(says_cover(yes), "{yes}");
}
for no in [
"discover-ch1",
"recovery-map",
"covers",
"Coverage",
"uncover",
"",
] {
assert!(!says_cover(no), "{no}");
}
assert_eq!(words("CoverImage_v2"), ["Cover", "Image", "v2"]);
}
#[test]
fn a_name_that_only_contains_cover_is_not_a_cover() {
let package = opf(
r#"<item id="ch1" href="images/discover-ch1.png" media-type="image/png"/>
<item id="i2" href="images/cover.jpg" media-type="image/jpeg"/>"#,
"",
);
let meta = book(
&package,
&[
("OEBPS/images/discover-ch1.png", PNG),
("OEBPS/images/cover.jpg", JPEG),
],
);
let cover = meta.cover.expect("a cover");
assert_eq!(cover.bytes, JPEG);
assert!(cover.guessed);
let alone = opf(
r#"<item id="map" href="images/recovery-map.png" media-type="image/png"/>"#,
"",
);
let meta = book(&alone, &[("OEBPS/images/recovery-map.png", PNG)]);
assert_eq!(meta.cover, None);
assert!(meta.notes.is_empty(), "{:?}", meta.notes);
}
#[test]
fn an_image_named_cover_outright_beats_one_listed_before_it() {
let package = opf(
r#"<item id="cover-thumb" href="images/thumb.png" media-type="image/png"/>
<item id="i2" href="images/Cover.jpg" media-type="image/jpeg"/>"#,
"",
);
let meta = book(
&package,
&[
("OEBPS/images/thumb.png", PNG),
("OEBPS/images/Cover.jpg", JPEG),
],
);
let cover = meta.cover.expect("a cover");
assert_eq!(cover.bytes, JPEG);
assert!(cover.guessed);
}
#[test]
fn the_guide_cover_page_prefers_the_image_that_says_cover() {
let package = opf(
"",
r#"<guide><reference type="cover" href="text/cover.xhtml"/></guide>"#,
);
for art in [
r#"<img src="../images/front-cover.jpg"/>"#,
r#"<img alt="Cover" src="../images/front-cover.jpg"/>"#,
r#"<img id="coverImage" src="../images/front-cover.jpg"/>"#,
r#"<img class="full cover" src="../images/front-cover.jpg"/>"#,
] {
let page = format!(r#"<html><body><img src="../images/logo.png"/>{art}</body></html>"#);
let meta = book(
&package,
&[
("OEBPS/text/cover.xhtml", page.as_bytes()),
("OEBPS/images/logo.png", PNG),
("OEBPS/images/front-cover.jpg", JPEG),
],
);
assert_eq!(meta.cover.expect(art).bytes, JPEG, "{art}");
assert!(meta.notes.is_empty(), "{art}: {:?}", meta.notes);
}
}
#[test]
fn a_guide_cover_page_with_several_unnamed_images_takes_the_first_and_says_so() {
let package = opf(
"",
r#"<guide><reference type="cover" href="text/cover.xhtml"/></guide>"#,
);
let page = br#"<html><body><img src="../images/logo.png"/><img src="../images/art.jpg"/></body></html>"#;
let meta = book(
&package,
&[
("OEBPS/text/cover.xhtml", page),
("OEBPS/images/logo.png", PNG),
("OEBPS/images/art.jpg", JPEG),
],
);
assert_eq!(meta.cover.expect("a cover").bytes, PNG);
assert_eq!(
meta.notes,
["the cover page, OEBPS/text/cover.xhtml, shows 2 images, and the first was taken"]
);
}
#[test]
fn a_search_for_a_cover_reads_no_more_than_its_budget() {
let count = MAX_READS + 4;
let manifest: String = (0..count)
.map(|n| format!(r#"<item id="cover-{n}" href="c{n}.jpg" media-type="image/jpeg"/>"#))
.collect();
let names: Vec<String> = (0..count).map(|n| format!("OEBPS/c{n}.jpg")).collect();
let members: Vec<(&str, &[u8])> = names
.iter()
.map(|name| (name.as_str(), &b"<html>not an image</html>"[..]))
.collect();
let meta = book(&opf(&manifest, ""), &members);
assert_eq!(meta.cover, None);
let mut expected: Vec<String> = (0..MAX_READS)
.map(|n| format!("the cover, OEBPS/c{n}.jpg, is not an image"))
.collect();
expected.push(format!(
"the search for a cover stopped after reading {MAX_READS} members"
));
assert_eq!(meta.notes, expected);
}
#[test]
fn a_book_that_names_no_cover_has_none_and_says_nothing() {
let package = opf(
@@ -938,6 +1192,8 @@ mod tests {
archive: &mut archive,
items: &[],
tried: Vec::new(),
reads: 0,
spent: false,
meta: &mut meta,
};
let got = look.fetch("big.jpg", cap, "cover").map(|bytes| bytes.len());
@@ -1030,6 +1286,8 @@ mod tests {
Some("OEBPS/text/b.jpg")
);
assert_eq!(resolve("", "a%2Fb.jpg").as_deref(), Some("a/b.jpg"));
assert_eq!(percent_decode("a%2fb%2F").as_deref(), Some("a/b/"));
assert_eq!(percent_decode("a%+fb"), None);
for hostile in [
"../../c.jpg",
"../../../etc/passwd",
@@ -1038,6 +1296,9 @@ mod tests {
"file:///etc/passwd",
"%ZZ.jpg",
"%ff.jpg",
"a%+fb",
"a%-1b",
"a% fb",
"c%",
"#only-a-fragment",
"",