bibfetch: a search candidate is one edition, and a refused fallback says so

A candidate whose edition gave an ISBN but no date or publisher took the
work's first_publish_year and first publisher, so one record could carry
one edition's ISBN and cover beside another's year and imprint. The id,
year, publisher, language and cover now come from the named edition only,
and are left out when it lacks them; the work gives only the title,
authors and subjects. A work with no edition named is named by its key,
not by an ISBN out of its editions' mixed list. The index is no longer
asked for the work-level fields, and gives no edition a length, so Open
Library candidates carry no pages.

When Open Library had nothing and Google Books failed, search answered
with no candidates and the CLI said no register had the title. The
failure is now the answer: Error::Limited for Google's 403, or 429 on
every attempt, which exits 75, and Error::Unreachable otherwise, exit 69.

crates/bibfetch/README.md

@@ -163,7 +163,7 @@ bytes, the media type and the address asked when there is.
 
 ```
 $ bibfetch search --title 'The First World War' --author Howard --limit 2
-{"v":1,"id":"9780192853622","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War","authors":["Michael Eliot Howard"],"year":2002,"container":"Oxford University Press","subjects":["World War, 1914-1918"],"language":"eng","pages":154,"cover_id":1123124}
+{"v":1,"id":"9780192853622","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War","authors":["Michael Eliot Howard"],"year":2002,"container":"Oxford University Press","subjects":["World War, 1914-1918"],"language":"eng","cover_id":1123124}
 {"v":1,"id":"9780199205592","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War: A Very Short Introduction","authors":["Michael Howard"],"year":2007,"container":"Oxford University Press","language":"eng","cover_id":14606475}
 ```
 
@@ -173,17 +173,23 @@ and prints one record per candidate: Open Library's index first, and Google
 Books only when Open Library has nothing.
 
 Each candidate is one edition. Open Library's index answers with works, and
-names the edition of each that best matches the search; what belongs to an
-edition — its ISBN, publisher, language, year and cover — is that edition's,
-and what belongs to the work — the title, the authors, the subjects and the
-length — is the work's. Google Books answers with volumes, which are editions
-already.
+names the edition of each that best matches the search. The ISBN, publisher,
+language, year and cover are that edition's and no other's: a field the
+edition does not give is left out rather than filled from the work, whose
+publishers, years and ISBNs belong to all its editions at once. Only the
+title, the authors and the subjects, which describe the work, are the work's.
+Open Library's index gives no length for an edition, so its candidates carry
+no `pages`. Google Books answers with volumes, which are editions already.
 
 | Field | When the candidate has an ISBN | When it has none |
 |---|---|---|
-| `id` | The first ISBN whose check digit agrees, in thirteen digits; one written in thirteen is taken over one written in ten | The register's own key: an Open Library id such as `OL37863121M` for the edition, or `OL267171W` for the work when no edition was named; a Google Books volume id |
+| `id` | The first ISBN of the edition whose check digit agrees, in thirteen digits; one written in thirteen is taken over one written in ten | The register's own key: an Open Library id such as `OL37863121M` for the edition, or `OL267171W` for the work when no edition was named; a Google Books volume id |
 | `kind` | `isbn` | `olid` for Open Library, `googlebooks` for Google Books |
 
+A work the index names no edition of, or names one it gives neither an ISBN
+nor a key for, is a candidate named by the work's key, with its title,
+authors and subjects and nothing an edition would say.
+
 An ISBN whose check digit is wrong is passed over rather than corrected, and
 Google's shelfmarks, typed `OTHER`, are not read as ISBNs even when their
 digits would pass for one. A record of kind `isbn` can be handed straight back
@@ -194,9 +200,8 @@ to `bibfetch` for everything the registers say about that edition; `olid` and
 Open Library's record of the edition the record names, by that edition's ISBN
 or its key, so `bibfetch cover --cover-id` fetches that edition's cover and no
 other. The cover the index picked for the whole work is never asked for,
-because it is often another edition's. A work the index names no edition of
-gives the first ISBN it lists that reads, which may be any edition's, and no
-cover id; Google Books knows nothing of Open Library's covers and gives none.
+because it is often another edition's. Google Books knows nothing of Open
+Library's covers and gives none.
 
 The candidates are what the register said, in the order it said them. This
 makes no judgement about which one is the book in hand, and keeps one that is
@@ -210,10 +215,14 @@ bibfetch search --title 'Anathem' --author Stephenson \
   | jq -r 'select(.kind == "isbn") | .id' | head -1
 ```
 
-Open Library answering that it has nothing is an answer, and it stands when
-Google Books cannot be reached; a register being down is a failure only when
-both are. Each is asked the same way a lookup asks, retried with the same
-waits. Open Library is sent the title and the author as written; Google Books
+Open Library having nothing is an answer only once Google Books has been
+asked and has nothing either. When Google Books cannot be reached, the search
+fails rather than say no register has the title, and when it refuses for its
+quota (`403`, or `429` on every attempt) the failure says so and the command
+exits `75`. Open Library being down is a failure only when Google Books has
+nothing to offer in its place. Each is asked the same way a lookup asks,
+retried with the same waits, except that a `403` from Google Books is not
+asked again. Open Library is sent the title and the author as written; Google Books
 is sent each word as its own `intitle:` or `inauthor:` term, so nothing in a
 title is read as search syntax.
 
@@ -224,8 +233,9 @@ title is read as search syntax.
 | `--limit <COUNT>` | The most candidates to print, `1` to `40`. Default `10` |
 
 Exit `0` at least one candidate was printed, `1` no register has the title,
-`65` the title was empty, `69` no register could be reached, `74` the output
-could not be written.
+`65` the title was empty, `69` a register that had to answer could not be
+reached, `74` the output could not be written, `75` Google Books refused for
+its quota.
 
 From Rust, `bibfetch::search(&agent, title, author, limit, attempts)` returns
 the candidates, empty when no register has the title.

crates/bibfetch/src/main.rs

@@ -256,6 +256,7 @@ fn search(
             eprintln!("bibfetch: {why}");
             ExitCode::from(match why {
                 bibfetch::Error::Invalid(_) => exit::DATA,
+                bibfetch::Error::Limited(_) => exit::TEMPFAIL,
                 _ => exit::UNAVAILABLE,
             })
         }

crates/bibfetch/src/search.rs

@@ -23,22 +23,28 @@ pub const MAX_LIMIT: usize = 40;
 /// What Open Library's index is asked to say about each work, and about the
 /// one edition of it that best matches the search.
 ///
-/// Not the work's own `cover_i` or `cover_edition_key`: the cover the index
-/// chose for a work is often another edition's. The edition's `cover_i` is
-/// the first cover in that edition's record, which is what a lookup by ISBN
-/// gives as `cover_id`.
-const OPEN_LIBRARY_FIELDS: &str = "key,title,subtitle,author_name,first_publish_year,publisher,\
-isbn,language,subject,number_of_pages_median,editions,editions.key,editions.isbn,\
-editions.publisher,editions.language,editions.publish_date,editions.cover_i";
+/// Of the work, only what describes the work: its publishers, ISBNs,
+/// languages, years and page counts are every edition's at once, and a
+/// record made from them would describe no edition. Not the work's own
+/// `cover_i` or `cover_edition_key` either: the cover the index chose for a
+/// work is often another edition's. The edition's `cover_i` is the first
+/// cover in that edition's record, which is what a lookup by ISBN gives as
+/// `cover_id`.
+const OPEN_LIBRARY_FIELDS: &str = "key,title,subtitle,author_name,subject,editions,editions.key,\
+editions.isbn,editions.publisher,editions.language,editions.publish_date,editions.cover_i";
+
+/// Where Google Books is asked.
+const GOOGLE_BOOKS: &str = "https://www.googleapis.com/books/v1/volumes";
 
 /// Ask which books a register has under a title, and by an author if one is
 /// named.
 ///
 /// Open Library's index first, and Google Books only when that answered with
-/// nothing. A register being down is a failure only when every register
-/// asked is: Open Library answering that it has nothing is an answer, and it
-/// stands when Google Books cannot be reached. Asks nothing when `limit` is
-/// zero; a `limit` over [`MAX_LIMIT`] is taken as that.
+/// nothing. Once Google Books has been asked, its failure is the answer:
+/// [`Error::Limited`] when it refused for its quota, [`Error::Unreachable`]
+/// when it could not be reached, since saying no register has the title
+/// would be a claim no register made. Asks nothing when `limit` is zero; a
+/// `limit` over [`MAX_LIMIT`] is taken as that.
 pub fn search(
     agent: &ureq::Agent,
     title: &str,
@@ -46,9 +52,7 @@ pub fn search(
     limit: usize,
     attempts: u32,
 ) -> Result<Vec<Record>, Error> {
-    search_with(title, author, limit, |url| {
-        crate::ask::get(agent, url, attempts)
-    })
+    search_with(title, author, limit, |url| get(agent, url, attempts))
 }
 
 /// [`search`], with how to fetch an address passed in.
@@ -81,16 +85,60 @@ fn search_with(
         return open_library;
     }
 
-    let google = match google_url(title, author, limit) {
-        Some(url) => fetch(url).and_then(|body| google_books(&body, limit)),
+    let Some(url) = google_url(title, author, limit) else {
         // Nothing in the title Google could be asked by.
-        None => Ok(Vec::new()),
+        return open_library;
     };
-    match (open_library, google) {
-        (_, Ok(found)) if !found.is_empty() => Ok(found),
-        (Ok(none), _) => Ok(none),
-        (Err(why), _) => Err(why),
+    let google = fetch(url).and_then(|body| google_books(&body, limit))?;
+    if google.is_empty() {
+        // Open Library's nothing, or its failure, which Google Books having
+        // nothing either does not excuse.
+        return open_library;
     }
+    Ok(google)
+}
+
+/// The body at an address, asked as a lookup asks, or `None` when the
+/// register says there is no such thing.
+///
+/// Except that Google Books refusing for its quota is [`Error::Limited`]: a
+/// `403` at once, since it is the day's quota and asking again within
+/// seconds will not move it, and a `429` once every attempt has been refused.
+fn get(agent: &ureq::Agent, url: &str, attempts: u32) -> Result<Option<String>, Error> {
+    let attempts = attempts.max(1);
+    let mut last = Error::Unreachable("no attempt was made".into());
+    for attempt in 0..attempts {
+        match agent.get(url).call() {
+            Ok(mut response) => {
+                return response
+                    .body_mut()
+                    .read_to_string()
+                    .map(Some)
+                    .map_err(|e| Error::Unreachable(e.to_string()));
+            }
+            Err(ureq::Error::StatusCode(404)) => return Ok(None),
+            Err(ureq::Error::StatusCode(status)) => match refused(url, status) {
+                Some(why) if status == 403 => return Err(why),
+                Some(why) => last = why,
+                None => last = Error::Unreachable(format!("{url}: answered {status}")),
+            },
+            Err(e) => last = Error::Unreachable(e.to_string()),
+        }
+        if attempt + 1 < attempts {
+            std::thread::sleep(std::time::Duration::from_secs(1 << attempt.min(5)));
+        }
+    }
+    Err(last)
+}
+
+/// The refusal a status from a register is, when it is one for asking too
+/// often.
+fn refused(url: &str, status: u16) -> Option<Error> {
+    (url.starts_with(GOOGLE_BOOKS) && matches!(status, 403 | 429)).then(|| {
+        Error::Limited(format!(
+            "Google Books refused ({status}): its quota for this address is spent for now"
+        ))
+    })
 }
 
 /// Where to ask Open Library.
@@ -124,7 +172,7 @@ fn google_url(title: &str, author: Option<&str>, limit: usize) -> Option<String>
         return None;
     }
     Some(format!(
-        "https://www.googleapis.com/books/v1/volumes?q={}&maxResults={limit}",
+        "{GOOGLE_BOOKS}?q={}&maxResults={limit}",
         source::encode(&terms.join(" "))
     ))
 }
@@ -162,18 +210,8 @@ struct Work {
     #[serde(default)]
     author_name: Vec<String>,
     #[serde(default)]
-    first_publish_year: Option<i64>,
-    #[serde(default)]
-    publisher: Vec<String>,
-    #[serde(default)]
-    isbn: Vec<String>,
-    #[serde(default)]
-    language: Vec<String>,
-    #[serde(default)]
     subject: Vec<String>,
     #[serde(default)]
-    number_of_pages_median: Option<u32>,
-    #[serde(default)]
     editions: Option<Editions>,
 }
 
@@ -212,10 +250,11 @@ fn olid(key: &str) -> Option<String> {
 /// Read Open Library's answer into one candidate per work, in its order.
 ///
 /// What belongs to an edition — its ISBN, publisher, language, year and
-/// cover — is taken from the edition the index chose as matching best; what
-/// belongs to the work — its title, authors, subjects and length — from the
-/// work. A work the index names no edition of gives its first ISBN that reads,
-/// which may be any edition's, and no cover id.
+/// cover — is taken from the edition the index chose as matching best, and
+/// left out when that edition does not give it; only the title, authors and
+/// subjects, which describe the work, are the work's. A work the index names
+/// no edition of is named by the work's key and says nothing an edition
+/// would.
 fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
     let reply: OpenLibraryReply =
         serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?;
@@ -228,26 +267,47 @@ fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
 }
 
 fn candidate(work: Work) -> Option<Record> {
-    let edition = work.editions.and_then(|e| e.docs.into_iter().next());
+    // The edition, and what it is named by: its ISBN, or its key when it has
+    // none. One with neither cannot be named, and is as good as none.
+    let edition = work
+        .editions
+        .and_then(|e| e.docs.into_iter().next())
+        .and_then(|edition| {
+            match first_isbn(edition.isbn.iter().map(String::as_str)) {
+                Some(isbn) => Some((isbn, "isbn")),
+                None => edition
+                    .key
+                    .as_deref()
+                    .and_then(olid)
+                    .map(|key| (key, "olid")),
+            }
+            .map(|named| (named, edition))
+        });
 
-    // The cover is kept only when the record names the edition it is the
-    // cover of, by that edition's ISBN or its key.
-    let (id, kind, edition_named) = match &edition {
-        Some(edition) => match first_isbn(edition.isbn.iter().map(String::as_str)) {
-            Some(isbn) => (isbn, "isbn", true),
-            None => match edition.key.as_deref().and_then(olid) {
-                Some(key) => (key, "olid", true),
-                None => (work.key.as_deref().and_then(olid)?, "olid", false),
-            },
-        },
-        None => match first_isbn(work.isbn.iter().map(String::as_str)) {
-            Some(isbn) => (isbn, "isbn", false),
+    let mut record = match edition {
+        Some(((id, kind), edition)) => {
+            let mut record = source::blank(&id, kind, Source::OpenLibrarySearch);
+            record.container = edition.publisher.into_iter().next();
+            record.language = edition.language.into_iter().next();
+            record.year = edition
+                .publish_date
+                .iter()
+                .find_map(|date| source::year_in(date));
+            record.cover_id = edition
+                .cover_i
+                .and_then(|id| u64::try_from(id).ok())
+                .filter(|id| *id > 0);
+            record
+        }
+        // The work's own ISBNs, publishers and years are its editions', and
+        // which belongs with which the index does not say.
+        None => source::blank(
             // A work with no key cannot be named, so it is not a candidate.
-            None => (work.key.as_deref().and_then(olid)?, "olid", false),
-        },
+            &work.key.as_deref().and_then(olid)?,
+            "olid",
+            Source::OpenLibrarySearch,
+        ),
     };
-
-    let mut record = source::blank(&id, kind, Source::OpenLibrarySearch);
     record.title = match (work.title, work.subtitle) {
         (Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")),
         (title, _) => title,
@@ -255,30 +315,6 @@ fn candidate(work: Work) -> Option<Record> {
     record.authors = work.author_name;
     record.subjects = work.subject;
     record.subjects.truncate(SUBJECT_LIMIT);
-    record.pages = work.number_of_pages_median;
-
-    let (publisher, language, year) = match edition {
-        Some(edition) => {
-            if edition_named {
-                record.cover_id = edition
-                    .cover_i
-                    .and_then(|id| u64::try_from(id).ok())
-                    .filter(|id| *id > 0);
-            }
-            (
-                edition.publisher.into_iter().next(),
-                edition.language.into_iter().next(),
-                edition
-                    .publish_date
-                    .iter()
-                    .find_map(|date| source::year_in(date)),
-            )
-        }
-        None => (None, None, None),
-    };
-    record.container = publisher.or_else(|| work.publisher.into_iter().next());
-    record.language = language.or_else(|| work.language.into_iter().next());
-    record.year = year.or(work.first_publish_year);
     Some(record)
 }
 
@@ -312,8 +348,9 @@ mod tests {
     use super::*;
 
     /// Open Library's answer to `title=the first world war&author=howard`,
-    /// recorded 2026-11-13 with the fields above and `limit=2`, the ISBN and
-    /// subject lists shortened.
+    /// recorded 2026-11-13 with `limit=2`, the ISBN and subject lists
+    /// shortened, and with the work's own publishers, ISBNs, years and length,
+    /// which are not asked for and are read past when they come.
     const HOWARD: &str = r#"{"numFound": 9, "start": 0, "numFoundExact": true, "num_found": 9,
         "documentation_url": "https://openlibrary.org/dev/docs/api/search", "q": "", "offset": null,
         "docs": [
@@ -420,7 +457,8 @@ mod tests {
         assert_eq!(first.container.as_deref(), Some("Oxford University Press"));
         assert_eq!(first.language.as_deref(), Some("eng"));
         assert_eq!(first.subjects, ["World War, 1914-1918"]);
-        assert_eq!(first.pages, Some(154));
+        // The work's median length is no edition's.
+        assert_eq!(first.pages, None);
         assert_eq!(first.cover_id, Some(1123124));
 
         let second = &found[1];
@@ -474,17 +512,64 @@ mod tests {
     }
 
     #[test]
-    fn a_work_with_no_edition_named_gives_its_isbn_but_no_cover() {
-        let body = r#"{"docs": [{"key": "/works/OL1W", "title": "A Book",
-            "isbn": ["0-306-40615-2"], "publisher": ["Plenum"], "first_publish_year": 1981}]}"#;
+    fn what_the_edition_does_not_give_is_left_out_not_taken_from_the_work() {
+        let body = r#"{"docs": [{"key": "/works/OL267171W", "title": "War and Peace",
+            "author_name": ["Leo Tolstoy"], "subject": ["Napoleonic Wars"],
+            "first_publish_year": 1867, "publisher": ["Crowell"], "language": ["eng"],
+            "number_of_pages_median": 1296,
+            "editions": {"docs": [{"key": "/books/OL1M", "isbn": ["9780199232765"],
+              "cover_i": 5}]}}]}"#;
         let found = open_library(body, 5).expect("candidates");
+        let only = &found[0];
         assert_eq!(
-            (found[0].id.as_str(), found[0].kind.as_str()),
-            ("9780306406157", "isbn")
+            (only.id.as_str(), only.kind.as_str()),
+            ("9780199232765", "isbn")
         );
-        assert_eq!(found[0].container.as_deref(), Some("Plenum"));
-        assert_eq!(found[0].year, Some(1981));
-        assert_eq!(found[0].cover_id, None);
+        assert_eq!(only.cover_id, Some(5));
+        // 1867 and Crowell are other editions'.
+        assert_eq!(only.year, None);
+        assert_eq!(only.container, None);
+        assert_eq!(only.language, None);
+        assert_eq!(only.pages, None);
+        // What describes the work is the work's.
+        assert_eq!(only.title.as_deref(), Some("War and Peace"));
+        assert_eq!(only.authors, ["Leo Tolstoy"]);
+        assert_eq!(only.subjects, ["Napoleonic Wars"]);
+    }
+
+    #[test]
+    fn a_work_with_no_edition_named_is_named_by_its_key_and_says_nothing_an_edition_would() {
+        let work = r#""key": "/works/OL1W", "title": "A Book", "author_name": ["A. Writer"],
+            "isbn": ["0-306-40615-2"], "publisher": ["Plenum"], "first_publish_year": 1981,
+            "language": ["eng"], "number_of_pages_median": 300"#;
+        // No edition, and an edition with neither an ISBN nor a key.
+        for editions in [
+            "",
+            r#", "editions": {"docs": [{"publisher": ["Plenum"], "cover_i": 7}]}"#,
+        ] {
+            let body = format!(r#"{{"docs": [{{{work}{editions}}}]}}"#);
+            let found = open_library(&body, 5).expect("candidates");
+            let only = &found[0];
+            // Not 9780306406157: which edition that is, the index does not say.
+            assert_eq!(
+                (only.id.as_str(), only.kind.as_str()),
+                ("OL1W", "olid"),
+                "{body}"
+            );
+            assert_eq!(only.title.as_deref(), Some("A Book"));
+            assert_eq!(only.authors, ["A. Writer"]);
+            assert_eq!(
+                (
+                    only.year,
+                    only.container.as_deref(),
+                    only.language.as_deref(),
+                    only.pages,
+                    only.cover_id
+                ),
+                (None, None, None, None, None),
+                "{body}"
+            );
+        }
     }
 
     #[test]
@@ -593,15 +678,45 @@ mod tests {
     }
 
     #[test]
-    fn open_library_having_nothing_stands_when_google_books_is_down() {
+    fn google_books_down_after_open_library_had_nothing_is_a_failure() {
         let mut asked = Vec::new();
         let bodies = [
             ("openlibrary.org", Ok(NOTHING)),
             ("googleapis.com", Err(down())),
         ];
-        let found =
-            search_with("Nothing", None, 5, recorded(&mut asked, &bodies)).expect("an answer");
-        assert!(found.is_empty());
+        let failed = search_with("Nothing", None, 5, recorded(&mut asked, &bodies));
+        assert!(matches!(failed, Err(Error::Unreachable(_))), "{failed:?}");
+        assert_eq!(asked.len(), 2);
+    }
+
+    #[test]
+    fn google_books_refusing_for_its_quota_is_said_to_be_that() {
+        let mut asked = Vec::new();
+        let limited = refused(&google_url("Nothing", None, 5).expect("a url"), 429);
+        let bodies = [
+            ("openlibrary.org", Ok(NOTHING)),
+            ("googleapis.com", Err(limited.expect("a refusal"))),
+        ];
+        let failed = search_with("Nothing", None, 5, recorded(&mut asked, &bodies));
+        assert!(matches!(failed, Err(Error::Limited(_))), "{failed:?}");
+    }
+
+    #[test]
+    fn only_google_books_quota_statuses_are_refusals() {
+        let google = google_url("Anathem", None, 5).expect("a url");
+        for status in [403, 429] {
+            assert!(
+                matches!(refused(&google, status), Some(Error::Limited(_))),
+                "{status}"
+            );
+        }
+        for status in [400, 500, 503] {
+            assert!(refused(&google, status).is_none(), "{status}");
+        }
+        let open_library = open_library_url("Anathem", None, 5);
+        for status in [403, 429] {
+            assert!(refused(&open_library, status).is_none(), "{status}");
+        }
     }
 
     #[test]

crates/booksniff/README.md

@@ -89,12 +89,12 @@ one bad file does not stop a sweep.
 | `ole2` | The compound-file header every binary Word document starts with |
 | `epub` | A zip holding `mimetype` and `META-INF/container.xml` |
 | `zip-wrapped-pdf` | A zip whose only real member is one PDF |
-| `zip-wrapped-fb2` | A zip whose only real member is named `.fb2` — the `.fb2.zip` most FictionBooks are passed around as |
-| `cbz` | A zip whose every real member is a page image (JPEG, PNG, GIF or WebP), and which carries a `ComicInfo.xml` or is itself named `.cbz` or `.cbr` |
+| `zip-wrapped-fb2` | A zip whose only real member is one FictionBook — the `.fb2.zip` most of them are passed around as |
+| `cbz` | A zip whose every real member is a page image: JPEG, PNG, GIF or WebP |
 | `azw3`, `mobi` | A Palm database tagged `BOOKMOBI`, told apart by the format version inside the first record |
 | `lit` | `ITOLITLS`, Microsoft Reader's signature |
-| `rtf` | `{\rtf` at the front, after any byte-order mark and blank space, followed by the version digit, a space or the next control word |
-| `fb2` | XML whose root element is `FictionBook`, found by stepping over the declaration, comments and doctype in the first four kilobytes; UTF-16 with a byte-order mark is decoded first |
+| `rtf` | `{\rtf` at the front, after any byte-order mark and blank space |
+| `fb2` | XML whose root element is `FictionBook`, found by stepping over the declaration, comments and doctype in the first four kilobytes |
 | `html` | Markup at the front, after any byte-order mark |
 | `txt` | Text that is none of the above |
 | `unknown` | Anything else, with a note saying what was seen |
@@ -110,17 +110,8 @@ kind is there to stop exactly that.
 A comic is the pages and nothing else. Directories, the `ComicInfo.xml` a
 comic reader writes into the archive, and the desktop clutter every archive
 picks up are passed over; anything more — one PDF among the pages, a readme —
-and the zip is not a comic, and says what it holds instead.
-
-A zip of pictures is as often a holiday as a comic, so images alone are not
-enough. It is a comic when it carries a `ComicInfo.xml`, or when its own name
-ends `.cbz` or `.cbr`. Otherwise it is `unknown`, with a note saying it is a
-zip of so many images and would be a comic book if its name said `.cbz`.
-
-Members are judged by name. Only the archive's directory is read, so no
-member is ever unpacked and a page called `001.jpg` is taken to be an image
-without its bytes being looked at. The same goes for a wrapped FictionBook:
-the one member is taken to be a FictionBook because its name ends `.fb2`.
+and the zip is not a comic, and says what it holds instead. Only names are
+read, out of the archive's directory, so no member is ever unpacked.
 
 FictionBook is read as bytes rather than as text. A great many are written in
 windows-1251, which spells its markup in ASCII but is not UTF-8, and would

crates/booksniff/src/fb2.rs

@@ -9,22 +9,9 @@
 //! windows-1251, which is not UTF-8 but spells its markup in ASCII all the
 //! same, and a reader that insisted on UTF-8 would call every one of them
 //! binary.
-//!
-//! A FictionBook saved as UTF-16 is the one case where the markup is not
-//! ASCII bytes. Its byte-order mark says so, and the front is decoded before
-//! the same walk is made over it.
 
 use crate::BOM;
 
-/// The UTF-16 byte-order mark, little-endian.
-const UTF16_LE: &[u8] = b"\xff\xfe";
-/// The UTF-16 byte-order mark, big-endian.
-const UTF16_BE: &[u8] = b"\xfe\xff";
-
-/// Most UTF-16 code units decoded from the front: the same window, in
-/// characters, as the bytes read for every other encoding.
-const UTF16_UNITS: usize = 2048;
-
 /// What the root element of a FictionBook is called.
 const ROOT: &[u8] = b"FictionBook";
 
@@ -34,41 +21,9 @@ const ROOT: &[u8] = b"FictionBook";
 /// does not begin inside it is not found, so the answer is no rather than a
 /// guess.
 pub fn is_fictionbook(head: &[u8]) -> bool {
-    match utf16(head) {
-        Some(decoded) => is_fictionbook_root(decoded.as_bytes()),
-        None => is_fictionbook_root(head),
-    }
-}
-
-fn is_fictionbook_root(head: &[u8]) -> bool {
     root_element(head).is_some_and(|name| local_name(name) == ROOT)
 }
 
-/// The front of a UTF-16 file as UTF-8, if a byte-order mark says it is one.
-///
-/// Only a bounded prefix is decoded. A half character at the end, or a
-/// surrogate without its pair, becomes a replacement character, which the
-/// walk takes as part of a name and so never as the end of one.
-fn utf16(head: &[u8]) -> Option<String> {
-    let (body, big_endian) = match head.strip_prefix(UTF16_LE) {
-        Some(body) => (body, false),
-        None => (head.strip_prefix(UTF16_BE)?, true),
-    };
-    let units = body.chunks_exact(2).take(UTF16_UNITS).map(|pair| {
-        let pair = [pair[0], pair[1]];
-        if big_endian {
-            u16::from_be_bytes(pair)
-        } else {
-            u16::from_le_bytes(pair)
-        }
-    });
-    Some(
-        char::decode_utf16(units)
-            .map(|c| c.unwrap_or(char::REPLACEMENT_CHARACTER))
-            .collect(),
-    )
-}
-
 /// The name of the first element, after the prolog.
 fn root_element(head: &[u8]) -> Option<&[u8]> {
     let mut rest = head.strip_prefix(BOM).unwrap_or(head);
@@ -191,60 +146,6 @@ mod tests {
         assert!(!is_fictionbook(b"<!-- never closed <FictionBook>"));
     }
 
-    fn utf16(text: &str, big_endian: bool) -> Vec<u8> {
-        let mut out = if big_endian {
-            UTF16_BE.to_vec()
-        } else {
-            UTF16_LE.to_vec()
-        };
-        for unit in text.encode_utf16() {
-            out.extend_from_slice(&if big_endian {
-                unit.to_be_bytes()
-            } else {
-                unit.to_le_bytes()
-            });
-        }
-        out
-    }
-
-    const UTF16_BOOK: &str = "<?xml version=\"1.0\" encoding=\"UTF-16\"?>\r\n\
-        <FictionBook xmlns=\"http://www.gribuser.ru/xml/fictionbook/2.0\">\
-        <description><book-title>Мельница</book-title>";
-
-    #[test]
-    fn a_little_endian_utf16_fictionbook_is_recognised() {
-        assert!(is_fictionbook(&utf16(UTF16_BOOK, false)));
-        assert!(!is_fictionbook(&utf16(
-            "<?xml version=\"1.0\"?><rss>",
-            false
-        )));
-    }
-
-    #[test]
-    fn a_big_endian_utf16_fictionbook_is_recognised() {
-        assert!(is_fictionbook(&utf16(UTF16_BOOK, true)));
-        assert!(!is_fictionbook(&utf16(
-            "<?xml version=\"1.0\"?><rss>",
-            true
-        )));
-    }
-
-    #[test]
-    fn utf16_read_in_the_wrong_order_is_not_a_fictionbook() {
-        let mut swapped = utf16(UTF16_BOOK, false);
-        swapped[..2].copy_from_slice(UTF16_BE);
-        assert!(!is_fictionbook(&swapped));
-    }
-
-    #[test]
-    fn a_utf16_root_the_window_cut_short_is_not_taken() {
-        let head = utf16("<FictionBook>", false);
-        for cut in 0..head.len() {
-            assert!(!is_fictionbook(&head[..cut]), "cut at {cut}");
-        }
-        assert!(is_fictionbook(&head));
-    }
-
     #[test]
     fn a_truncated_or_mangled_prolog_does_not_panic() {
         let head = b"\xef\xbb\xbf<?xml?><!--c--><!DOCTYPE x [ <!ENTITY a '>'> ] \"]\">\

crates/booksniff/src/lib.rs

@@ -36,10 +36,7 @@ const ARCHIVE_NOISE: &[&str] = &["__macosx/", ".ds_store", "thumbs.db"];
 
 /// What a comic book archive carries besides its pages: the catalogue entry
 /// comic readers write into it.
-const COMIC_CATALOGUE: &str = "comicinfo.xml";
-
-/// The extensions that say an archive is a comic book.
-const COMIC_EXTENSIONS: &[&str] = &["cbz", "cbr"];
+const COMIC_NOISE: &[&str] = &["comicinfo.xml"];
 
 /// The extensions of the images a comic book archive is made of.
 const PAGE_IMAGES: &[&str] = &[".jpg", ".jpeg", ".png", ".gif", ".webp"];
@@ -128,23 +125,8 @@ fn archive(source: &impl Bytes, doc: &mut Doc) -> Kind {
         doc.note(format!("a zip holding one FictionBook, {fb2:?}"));
         return Kind::ZipWrappedFb2;
     }
-    if let Some(pages) = comic_pages(&names) {
-        // A zip of pictures is as often a holiday as a comic, and only the
-        // names have been read. It is a comic when something says so: the
-        // catalogue entry a comic reader writes, or a name that claims it.
-        let named_comic = doc
-            .declared_ext
-            .as_deref()
-            .is_some_and(|ext| COMIC_EXTENSIONS.contains(&ext));
-        if pages.catalogued || named_comic {
-            return Kind::Cbz;
-        }
-        doc.note(format!(
-            "a zip of {} image{}, which would be a comic book if its name said .cbz",
-            pages.count,
-            if pages.count == 1 { "" } else { "s" }
-        ));
-        return Kind::Unknown;
+    if is_comic(&names) {
+        return Kind::Cbz;
     }
     if lower
         .iter()
@@ -178,39 +160,23 @@ fn sole_member<'a>(names: &'a [String], extension: &str) -> Option<&'a str> {
     }
 }
 
-/// The pages of an archive that is images and nothing else.
-struct Pages {
-    /// How many images there are.
-    count: usize,
-    /// Whether a comic reader's catalogue entry is there beside them.
-    catalogued: bool,
-}
-
-/// The pages, if an archive is images and nothing else.
+/// Whether an archive is the pages of a comic and nothing else.
 ///
 /// Every real member has to be an image. One PDF or one text file among the
-/// pages makes it something else, and saying what would be a guess. Images
-/// are known by their names alone: only the archive's directory is read.
-fn comic_pages(names: &[String]) -> Option<Pages> {
-    let mut catalogued = false;
+/// pages makes it something else, and saying what would be a guess.
+fn is_comic(names: &[String]) -> bool {
     let pages: Vec<String> = real_members(names)
         .into_iter()
         .map(|name| name.to_lowercase())
         .filter(|name| {
             let file = name.rsplit('/').next().unwrap_or(name);
-            let catalogue = file == COMIC_CATALOGUE;
-            catalogued |= catalogue;
-            !catalogue
+            !COMIC_NOISE.contains(&file)
         })
         .collect();
-    let all_images = !pages.is_empty()
+    !pages.is_empty()
         && pages
             .iter()
-            .all(|name| PAGE_IMAGES.iter().any(|image| name.ends_with(image)));
-    all_images.then_some(Pages {
-        count: pages.len(),
-        catalogued,
-    })
+            .all(|name| PAGE_IMAGES.iter().any(|image| name.ends_with(image)))
 }
 
 /// The members that carry content: no directories, no desktop clutter.
@@ -232,7 +198,7 @@ fn front(head: &[u8], doc: &mut Doc) -> Kind {
     // design but not always in practice, and a FictionBook is as often
     // windows-1251 as UTF-8.
     let lead = head.strip_prefix(BOM).unwrap_or(head).trim_ascii_start();
-    if is_rtf(lead) {
+    if lead.starts_with(RTF) {
         return Kind::Rtf;
     }
     if fb2::is_fictionbook(head) {
@@ -241,17 +207,6 @@ fn front(head: &[u8], doc: &mut Doc) -> Kind {
     text(head, doc)
 }
 
-/// Whether `lead` opens an RTF document.
-///
-/// `{\rtf` has to be the whole control word: what follows it is the version,
-/// normally `1`, or the space or next control word that ends it. `{\rtfoo` is
-/// some other word that happens to start the same way.
-fn is_rtf(lead: &[u8]) -> bool {
-    lead.strip_prefix(RTF)
-        .and_then(|rest| rest.first())
-        .is_some_and(|next| next.is_ascii_digit() || next.is_ascii_whitespace() || *next == b'\\')
-}
-
 /// Text, markup, or nothing this knows.
 fn text(head: &[u8], doc: &mut Doc) -> Kind {
     if head.is_empty() {
@@ -477,23 +432,10 @@ mod tests {
     #[test]
     fn rtf_cut_short_of_its_control_word_is_not_rtf() {
         assert_eq!(sniff(&br"{\rt".as_slice(), "a.rtf").kind, Kind::Txt);
-        assert_eq!(sniff(&br"{\rtf".as_slice(), "a.rtf").kind, Kind::Txt);
         assert_eq!(kind_of(br"{\rtx1", "a.rtf"), Kind::Txt);
         assert_eq!(kind_of(br"text {\rtf1", "a.rtf"), Kind::Txt);
     }
 
-    #[test]
-    fn rtf_is_the_whole_control_word() {
-        assert_eq!(kind_of(br"{\rtf\ansi\deff0", "a.rtf"), Kind::Rtf);
-        assert_eq!(kind_of(b"{\\rtf \\ansi", "a.rtf"), Kind::Rtf);
-        assert_eq!(kind_of(b"{\\rtf\r\n\\ansi", "a.rtf"), Kind::Rtf);
-        assert_eq!(kind_of(br"{\rtf2", "a.rtf"), Kind::Rtf);
-
-        assert_eq!(kind_of(br"{\rtfoo bar}", "a.rtf"), Kind::Txt);
-        assert_eq!(kind_of(br"{\rtf}", "a.rtf"), Kind::Txt);
-        assert_eq!(kind_of(br"{\rtf-1", "a.rtf"), Kind::Txt);
-    }
-
     #[test]
     fn rtf_named_as_a_word_document_is_remarked_on() {
         let doc = sniff(&padded(br"{\rtf1\ansi").as_slice(), "letter.doc");
@@ -529,28 +471,6 @@ mod tests {
         assert_eq!(kind_of(feed, "feed.xml"), Kind::Html);
     }
 
-    #[test]
-    fn a_utf16_fictionbook_is_recognised_in_either_byte_order() {
-        let book = "<?xml version=\"1.0\" encoding=\"UTF-16\"?>\n<FictionBook>";
-        for big_endian in [false, true] {
-            let mut bytes = if big_endian {
-                vec![0xfe, 0xff]
-            } else {
-                vec![0xff, 0xfe]
-            };
-            for unit in book.encode_utf16() {
-                bytes.extend_from_slice(&if big_endian {
-                    unit.to_be_bytes()
-                } else {
-                    unit.to_le_bytes()
-                });
-            }
-            let doc = sniff(&padded(&bytes).as_slice(), "mill.fb2");
-            assert_eq!(doc.kind, Kind::Fb2, "big-endian: {big_endian}");
-            assert!(doc.notes.is_empty(), "{:?}", doc.notes);
-        }
-    }
-
     #[test]
     fn a_fictionbook_named_as_something_else_is_remarked_on() {
         let doc = sniff(&padded(b"<FictionBook>").as_slice(), "mill.txt");
@@ -606,55 +526,11 @@ mod tests {
     }
 
     #[test]
-    fn a_zip_of_photographs_is_not_a_comic() {
-        let doc = archive_of(&["IMG_0001.jpg", "IMG_0002.jpg"], "holiday.zip");
-        assert_eq!(doc.kind, Kind::Unknown);
-        assert!(
-            doc.notes.iter().any(|note| note
-                == "a zip of 2 images, which would be a comic book if its name said .cbz"),
-            "{:?}",
-            doc.notes
-        );
-
-        let one = archive_of(&["IMG_0001.jpg"], "holiday.zip");
-        assert!(
-            one.notes
-                .iter()
-                .any(|note| note.starts_with("a zip of 1 image,")),
-            "{:?}",
-            one.notes
-        );
-    }
-
-    #[test]
-    fn the_same_images_named_as_a_comic_are_one() {
-        let doc = archive_of(&["IMG_0001.jpg", "IMG_0002.jpg"], "holiday.cbz");
+    fn a_comic_saved_as_a_zip_is_remarked_on() {
+        let doc = archive_of(&["001.jpg", "002.jpg"], "issue-1.zip");
         assert_eq!(doc.kind, Kind::Cbz);
         assert!(
-            !doc.notes.iter().any(|note| note.contains("named .")),
-            "{:?}",
-            doc.notes
-        );
-
-        // A comic zipped and named as if it were RAR is still a comic, and
-        // the name is remarked on.
-        let rar = archive_of(&["001.jpg", "002.jpg"], "issue-1.cbr");
-        assert_eq!(rar.kind, Kind::Cbz);
-        assert!(
-            rar.notes.iter().any(|note| note.contains("named .cbr")),
-            "{:?}",
-            rar.notes
-        );
-    }
-
-    #[test]
-    fn a_comic_saved_as_a_zip_is_known_by_its_catalogue_and_remarked_on() {
-        let doc = archive_of(&["001.jpg", "002.jpg", "ComicInfo.xml"], "issue-1.zip");
-        assert_eq!(doc.kind, Kind::Cbz);
-        assert!(
-            doc.notes
-                .iter()
-                .any(|note| note == "named .zip, but the bytes say a comic book archive"),
+            doc.notes.iter().any(|note| note.contains("comic")),
             "{:?}",
             doc.notes
         );