Merge remote-tracking branch 'origin/fixes-040-search' into fixes-040

crates/bibfetch/README.md

@@ -163,7 +163,7 @@ bytes, the media type and the address asked when there is.
 
 ```
 $ bibfetch search --title 'The First World War' --author Howard --limit 2
-{"v":1,"id":"9780192853622","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War","authors":["Michael Eliot Howard"],"year":2002,"container":"Oxford University Press","subjects":["World War, 1914-1918"],"language":"eng","pages":154,"cover_id":1123124}
+{"v":1,"id":"9780192853622","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War","authors":["Michael Eliot Howard"],"year":2002,"container":"Oxford University Press","subjects":["World War, 1914-1918"],"language":"eng","cover_id":1123124}
 {"v":1,"id":"9780199205592","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War: A Very Short Introduction","authors":["Michael Howard"],"year":2007,"container":"Oxford University Press","language":"eng","cover_id":14606475}
 ```
 
@@ -173,17 +173,23 @@ and prints one record per candidate: Open Library's index first, and Google
 Books only when Open Library has nothing.
 
 Each candidate is one edition. Open Library's index answers with works, and
-names the edition of each that best matches the search; what belongs to an
+names the edition of each that best matches the search. The ISBN, publisher,
-edition — its ISBN, publisher, language, year and cover — is that edition's,
+language, year and cover are that edition's and no other's: a field the
-and what belongs to the work — the title, the authors, the subjects and the
+edition does not give is left out rather than filled from the work, whose
-length — is the work's. Google Books answers with volumes, which are editions
+publishers, years and ISBNs belong to all its editions at once. Only the
-already.
+title, the authors and the subjects, which describe the work, are the work's.
+Open Library's index gives no length for an edition, so its candidates carry
+no `pages`. Google Books answers with volumes, which are editions already.
 
 | Field | When the candidate has an ISBN | When it has none |
 |---|---|---|
-| `id` | The first ISBN whose check digit agrees, in thirteen digits; one written in thirteen is taken over one written in ten | The register's own key: an Open Library id such as `OL37863121M` for the edition, or `OL267171W` for the work when no edition was named; a Google Books volume id |
+| `id` | The first ISBN of the edition whose check digit agrees, in thirteen digits; one written in thirteen is taken over one written in ten | The register's own key: an Open Library id such as `OL37863121M` for the edition, or `OL267171W` for the work when no edition was named; a Google Books volume id |
 | `kind` | `isbn` | `olid` for Open Library, `googlebooks` for Google Books |
 
+A work the index names no edition of, or names one it gives neither an ISBN
+nor a key for, is a candidate named by the work's key, with its title,
+authors and subjects and nothing an edition would say.
+
 An ISBN whose check digit is wrong is passed over rather than corrected, and
 Google's shelfmarks, typed `OTHER`, are not read as ISBNs even when their
 digits would pass for one. A record of kind `isbn` can be handed straight back
@@ -194,9 +200,8 @@ to `bibfetch` for everything the registers say about that edition; `olid` and
 Open Library's record of the edition the record names, by that edition's ISBN
 or its key, so `bibfetch cover --cover-id` fetches that edition's cover and no
 other. The cover the index picked for the whole work is never asked for,
-because it is often another edition's. A work the index names no edition of
+because it is often another edition's. Google Books knows nothing of Open
-gives the first ISBN it lists that reads, which may be any edition's, and no
+Library's covers and gives none.
-cover id; Google Books knows nothing of Open Library's covers and gives none.
 
 The candidates are what the register said, in the order it said them. This
 makes no judgement about which one is the book in hand, and keeps one that is
@@ -210,10 +215,14 @@ bibfetch search --title 'Anathem' --author Stephenson \
   | jq -r 'select(.kind == "isbn") | .id' | head -1
 ```
 
-Open Library answering that it has nothing is an answer, and it stands when
+Open Library having nothing is an answer only once Google Books has been
-Google Books cannot be reached; a register being down is a failure only when
+asked and has nothing either. When Google Books cannot be reached, the search
-both are. Each is asked the same way a lookup asks, retried with the same
+fails rather than say no register has the title, and when it refuses for its
-waits. Open Library is sent the title and the author as written; Google Books
+quota (`403`, or `429` on every attempt) the failure says so and the command
+exits `75`. Open Library being down is a failure only when Google Books has
+nothing to offer in its place. Each is asked the same way a lookup asks,
+retried with the same waits, except that a `403` from Google Books is not
+asked again. Open Library is sent the title and the author as written; Google Books
 is sent each word as its own `intitle:` or `inauthor:` term, so nothing in a
 title is read as search syntax.
 
@@ -224,8 +233,9 @@ title is read as search syntax.
 | `--limit <COUNT>` | The most candidates to print, `1` to `40`. Default `10` |
 
 Exit `0` at least one candidate was printed, `1` no register has the title,
-`65` the title was empty, `69` no register could be reached, `74` the output
+`65` the title was empty, `69` a register that had to answer could not be
-could not be written.
+reached, `74` the output could not be written, `75` Google Books refused for
+its quota.
 
 From Rust, `bibfetch::search(&agent, title, author, limit, attempts)` returns
 the candidates, empty when no register has the title.

crates/bibfetch/src/main.rs

@@ -256,6 +256,7 @@ fn search(
             eprintln!("bibfetch: {why}");
             ExitCode::from(match why {
                 bibfetch::Error::Invalid(_) => exit::DATA,
+                bibfetch::Error::Limited(_) => exit::TEMPFAIL,
                 _ => exit::UNAVAILABLE,
             })
         }

crates/bibfetch/src/search.rs

@@ -23,22 +23,28 @@ pub const MAX_LIMIT: usize = 40;
 /// What Open Library's index is asked to say about each work, and about the
 /// one edition of it that best matches the search.
 ///
-/// Not the work's own `cover_i` or `cover_edition_key`: the cover the index
+/// Of the work, only what describes the work: its publishers, ISBNs,
-/// chose for a work is often another edition's. The edition's `cover_i` is
+/// languages, years and page counts are every edition's at once, and a
-/// the first cover in that edition's record, which is what a lookup by ISBN
+/// record made from them would describe no edition. Not the work's own
-/// gives as `cover_id`.
+/// `cover_i` or `cover_edition_key` either: the cover the index chose for a
-const OPEN_LIBRARY_FIELDS: &str = "key,title,subtitle,author_name,first_publish_year,publisher,\
+/// work is often another edition's. The edition's `cover_i` is the first
-isbn,language,subject,number_of_pages_median,editions,editions.key,editions.isbn,\
+/// cover in that edition's record, which is what a lookup by ISBN gives as
-editions.publisher,editions.language,editions.publish_date,editions.cover_i";
+/// `cover_id`.
+const OPEN_LIBRARY_FIELDS: &str = "key,title,subtitle,author_name,subject,editions,editions.key,\
+editions.isbn,editions.publisher,editions.language,editions.publish_date,editions.cover_i";
+
+/// Where Google Books is asked.
+const GOOGLE_BOOKS: &str = "https://www.googleapis.com/books/v1/volumes";
 
 /// Ask which books a register has under a title, and by an author if one is
 /// named.
 ///
 /// Open Library's index first, and Google Books only when that answered with
-/// nothing. A register being down is a failure only when every register
+/// nothing. Once Google Books has been asked, its failure is the answer:
-/// asked is: Open Library answering that it has nothing is an answer, and it
+/// [`Error::Limited`] when it refused for its quota, [`Error::Unreachable`]
-/// stands when Google Books cannot be reached. Asks nothing when `limit` is
+/// when it could not be reached, since saying no register has the title
-/// zero; a `limit` over [`MAX_LIMIT`] is taken as that.
+/// would be a claim no register made. Asks nothing when `limit` is zero; a
+/// `limit` over [`MAX_LIMIT`] is taken as that.
 pub fn search(
     agent: &ureq::Agent,
     title: &str,
@@ -46,9 +52,7 @@ pub fn search(
     limit: usize,
     attempts: u32,
 ) -> Result<Vec<Record>, Error> {
-    search_with(title, author, limit, |url| {
+    search_with(title, author, limit, |url| get(agent, url, attempts))
-        crate::ask::get(agent, url, attempts)
-    })
 }
 
 /// [`search`], with how to fetch an address passed in.
@@ -81,16 +85,60 @@ fn search_with(
         return open_library;
     }
 
-    let google = match google_url(title, author, limit) {
+    let Some(url) = google_url(title, author, limit) else {
-        Some(url) => fetch(url).and_then(|body| google_books(&body, limit)),
         // Nothing in the title Google could be asked by.
-        None => Ok(Vec::new()),
+        return open_library;
     };
-    match (open_library, google) {
+    let google = fetch(url).and_then(|body| google_books(&body, limit))?;
-        (_, Ok(found)) if !found.is_empty() => Ok(found),
+    if google.is_empty() {
-        (Ok(none), _) => Ok(none),
+        // Open Library's nothing, or its failure, which Google Books having
-        (Err(why), _) => Err(why),
+        // nothing either does not excuse.
+        return open_library;
     }
+    Ok(google)
+}
+
+/// The body at an address, asked as a lookup asks, or `None` when the
+/// register says there is no such thing.
+///
+/// Except that Google Books refusing for its quota is [`Error::Limited`]: a
+/// `403` at once, since it is the day's quota and asking again within
+/// seconds will not move it, and a `429` once every attempt has been refused.
+fn get(agent: &ureq::Agent, url: &str, attempts: u32) -> Result<Option<String>, Error> {
+    let attempts = attempts.max(1);
+    let mut last = Error::Unreachable("no attempt was made".into());
+    for attempt in 0..attempts {
+        match agent.get(url).call() {
+            Ok(mut response) => {
+                return response
+                    .body_mut()
+                    .read_to_string()
+                    .map(Some)
+                    .map_err(|e| Error::Unreachable(e.to_string()));
+            }
+            Err(ureq::Error::StatusCode(404)) => return Ok(None),
+            Err(ureq::Error::StatusCode(status)) => match refused(url, status) {
+                Some(why) if status == 403 => return Err(why),
+                Some(why) => last = why,
+                None => last = Error::Unreachable(format!("{url}: answered {status}")),
+            },
+            Err(e) => last = Error::Unreachable(e.to_string()),
+        }
+        if attempt + 1 < attempts {
+            std::thread::sleep(std::time::Duration::from_secs(1 << attempt.min(5)));
+        }
+    }
+    Err(last)
+}
+
+/// The refusal a status from a register is, when it is one for asking too
+/// often.
+fn refused(url: &str, status: u16) -> Option<Error> {
+    (url.starts_with(GOOGLE_BOOKS) && matches!(status, 403 | 429)).then(|| {
+        Error::Limited(format!(
+            "Google Books refused ({status}): its quota for this address is spent for now"
+        ))
+    })
 }
 
 /// Where to ask Open Library.
@@ -124,7 +172,7 @@ fn google_url(title: &str, author: Option<&str>, limit: usize) -> Option<String>
         return None;
     }
     Some(format!(
-        "https://www.googleapis.com/books/v1/volumes?q={}&maxResults={limit}",
+        "{GOOGLE_BOOKS}?q={}&maxResults={limit}",
         source::encode(&terms.join(" "))
     ))
 }
@@ -162,18 +210,8 @@ struct Work {
     #[serde(default)]
     author_name: Vec<String>,
     #[serde(default)]
-    first_publish_year: Option<i64>,
-    #[serde(default)]
-    publisher: Vec<String>,
-    #[serde(default)]
-    isbn: Vec<String>,
-    #[serde(default)]
-    language: Vec<String>,
-    #[serde(default)]
     subject: Vec<String>,
     #[serde(default)]
-    number_of_pages_median: Option<u32>,
-    #[serde(default)]
     editions: Option<Editions>,
 }
 
@@ -212,10 +250,11 @@ fn olid(key: &str) -> Option<String> {
 /// Read Open Library's answer into one candidate per work, in its order.
 ///
 /// What belongs to an edition — its ISBN, publisher, language, year and
-/// cover — is taken from the edition the index chose as matching best; what
+/// cover — is taken from the edition the index chose as matching best, and
-/// belongs to the work — its title, authors, subjects and length — from the
+/// left out when that edition does not give it; only the title, authors and
-/// work. A work the index names no edition of gives its first ISBN that reads,
+/// subjects, which describe the work, are the work's. A work the index names
-/// which may be any edition's, and no cover id.
+/// no edition of is named by the work's key and says nothing an edition
+/// would.
 fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
     let reply: OpenLibraryReply =
         serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?;
@@ -228,26 +267,47 @@ fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
 }
 
 fn candidate(work: Work) -> Option<Record> {
-    let edition = work.editions.and_then(|e| e.docs.into_iter().next());
+    // The edition, and what it is named by: its ISBN, or its key when it has
+    // none. One with neither cannot be named, and is as good as none.
+    let edition = work
+        .editions
+        .and_then(|e| e.docs.into_iter().next())
+        .and_then(|edition| {
+            match first_isbn(edition.isbn.iter().map(String::as_str)) {
+                Some(isbn) => Some((isbn, "isbn")),
+                None => edition
+                    .key
+                    .as_deref()
+                    .and_then(olid)
+                    .map(|key| (key, "olid")),
+            }
+            .map(|named| (named, edition))
+        });
 
-    // The cover is kept only when the record names the edition it is the
+    let mut record = match edition {
-    // cover of, by that edition's ISBN or its key.
+        Some(((id, kind), edition)) => {
-    let (id, kind, edition_named) = match &edition {
+            let mut record = source::blank(&id, kind, Source::OpenLibrarySearch);
-        Some(edition) => match first_isbn(edition.isbn.iter().map(String::as_str)) {
+            record.container = edition.publisher.into_iter().next();
-            Some(isbn) => (isbn, "isbn", true),
+            record.language = edition.language.into_iter().next();
-            None => match edition.key.as_deref().and_then(olid) {
+            record.year = edition
-                Some(key) => (key, "olid", true),
+                .publish_date
-                None => (work.key.as_deref().and_then(olid)?, "olid", false),
+                .iter()
-            },
+                .find_map(|date| source::year_in(date));
-        },
+            record.cover_id = edition
-        None => match first_isbn(work.isbn.iter().map(String::as_str)) {
+                .cover_i
-            Some(isbn) => (isbn, "isbn", false),
+                .and_then(|id| u64::try_from(id).ok())
+                .filter(|id| *id > 0);
+            record
+        }
+        // The work's own ISBNs, publishers and years are its editions', and
+        // which belongs with which the index does not say.
+        None => source::blank(
             // A work with no key cannot be named, so it is not a candidate.
-            None => (work.key.as_deref().and_then(olid)?, "olid", false),
+            &work.key.as_deref().and_then(olid)?,
-        },
+            "olid",
+            Source::OpenLibrarySearch,
+        ),
     };
-
-    let mut record = source::blank(&id, kind, Source::OpenLibrarySearch);
     record.title = match (work.title, work.subtitle) {
         (Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")),
         (title, _) => title,
@@ -255,30 +315,6 @@ fn candidate(work: Work) -> Option<Record> {
     record.authors = work.author_name;
     record.subjects = work.subject;
     record.subjects.truncate(SUBJECT_LIMIT);
-    record.pages = work.number_of_pages_median;
-
-    let (publisher, language, year) = match edition {
-        Some(edition) => {
-            if edition_named {
-                record.cover_id = edition
-                    .cover_i
-                    .and_then(|id| u64::try_from(id).ok())
-                    .filter(|id| *id > 0);
-            }
-            (
-                edition.publisher.into_iter().next(),
-                edition.language.into_iter().next(),
-                edition
-                    .publish_date
-                    .iter()
-                    .find_map(|date| source::year_in(date)),
-            )
-        }
-        None => (None, None, None),
-    };
-    record.container = publisher.or_else(|| work.publisher.into_iter().next());
-    record.language = language.or_else(|| work.language.into_iter().next());
-    record.year = year.or(work.first_publish_year);
     Some(record)
 }
 
@@ -312,8 +348,9 @@ mod tests {
     use super::*;
 
     /// Open Library's answer to `title=the first world war&author=howard`,
-    /// recorded 2026-11-13 with the fields above and `limit=2`, the ISBN and
+    /// recorded 2026-11-13 with `limit=2`, the ISBN and subject lists
-    /// subject lists shortened.
+    /// shortened, and with the work's own publishers, ISBNs, years and length,
+    /// which are not asked for and are read past when they come.
     const HOWARD: &str = r#"{"numFound": 9, "start": 0, "numFoundExact": true, "num_found": 9,
         "documentation_url": "https://openlibrary.org/dev/docs/api/search", "q": "", "offset": null,
         "docs": [
@@ -420,7 +457,8 @@ mod tests {
         assert_eq!(first.container.as_deref(), Some("Oxford University Press"));
         assert_eq!(first.language.as_deref(), Some("eng"));
         assert_eq!(first.subjects, ["World War, 1914-1918"]);
-        assert_eq!(first.pages, Some(154));
+        // The work's median length is no edition's.
+        assert_eq!(first.pages, None);
         assert_eq!(first.cover_id, Some(1123124));
 
         let second = &found[1];
@@ -474,17 +512,64 @@ mod tests {
     }
 
     #[test]
-    fn a_work_with_no_edition_named_gives_its_isbn_but_no_cover() {
+    fn what_the_edition_does_not_give_is_left_out_not_taken_from_the_work() {
-        let body = r#"{"docs": [{"key": "/works/OL1W", "title": "A Book",
+        let body = r#"{"docs": [{"key": "/works/OL267171W", "title": "War and Peace",
-            "isbn": ["0-306-40615-2"], "publisher": ["Plenum"], "first_publish_year": 1981}]}"#;
+            "author_name": ["Leo Tolstoy"], "subject": ["Napoleonic Wars"],
+            "first_publish_year": 1867, "publisher": ["Crowell"], "language": ["eng"],
+            "number_of_pages_median": 1296,
+            "editions": {"docs": [{"key": "/books/OL1M", "isbn": ["9780199232765"],
+              "cover_i": 5}]}}]}"#;
         let found = open_library(body, 5).expect("candidates");
+        let only = &found[0];
         assert_eq!(
-            (found[0].id.as_str(), found[0].kind.as_str()),
+            (only.id.as_str(), only.kind.as_str()),
-            ("9780306406157", "isbn")
+            ("9780199232765", "isbn")
         );
-        assert_eq!(found[0].container.as_deref(), Some("Plenum"));
+        assert_eq!(only.cover_id, Some(5));
-        assert_eq!(found[0].year, Some(1981));
+        // 1867 and Crowell are other editions'.
-        assert_eq!(found[0].cover_id, None);
+        assert_eq!(only.year, None);
+        assert_eq!(only.container, None);
+        assert_eq!(only.language, None);
+        assert_eq!(only.pages, None);
+        // What describes the work is the work's.
+        assert_eq!(only.title.as_deref(), Some("War and Peace"));
+        assert_eq!(only.authors, ["Leo Tolstoy"]);
+        assert_eq!(only.subjects, ["Napoleonic Wars"]);
+    }
+
+    #[test]
+    fn a_work_with_no_edition_named_is_named_by_its_key_and_says_nothing_an_edition_would() {
+        let work = r#""key": "/works/OL1W", "title": "A Book", "author_name": ["A. Writer"],
+            "isbn": ["0-306-40615-2"], "publisher": ["Plenum"], "first_publish_year": 1981,
+            "language": ["eng"], "number_of_pages_median": 300"#;
+        // No edition, and an edition with neither an ISBN nor a key.
+        for editions in [
+            "",
+            r#", "editions": {"docs": [{"publisher": ["Plenum"], "cover_i": 7}]}"#,
+        ] {
+            let body = format!(r#"{{"docs": [{{{work}{editions}}}]}}"#);
+            let found = open_library(&body, 5).expect("candidates");
+            let only = &found[0];
+            // Not 9780306406157: which edition that is, the index does not say.
+            assert_eq!(
+                (only.id.as_str(), only.kind.as_str()),
+                ("OL1W", "olid"),
+                "{body}"
+            );
+            assert_eq!(only.title.as_deref(), Some("A Book"));
+            assert_eq!(only.authors, ["A. Writer"]);
+            assert_eq!(
+                (
+                    only.year,
+                    only.container.as_deref(),
+                    only.language.as_deref(),
+                    only.pages,
+                    only.cover_id
+                ),
+                (None, None, None, None, None),
+                "{body}"
+            );
+        }
     }
 
     #[test]
@@ -593,15 +678,45 @@ mod tests {
     }
 
     #[test]
-    fn open_library_having_nothing_stands_when_google_books_is_down() {
+    fn google_books_down_after_open_library_had_nothing_is_a_failure() {
         let mut asked = Vec::new();
         let bodies = [
             ("openlibrary.org", Ok(NOTHING)),
             ("googleapis.com", Err(down())),
         ];
-        let found =
+        let failed = search_with("Nothing", None, 5, recorded(&mut asked, &bodies));
-            search_with("Nothing", None, 5, recorded(&mut asked, &bodies)).expect("an answer");
+        assert!(matches!(failed, Err(Error::Unreachable(_))), "{failed:?}");
-        assert!(found.is_empty());
+        assert_eq!(asked.len(), 2);
+    }
+
+    #[test]
+    fn google_books_refusing_for_its_quota_is_said_to_be_that() {
+        let mut asked = Vec::new();
+        let limited = refused(&google_url("Nothing", None, 5).expect("a url"), 429);
+        let bodies = [
+            ("openlibrary.org", Ok(NOTHING)),
+            ("googleapis.com", Err(limited.expect("a refusal"))),
+        ];
+        let failed = search_with("Nothing", None, 5, recorded(&mut asked, &bodies));
+        assert!(matches!(failed, Err(Error::Limited(_))), "{failed:?}");
+    }
+
+    #[test]
+    fn only_google_books_quota_statuses_are_refusals() {
+        let google = google_url("Anathem", None, 5).expect("a url");
+        for status in [403, 429] {
+            assert!(
+                matches!(refused(&google, status), Some(Error::Limited(_))),
+                "{status}"
+            );
+        }
+        for status in [400, 500, 503] {
+            assert!(refused(&google, status).is_none(), "{status}");
+        }
+        let open_library = open_library_url("Anathem", None, 5);
+        for status in [403, 429] {
+            assert!(refused(&open_library, status).is_none(), "{status}");
+        }
     }
 
     #[test]