bookmeta, bibfetch, bindings: what a caller needs to trust a cover

bookmeta: Cover::guessed says the file left the choice open — a Kindle
book's first image record, or an EPUB image picked because its name says
cover — so a caller with a register to ask can prefer that. A cover the
file names is never a guess. Cover::extension matches bibfetch's.

bibfetch: Open Library answers 403 to more than a hundred covers by ISBN
in five minutes from one address. That is now Error::Limited, not
Unreachable: the other form of the ISBN is not asked, the cover id still
is, since those are not limited, and the CLI exits 75. Covers asked by
ISBN are kept three seconds apart across the process (Covers::pace), which
stays under the limit however long a run goes. A deadline too long to add
to the clock no longer panics; it is no deadline.

bindings: fetch_cover and search_title for Python, and catena.Limited,
which fetch_identifier raises too when a register refuses for being asked
too often.

corpus-core: exit 75, a service refused for now.

crates/bibfetch/README.md

@@ -136,6 +136,14 @@ redirect included, stops after two minutes. The two minutes start once it is
 the fetch's turn, so a fetch queued behind others in the same process can wait
 that long again first.
 
+Open Library allows a hundred covers by ISBN every five minutes from one
+address, and refuses the rest with `403`. Covers asked by ISBN are kept three
+seconds apart across the whole process (`Covers::pace`), which stays under that
+limit however long a run goes. A refusal all the same is not mistaken for an
+outage: the other form of the ISBN is not asked, the cover id is, because
+Open Library does not limit those, and with no cover found that way the fetch
+fails with `Error::Limited`, which means ask again later.
+
 | Option | Effect |
 |---|---|
 | `-o`, `--output <FILE>` | Where to write the image |
@@ -144,7 +152,8 @@ that long again first.
 
 Exit `0` the cover was written, `1` Open Library has none, `65` the argument
 was not an ISBN or the answer was not an image, `69` Open Library could not be
-reached, `74` the file could not be written.
+reached, `74` the file could not be written, `75` Open Library refused for
+being asked too often and will answer later.
 
 From Rust, `bibfetch::cover(&agent, isbn, cover_id, size)` returns
 `Ok(None)` when there is no cover, and a [`Cover`](src/cover.rs) with the

crates/bibfetch/src/cover.rs

@@ -46,6 +46,11 @@ const REDIRECT_HOSTS: [&str; 2] = ["covers.openlibrary.org", "archive.org"];
 /// agent's timeout each, and a minute's `Retry-After` between attempts.
 pub const DEADLINE: Duration = Duration::from_secs(120);
 
+/// How long to leave between two covers asked by ISBN. Open Library allows a
+/// hundred every five minutes from one address and refuses the rest with
+/// `403`; one every three seconds stays under that however long a run goes.
+pub const PACE: Duration = Duration::from_secs(3);
+
 /// The longest a `Retry-After` is honoured for. A register asking for more
 /// than this is better treated as down than waited on.
 const MAX_WAIT: Duration = Duration::from_secs(60);
@@ -54,7 +59,10 @@ const MAX_WAIT: Duration = Duration::from_secs(60);
 /// this, and limits covers asked by ISBN to a hundred every five minutes
 /// from one address; threads asking at once would only meet the limit
 /// sooner and all back off together.
-static ONE_AT_A_TIME: Mutex<()> = Mutex::new(());
+///
+/// It holds when the last cover was asked for by ISBN, so that [`PACE`] is
+/// kept across fetches and threads.
+static ONE_AT_A_TIME: Mutex<Option<Instant>> = Mutex::new(None);
 
 /// Which of Open Library's three sizes to ask for.
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
@@ -127,6 +135,9 @@ pub struct Covers {
     /// request is given whatever is left of it, or the agent's own timeout
     /// if that is shorter, and a wait that would run past it is not waited.
     pub deadline: Duration,
+    /// The least time between two covers asked by ISBN; [`PACE`] by default.
+    /// Covers asked by id are not limited and are not spaced out.
+    pub pace: Duration,
 }
 
 impl Default for Covers {
@@ -135,6 +146,7 @@ impl Default for Covers {
             base: OPEN_LIBRARY_COVERS.to_string(),
             attempts: crate::ATTEMPTS,
             deadline: DEADLINE,
+            pace: PACE,
         }
     }
 }
@@ -162,6 +174,12 @@ impl Covers {
     /// of the ISBN does not stop the other being asked, but it does stop the
     /// cover id being asked: the ISBN's own cover may exist, and is the one
     /// wanted. The failure is returned when no ISBN form gave a cover.
+    ///
+    /// Refusal for asking too often is the exception. It is
+    /// [`Error::Limited`], the other form is not asked, since it would only
+    /// be refused too, and the cover id is still asked, because Open Library
+    /// does not limit those. A cover found that way is returned; otherwise the
+    /// refusal is, so that the caller knows to ask again later.
     pub fn fetch(
         &self,
         agent: &ureq::Agent,
@@ -186,31 +204,54 @@ impl Covers {
             .map(|form| format!("{}/b/isbn/{form}-{letter}.jpg?default=false", self.base))
             .collect();
 
-        let _turn = ONE_AT_A_TIME
+        let mut last_by_isbn = ONE_AT_A_TIME
             .lock()
             .unwrap_or_else(|held| held.into_inner());
         // From here, not from the call: waiting for the lock is time another
         // fetch spent, and counting it would let a busy process starve this
-        // one of the time its own requests need.
-        let until = Instant::now() + self.deadline;
+        // one of the time its own requests need. A deadline too long to add
+        // to the clock is no deadline at all.
+        let now = Instant::now();
+        let until = now
+            .checked_add(self.deadline)
+            .unwrap_or_else(|| now + Duration::from_secs(365 * 24 * 60 * 60));
         let mut failed = None;
         for address in &addresses {
-            match self.ask(agent, address, until) {
+            if let Some(ready) = last_by_isbn.and_then(|at| at.checked_add(self.pace)) {
+                let now = Instant::now();
+                if ready > now {
+                    if ready >= until {
+                        return Err(Error::Unreachable(format!(
+                            "{address}: past the {:?} deadline, waiting to keep {:?} \
+between covers asked by ISBN",
+                            self.deadline, self.pace
+                        )));
+                    }
+                    std::thread::sleep(ready - now);
+                }
+            }
+            let answer = self.ask(agent, address, until);
+            *last_by_isbn = Some(Instant::now());
+            match answer {
                 Ok(Answer::Image(cover)) => return Ok(Some(cover)),
                 Ok(Answer::Absent) => {}
+                Err(why @ Error::Limited(_)) => {
+                    failed = Some(why);
+                    break;
+                }
                 Err(why) => failed = Some(why),
             }
         }
-        if let Some(why) = failed {
+        if let Some(why) = failed.take_if(|why| !matches!(why, Error::Limited(_))) {
             return Err(why);
         }
         let Some(id) = cover_id else {
-            return Ok(None);
+            return failed.map_or(Ok(None), Err);
         };
         let address = format!("{}/b/id/{id}-{letter}.jpg?default=false", self.base);
         match self.ask(agent, &address, until)? {
             Answer::Image(cover) => Ok(Some(cover)),
-            Answer::Absent => Ok(None),
+            Answer::Absent => failed.map_or(Ok(None), Err),
         }
     }
 
@@ -311,6 +352,15 @@ Open Library's covers or the Internet Archive"
                 // `default=false` makes a missing cover a 404 rather than a
                 // stand-in image.
                 404 | 410 => return Ok(Answer::Absent),
+                // Open Library's answer to more than a hundred covers asked by
+                // ISBN in five minutes from one address. Asking again at once
+                // would only be refused again.
+                403 => {
+                    return Err(Fault::Final(Error::Limited(format!(
+                        "{at}: refused (403); Open Library allows a hundred covers \
+by ISBN every five minutes from one address"
+                    ))));
+                }
                 429 | 503 => {
                     let wait = header(&response, "retry-after")
                         .and_then(|value| value.trim().parse::<u64>().ok())
@@ -603,6 +653,7 @@ mod tests {
                 base: self.base.clone(),
                 attempts: 3,
                 deadline: DEADLINE,
+                pace: Duration::ZERO,
             }
         }
 
@@ -1052,6 +1103,73 @@ mod tests {
         assert_eq!(stub.asked(), [ISBN13, ISBN10]);
     }
 
+    #[test]
+    fn a_refusal_for_asking_too_often_still_asks_the_cover_id() {
+        let stub = Stub::start(|_, path| match path {
+            ISBN13 => Reply::status(403),
+            "/b/id/42-L.jpg?default=false" => Reply::image("image/jpeg", jpeg(4096)),
+            _ => Reply::status(404),
+        });
+        let cover = stub
+            .covers()
+            .fetch(&agent(), "9780306406157", Some(42), CoverSize::Large)
+            .expect("an answer")
+            .expect("a cover");
+        assert!(cover.url.ends_with("/b/id/42-L.jpg?default=false"));
+        // The ten-digit form would only have been refused too.
+        assert_eq!(stub.asked(), [ISBN13, "/b/id/42-L.jpg?default=false"]);
+    }
+
+    #[test]
+    fn a_refusal_for_asking_too_often_is_not_an_absence() {
+        let stub = Stub::start(|_, path| match path {
+            ISBN13 => Reply::status(403),
+            _ => Reply::status(404),
+        });
+        let found = stub
+            .covers()
+            .fetch(&agent(), "9780306406157", None, CoverSize::Large)
+            .map(|cover| cover.map(|cover| cover.url));
+        assert!(matches!(found, Err(Error::Limited(_))), "{found:?}");
+        assert_eq!(stub.asked(), [ISBN13]);
+
+        // With a cover id that has no cover either, the refusal still stands:
+        // the ISBN's own cover may be there once the limit passes.
+        let found = stub
+            .covers()
+            .fetch(&agent(), "9780306406157", Some(42), CoverSize::Large)
+            .map(|cover| cover.map(|cover| cover.url));
+        assert!(matches!(found, Err(Error::Limited(_))), "{found:?}");
+    }
+
+    #[test]
+    fn a_deadline_too_long_to_count_is_no_deadline() {
+        let stub = Stub::start(|_, _| Reply::image("image/jpeg", jpeg(2048)));
+        let covers = Covers {
+            deadline: Duration::MAX,
+            ..stub.covers()
+        };
+        let cover = covers
+            .fetch(&agent(), "9780306406157", None, CoverSize::Large)
+            .expect("an answer");
+        assert!(cover.is_some());
+    }
+
+    #[test]
+    fn covers_asked_by_isbn_are_spaced_out() {
+        let stub = Stub::start(|_, _| Reply::status(404));
+        let covers = Covers {
+            pace: Duration::from_millis(300),
+            ..stub.covers()
+        };
+        let started = Instant::now();
+        let found = covers.fetch(&agent(), "9780306406157", None, CoverSize::Large);
+        assert!(matches!(found, Ok(None)), "{found:?}");
+        // Two forms of the ISBN, so at least one pause between them.
+        assert!(started.elapsed() >= Duration::from_millis(300));
+        assert_eq!(stub.asked(), [ISBN13, ISBN10]);
+    }
+
     #[test]
     fn a_redirect_to_an_address_by_number_is_refused() {
         // A listener standing in for a service on the LAN; it only counts.

crates/bibfetch/src/lib.rs

@@ -112,6 +112,9 @@ pub enum Error {
     NoRegister(String),
     /// What was asked about is not an identifier of the kind it should be.
     Invalid(String),
+    /// The register refused because it has been asked too often from this
+    /// address. It is not down, and asking again later will work.
+    Limited(String),
 }
 
 impl std::fmt::Display for Error {
@@ -123,6 +126,7 @@ impl std::fmt::Display for Error {
                 write!(f, "no register here answers about {kind} identifiers")
             }
             Error::Invalid(why) => write!(f, "{why}"),
+            Error::Limited(why) => write!(f, "{why}"),
         }
     }
 }

crates/bibfetch/src/main.rs

@@ -225,6 +225,7 @@ fn cover(
             eprintln!("bibfetch: {isbn}: {why}");
             ExitCode::from(match why {
                 bibfetch::Error::Unreachable(_) => exit::UNAVAILABLE,
+                bibfetch::Error::Limited(_) => exit::TEMPFAIL,
                 _ => exit::DATA,
             })
         }