bookmeta, bibfetch, bindings: what a caller needs to trust a cover

bookmeta: Cover::guessed says the file left the choice open — a Kindle
book's first image record, or an EPUB image picked because its name says
cover — so a caller with a register to ask can prefer that. A cover the
file names is never a guess. Cover::extension matches bibfetch's.

bibfetch: Open Library answers 403 to more than a hundred covers by ISBN
in five minutes from one address. That is now Error::Limited, not
Unreachable: the other form of the ISBN is not asked, the cover id still
is, since those are not limited, and the CLI exits 75. Covers asked by
ISBN are kept three seconds apart across the process (Covers::pace), which
stays under the limit however long a run goes. A deadline too long to add
to the clock no longer panics; it is no deadline.

bindings: fetch_cover and search_title for Python, and catena.Limited,
which fetch_identifier raises too when a register refuses for being asked
too often.

corpus-core: exit 75, a service refused for now.
This commit is contained in:
funkpower committed 2026-11-13 15:44:50 -05:00
1 parent 7aa1783a3e
commit de23a7abad
14 files changed
+334 -16

No files matched your search

+10 -1
View File
@@ -136,6 +136,14 @@ redirect included, stops after two minutes. The two minutes start once it is
the fetch's turn, so a fetch queued behind others in the same process can wait
that long again first.
Open Library allows a hundred covers by ISBN every five minutes from one
address, and refuses the rest with `403`. Covers asked by ISBN are kept three
seconds apart across the whole process (`Covers::pace`), which stays under that
limit however long a run goes. A refusal all the same is not mistaken for an
outage: the other form of the ISBN is not asked, the cover id is, because
Open Library does not limit those, and with no cover found that way the fetch
fails with `Error::Limited`, which means ask again later.
| Option | Effect |
|---|---|
| `-o`, `--output <FILE>` | Where to write the image |
@@ -144,7 +152,8 @@ that long again first.
Exit `0` the cover was written, `1` Open Library has none, `65` the argument
was not an ISBN or the answer was not an image, `69` Open Library could not be
reached, `74` the file could not be written.
reached, `74` the file could not be written, `75` Open Library refused for
being asked too often and will answer later.
From Rust, `bibfetch::cover(&agent, isbn, cover_id, size)` returns
`Ok(None)` when there is no cover, and a [`Cover`](src/cover.rs) with the
+126 -8
View File
@@ -46,6 +46,11 @@ const REDIRECT_HOSTS: [&str; 2] = ["covers.openlibrary.org", "archive.org"];
/// agent's timeout each, and a minute's `Retry-After` between attempts.
pub const DEADLINE: Duration = Duration::from_secs(120);
/// How long to leave between two covers asked by ISBN. Open Library allows a
/// hundred every five minutes from one address and refuses the rest with
/// `403`; one every three seconds stays under that however long a run goes.
pub const PACE: Duration = Duration::from_secs(3);
/// The longest a `Retry-After` is honoured for. A register asking for more
/// than this is better treated as down than waited on.
const MAX_WAIT: Duration = Duration::from_secs(60);
@@ -54,7 +59,10 @@ const MAX_WAIT: Duration = Duration::from_secs(60);
/// this, and limits covers asked by ISBN to a hundred every five minutes
/// from one address; threads asking at once would only meet the limit
/// sooner and all back off together.
static ONE_AT_A_TIME: Mutex<()> = Mutex::new(());
///
/// It holds when the last cover was asked for by ISBN, so that [`PACE`] is
/// kept across fetches and threads.
static ONE_AT_A_TIME: Mutex<Option<Instant>> = Mutex::new(None);
/// Which of Open Library's three sizes to ask for.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
@@ -127,6 +135,9 @@ pub struct Covers {
/// request is given whatever is left of it, or the agent's own timeout
/// if that is shorter, and a wait that would run past it is not waited.
pub deadline: Duration,
/// The least time between two covers asked by ISBN; [`PACE`] by default.
/// Covers asked by id are not limited and are not spaced out.
pub pace: Duration,
}
impl Default for Covers {
@@ -135,6 +146,7 @@ impl Default for Covers {
base: OPEN_LIBRARY_COVERS.to_string(),
attempts: crate::ATTEMPTS,
deadline: DEADLINE,
pace: PACE,
}
}
}
@@ -162,6 +174,12 @@ impl Covers {
/// of the ISBN does not stop the other being asked, but it does stop the
/// cover id being asked: the ISBN's own cover may exist, and is the one
/// wanted. The failure is returned when no ISBN form gave a cover.
///
/// Refusal for asking too often is the exception. It is
/// [`Error::Limited`], the other form is not asked, since it would only
/// be refused too, and the cover id is still asked, because Open Library
/// does not limit those. A cover found that way is returned; otherwise the
/// refusal is, so that the caller knows to ask again later.
pub fn fetch(
&self,
agent: &ureq::Agent,
@@ -186,31 +204,54 @@ impl Covers {
.map(|form| format!("{}/b/isbn/{form}-{letter}.jpg?default=false", self.base))
.collect();
let _turn = ONE_AT_A_TIME
let mut last_by_isbn = ONE_AT_A_TIME
.lock()
.unwrap_or_else(|held| held.into_inner());
// From here, not from the call: waiting for the lock is time another
// fetch spent, and counting it would let a busy process starve this
// one of the time its own requests need.
let until = Instant::now() + self.deadline;
// one of the time its own requests need. A deadline too long to add
// to the clock is no deadline at all.
let now = Instant::now();
let until = now
.checked_add(self.deadline)
.unwrap_or_else(|| now + Duration::from_secs(365 * 24 * 60 * 60));
let mut failed = None;
for address in &addresses {
match self.ask(agent, address, until) {
if let Some(ready) = last_by_isbn.and_then(|at| at.checked_add(self.pace)) {
let now = Instant::now();
if ready > now {
if ready >= until {
return Err(Error::Unreachable(format!(
"{address}: past the {:?} deadline, waiting to keep {:?} \
between covers asked by ISBN",
self.deadline, self.pace
)));
}
std::thread::sleep(ready - now);
}
}
let answer = self.ask(agent, address, until);
*last_by_isbn = Some(Instant::now());
match answer {
Ok(Answer::Image(cover)) => return Ok(Some(cover)),
Ok(Answer::Absent) => {}
Err(why @ Error::Limited(_)) => {
failed = Some(why);
break;
}
Err(why) => failed = Some(why),
}
}
if let Some(why) = failed {
if let Some(why) = failed.take_if(|why| !matches!(why, Error::Limited(_))) {
return Err(why);
}
let Some(id) = cover_id else {
return Ok(None);
return failed.map_or(Ok(None), Err);
};
let address = format!("{}/b/id/{id}-{letter}.jpg?default=false", self.base);
match self.ask(agent, &address, until)? {
Answer::Image(cover) => Ok(Some(cover)),
Answer::Absent => Ok(None),
Answer::Absent => failed.map_or(Ok(None), Err),
}
}
@@ -311,6 +352,15 @@ Open Library's covers or the Internet Archive"
// `default=false` makes a missing cover a 404 rather than a
// stand-in image.
404 | 410 => return Ok(Answer::Absent),
// Open Library's answer to more than a hundred covers asked by
// ISBN in five minutes from one address. Asking again at once
// would only be refused again.
403 => {
return Err(Fault::Final(Error::Limited(format!(
"{at}: refused (403); Open Library allows a hundred covers \
by ISBN every five minutes from one address"
))));
}
429 | 503 => {
let wait = header(&response, "retry-after")
.and_then(|value| value.trim().parse::<u64>().ok())
@@ -603,6 +653,7 @@ mod tests {
base: self.base.clone(),
attempts: 3,
deadline: DEADLINE,
pace: Duration::ZERO,
}
}
@@ -1052,6 +1103,73 @@ mod tests {
assert_eq!(stub.asked(), [ISBN13, ISBN10]);
}
#[test]
fn a_refusal_for_asking_too_often_still_asks_the_cover_id() {
let stub = Stub::start(|_, path| match path {
ISBN13 => Reply::status(403),
"/b/id/42-L.jpg?default=false" => Reply::image("image/jpeg", jpeg(4096)),
_ => Reply::status(404),
});
let cover = stub
.covers()
.fetch(&agent(), "9780306406157", Some(42), CoverSize::Large)
.expect("an answer")
.expect("a cover");
assert!(cover.url.ends_with("/b/id/42-L.jpg?default=false"));
// The ten-digit form would only have been refused too.
assert_eq!(stub.asked(), [ISBN13, "/b/id/42-L.jpg?default=false"]);
}
#[test]
fn a_refusal_for_asking_too_often_is_not_an_absence() {
let stub = Stub::start(|_, path| match path {
ISBN13 => Reply::status(403),
_ => Reply::status(404),
});
let found = stub
.covers()
.fetch(&agent(), "9780306406157", None, CoverSize::Large)
.map(|cover| cover.map(|cover| cover.url));
assert!(matches!(found, Err(Error::Limited(_))), "{found:?}");
assert_eq!(stub.asked(), [ISBN13]);
// With a cover id that has no cover either, the refusal still stands:
// the ISBN's own cover may be there once the limit passes.
let found = stub
.covers()
.fetch(&agent(), "9780306406157", Some(42), CoverSize::Large)
.map(|cover| cover.map(|cover| cover.url));
assert!(matches!(found, Err(Error::Limited(_))), "{found:?}");
}
#[test]
fn a_deadline_too_long_to_count_is_no_deadline() {
let stub = Stub::start(|_, _| Reply::image("image/jpeg", jpeg(2048)));
let covers = Covers {
deadline: Duration::MAX,
..stub.covers()
};
let cover = covers
.fetch(&agent(), "9780306406157", None, CoverSize::Large)
.expect("an answer");
assert!(cover.is_some());
}
#[test]
fn covers_asked_by_isbn_are_spaced_out() {
let stub = Stub::start(|_, _| Reply::status(404));
let covers = Covers {
pace: Duration::from_millis(300),
..stub.covers()
};
let started = Instant::now();
let found = covers.fetch(&agent(), "9780306406157", None, CoverSize::Large);
assert!(matches!(found, Ok(None)), "{found:?}");
// Two forms of the ISBN, so at least one pause between them.
assert!(started.elapsed() >= Duration::from_millis(300));
assert_eq!(stub.asked(), [ISBN13, ISBN10]);
}
#[test]
fn a_redirect_to_an_address_by_number_is_refused() {
// A listener standing in for a service on the LAN; it only counts.
+4
View File
@@ -112,6 +112,9 @@ pub enum Error {
NoRegister(String),
/// What was asked about is not an identifier of the kind it should be.
Invalid(String),
/// The register refused because it has been asked too often from this
/// address. It is not down, and asking again later will work.
Limited(String),
}
impl std::fmt::Display for Error {
@@ -123,6 +126,7 @@ impl std::fmt::Display for Error {
write!(f, "no register here answers about {kind} identifiers")
}
Error::Invalid(why) => write!(f, "{why}"),
Error::Limited(why) => write!(f, "{why}"),
}
}
}
+1
View File
@@ -225,6 +225,7 @@ fn cover(
eprintln!("bibfetch: {isbn}: {why}");
ExitCode::from(match why {
bibfetch::Error::Unreachable(_) => exit::UNAVAILABLE,
bibfetch::Error::Limited(_) => exit::TEMPFAIL,
_ => exit::DATA,
})
}
@@ -21,6 +21,8 @@ from ._catena import (
compose_subjects,
count_tokens,
cue_windows,
Limited,
fetch_cover,
fetch_identifier,
find_citations,
fuse_hits,
@@ -41,6 +43,7 @@ from ._catena import (
record_severity,
same_work,
scrub,
search_title,
segment_cues,
series_of,
shelve,
@@ -53,6 +56,7 @@ from ._catena import (
__all__ = [
"__version__",
"Limited",
"canonical_subject",
"chunk_markdown",
"citation_prompt",
@@ -62,6 +66,7 @@ __all__ = [
"compose_subjects",
"count_tokens",
"cue_windows",
"fetch_cover",
"fetch_identifier",
"find_citations",
"fuse_hits",
@@ -82,6 +87,7 @@ __all__ = [
"record_severity",
"same_work",
"scrub",
"search_title",
"segment_cues",
"series_of",
"shelve",
@@ -103,6 +103,40 @@ def fetch_identifier(
"""
...
class Limited(Exception):
"""A register refused because it has been asked too often; ask again later."""
def fetch_cover(
isbn: str,
cover_id: int | None = ...,
size: str = ...,
polite: str | None = ...,
timeout: int = ...,
deadline: int = ...,
) -> dict[str, Any] | None:
"""The cover Open Library has for a book, by ISBN, then by cover id.
A dict with `bytes`, `media_type`, `extension`, `source` and `url`, or
`None` when there is none. Raises `Limited` when Open Library refused for
being asked too often. Blocks on the network.
"""
...
def search_title(
title: str,
author: str | None = ...,
limit: int = ...,
polite: str | None = ...,
timeout: int = ...,
attempts: int = ...,
) -> list[dict[str, Any]]:
"""The editions the registers list for a title and author, in their order.
Records shaped like `fetch_identifier`'s. Use `same_work` to decide which
one is the book. Blocks on the network.
"""
...
def colophon_prompt(
record: dict[str, Any], text: str, pages: int = ...
) -> str | None:
+104 -2
View File
@@ -10,7 +10,7 @@
use pyo3::exceptions::PyValueError;
use pyo3::prelude::*;
use pyo3::types::{PyDict, PyList};
use pyo3::types::{PyBytes, PyDict, PyList};
use crate::{ChunkBudgets, CueBudgets};
@@ -434,6 +434,9 @@ fn _catena(module: &Bound<'_, PyModule>) -> PyResult<()> {
module.add_function(wrap_pyfunction!(read_colophon, module)?)?;
module.add_function(wrap_pyfunction!(shelve, module)?)?;
module.add_function(wrap_pyfunction!(fetch_identifier, module)?)?;
module.add_function(wrap_pyfunction!(fetch_cover, module)?)?;
module.add_function(wrap_pyfunction!(search_title, module)?)?;
module.add("Limited", module.py().get_type::<Limited>())?;
module.add_function(wrap_pyfunction!(colophon_prompt, module)?)?;
module.add_function(wrap_pyfunction!(merge_colophon, module)?)?;
Ok(())
@@ -512,13 +515,112 @@ fn fetch_identifier<'py>(
// would stop every other thread in the interpreter for the whole request.
let found = py
.detach(|| bibfetch::ask::lookup(&agent, id, kind, ask.attempts))
.map_err(|e| PyValueError::new_err(e.to_string()))?;
.map_err(register_error)?;
match found {
Some(record) => Ok(Some(to_py(py, &record)?)),
None => Ok(None),
}
}
pyo3::create_exception!(
_catena,
Limited,
pyo3::exceptions::PyException,
"A register refused because it has been asked too often; ask again later."
);
/// A register's error as the Python exception that says what to do about it.
fn register_error(error: bibfetch::Error) -> PyErr {
match error {
bibfetch::Error::Limited(why) => Limited::new_err(why),
other => PyValueError::new_err(other.to_string()),
}
}
/// The cover Open Library has for a book, by its ISBN, then by the cover id
/// a record named.
///
/// Returns a dict with `bytes`, `media_type` (`image/jpeg` or `image/png`),
/// `extension`, `source` and `url`, or `None` when there is no cover. Covers
/// asked by ISBN are kept three seconds apart across the process, and the
/// whole fetch stops after `deadline` seconds. Raises `catena.Limited` when
/// Open Library refused for being asked too often, and `ValueError` when the
/// ISBN is not one or the store could not be reached. Blocks on the network.
#[pyfunction]
#[pyo3(signature = (isbn, cover_id = None, size = "large", polite = None, timeout = 20, deadline = 120))]
fn fetch_cover<'py>(
py: Python<'py>,
isbn: &str,
cover_id: Option<u64>,
size: &str,
polite: Option<String>,
timeout: u64,
deadline: u64,
) -> PyResult<Option<Bound<'py, PyDict>>> {
let size = match size {
"small" => bibfetch::CoverSize::Small,
"medium" => bibfetch::CoverSize::Medium,
"large" => bibfetch::CoverSize::Large,
other => {
return Err(PyValueError::new_err(format!(
"size is small, medium or large, not {other:?}"
)));
}
};
let agent = bibfetch::ask::Ask {
polite,
timeout: std::time::Duration::from_secs(timeout.max(1)),
..Default::default()
}
.agent();
let covers = bibfetch::cover::Covers {
deadline: std::time::Duration::from_secs(deadline.max(1)),
..Default::default()
};
let found = py
.detach(|| covers.fetch(&agent, isbn, cover_id, size))
.map_err(register_error)?;
let Some(cover) = found else {
return Ok(None);
};
let out = PyDict::new(py);
out.set_item("bytes", PyBytes::new(py, &cover.bytes))?;
out.set_item("media_type", cover.media_type)?;
out.set_item("extension", cover.extension())?;
out.set_item("source", cover.source)?;
out.set_item("url", cover.url)?;
Ok(Some(out))
}
/// The editions the registers list for a title, and an author if one is known.
///
/// Open Library is asked first and Google Books only when it has nothing.
/// The candidates come back in the register's own order, as records shaped
/// like `fetch_identifier`'s; nothing here decides which is the book, which is
/// what `same_work` is for. Blocks on the network.
#[pyfunction]
#[pyo3(signature = (title, author = None, limit = 10, polite = None, timeout = 20, attempts = 3))]
fn search_title<'py>(
py: Python<'py>,
title: &str,
author: Option<&str>,
limit: usize,
polite: Option<String>,
timeout: u64,
attempts: u32,
) -> PyResult<Bound<'py, PyList>> {
let ask = bibfetch::ask::Ask {
polite,
timeout: std::time::Duration::from_secs(timeout.max(1)),
attempts: attempts.max(1),
};
let agent = ask.agent();
let found = py
.detach(|| bibfetch::search(&agent, title, author, limit, ask.attempts))
.map_err(register_error)?;
list_to_py(py, &found)
}
/// The question to put to a model about the fields a book's pages did not
/// settle, or `None` when they settled everything.
///
+15 -3
View File
@@ -99,8 +99,10 @@ A Kindle book's title is EXTH 503 where it is there, then the MOBI header's
full name, then the Palm database name (cut to 31 bytes, with a note). Authors
(100) and subjects (105) are every record of their kind, in order; publisher
(101), description (103), date (106) and language (524) the first. EXTH 104 is
an `isbn` when it is shaped like one and `other` when not, and 113 is an
`asin`. A contributor (108) is not an author, the same as an EPUB's
an `isbn` when it is shaped like one and `other` when not. 113 is filed by
its shape: a `uuid` when it is one (calibre writes its book id there), an
`asin` when it is ten letters and digits, and `other` when it is neither. A
contributor (108) is not an author, the same as an EPUB's
`dc:contributor`, and is not reported. With no EXTH language, the language
comes from the header's locale. Strings are decoded as the header says,
UTF-8 or Windows-1252. A book whose text is encrypted still has its header
@@ -117,7 +119,10 @@ read off its first bytes: JPEG, PNG, GIF, WebP or BMP. It is for library
callers and is not part of the JSON record, which is a line of text.
A Kindle book's cover is EXTH 201, or the thumbnail, 202, when no cover is
named, and with neither the first image record.
named. With neither, it is the record at the header's first image index, as
calibre does, and only that record: when it is not a JPEG, PNG or GIF there is
no cover, because the records after it can be illustrations, and kindlegen
sometimes puts a font there first.
An EPUB's is the first of these that reaches an image:
@@ -142,6 +147,13 @@ What the bytes say beats what the manifest says: a JPEG listed as
an EPUB's is never decompressed further than that, whatever size the archive
claims for it.
`Cover::guessed` says the file left the choice open: a Kindle book's first
image record, or an EPUB image picked because its name says `cover`. Either
can be an illustration from inside the book, so a caller that can get a cover
another way, from a register by ISBN, should prefer that one. A cover the file
names is never a guess. `Cover::extension` gives the usual file extension for
its media type.
## Asking a model
Some files say nothing useful. `--prompt` writes the question and **sends
+9 -2
View File
@@ -350,7 +350,8 @@ fn cover<R: std::io::Read + std::io::Seek>(
&& (item.id.to_lowercase().contains("cover")
|| item.href.to_lowercase().contains("cover"))
}) {
if let Some(found) = look.image(base, &item.href) {
if let Some(mut found) = look.image(base, &item.href) {
found.guessed = true;
return Some(found);
}
}
@@ -469,6 +470,7 @@ impl<R: std::io::Read + std::io::Seek> Look<'_, R> {
Some(Cover {
media_type: media_type.to_string(),
bytes,
guessed: false,
})
}
}
@@ -767,6 +769,8 @@ mod tests {
let cover = meta.cover.expect("a cover");
assert_eq!(cover.media_type, "image/jpeg");
assert_eq!(cover.bytes, JPEG);
assert!(!cover.guessed, "a declared cover is not a guess");
assert_eq!(cover.extension(), "jpg");
assert!(meta.notes.is_empty(), "{:?}", meta.notes);
}
@@ -861,7 +865,10 @@ mod tests {
("OEBPS/images/i2.gif", GIF),
],
);
assert_eq!(meta.cover.expect("a cover").bytes, GIF);
let cover = meta.cover.expect("a cover");
assert_eq!(cover.bytes, GIF);
assert!(cover.guessed, "a cover found by its name is a guess");
assert_eq!(cover.extension(), "gif");
let by_address = opf(
r#"<item id="i9" href="images/Book_COVER.jpeg" media-type="image/jpeg"/>"#,
+18
View File
@@ -110,6 +110,24 @@ pub struct Cover {
pub media_type: String,
/// The image itself, never more than [`MAX_COVER`] bytes.
pub bytes: Vec<u8>,
/// Whether the file left the choice open. A MOBI that names no cover, or
/// an EPUB whose only cover is an image with "cover" in its name, has one
/// picked by position or by name, and that pick can be an illustration
/// from inside the book. A caller with a better source should prefer it.
pub guessed: bool,
}
impl Cover {
/// The usual file extension for the image, without the dot.
pub fn extension(&self) -> &'static str {
match self.media_type.as_str() {
"image/png" => "png",
"image/gif" => "gif",
"image/webp" => "webp",
"image/bmp" => "bmp",
_ => "jpg",
}
}
}
/// How large a cover is worth keeping.
+2
View File
@@ -342,6 +342,7 @@ fn cover(
Some(Cover {
media_type: media_type.to_string(),
bytes: data.to_vec(),
guessed: false,
})
}
@@ -361,6 +362,7 @@ fn first_image_record(bytes: &[u8], records: &[usize], first_image: Option<u32>)
Some(Cover {
media_type: media_type.to_string(),
bytes: data.to_vec(),
guessed: true,
})
}
+2
View File
@@ -31,6 +31,7 @@ fn says_what_it_was_given(meta: &bookmeta::Metadata) {
let cover = meta.cover.as_ref().expect("a cover");
assert_eq!(cover.media_type, "image/jpeg");
assert!(!cover.guessed, "EXTH 201 names the cover");
let original = std::fs::read(fixture("lantern-cover.jpg")).expect("the cover fixture");
assert_eq!(cover.bytes, original);
}
@@ -84,6 +85,7 @@ fn a_file_naming_no_cover_gets_its_first_image_record() {
let meta = bookmeta::from_bytes("x.mobi", &lantern_naming_no_cover());
let cover = meta.cover.expect("the first image record");
assert_eq!(cover.media_type, "image/jpeg");
assert!(cover.guessed, "a cover taken by position is a guess");
let original = std::fs::read(fixture("lantern-cover.jpg")).expect("the cover fixture");
assert_eq!(cover.bytes, original);
}
+2
View File
@@ -30,6 +30,8 @@ pub const UNAVAILABLE: u8 = 69;
pub const SOFTWARE: u8 = 70;
/// Reading or writing failed.
pub const IO: u8 = 74;
/// A service refused for now, and asking again later will work.
pub const TEMPFAIL: u8 = 75;
/// The answer a predicate tool gives.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+1
View File
@@ -47,6 +47,7 @@ that was concatenated or written on Windows still reads.
| `69` | A program it needs is not installed |
| `70` | Its own fault |
| `74` | Reading or writing failed |
| `75` | A service refused for now; try again later |
Failures start where `sysexits.h` puts them, so "no" and "broke" are never the
same number. That is what makes this work: