diff --git a/README.md b/README.md index c9cdee9..946fd46 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ Each tool installs by name, so you can take one without taking the rest. | [`vocabmerge`](crates/bibclean/README.md#vocabmerge--the-other-command) | The near-duplicate headings in a catalogue's vocabulary | | [`seriesof`](crates/seriesof/README.md) | The series a book belongs to, and which volume this is | | [`idnorm`](crates/idnorm/README.md) | DOIs, ISSNs, ISBNs and arXiv ids, read and checked | -| [`bibfetch`](crates/bibfetch/README.md) | What a register says about an identifier | +| [`bibfetch`](crates/bibfetch/README.md) | What a register says about an identifier, and which books it has under a title | | [`citefind`](crates/citefind/README.md) | The references in a bibliography or a page of notes | | [`mdscrub`](crates/mdscrub/README.md) | Converted markdown in, clean markdown out | | [`mdsplit`](crates/mdsplit/README.md) | One markdown file into the works inside it | diff --git a/crates/bibfetch/README.md b/crates/bibfetch/README.md index 83b592e..a87eee5 100644 --- a/crates/bibfetch/README.md +++ b/crates/bibfetch/README.md @@ -20,6 +20,8 @@ So this asks, and that is the only thing it does. - **Filling in a reference** you have only the identifier for. - **Before citing something**, to check the year and the authors are what you think they are. +- **Cataloguing a book that came with no identifier**, to find the editions a + register has under its title, and an ISBN to file it by. ## Install @@ -148,6 +150,77 @@ From Rust, `bibfetch::cover(&agent, isbn, cover_id, size)` returns `Ok(None)` when there is no cover, and a [`Cover`](src/cover.rs) with the bytes, the media type and the address asked when there is. +## Searching by title + +``` +$ bibfetch search --title 'The First World War' --author Howard --limit 2 +{"v":1,"id":"9780192853622","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War","authors":["Michael Eliot Howard"],"year":2002,"container":"Oxford University Press","subjects":["World War, 1914-1918"],"language":"eng","pages":154,"cover_id":1123124} +{"v":1,"id":"9780199205592","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War: A Very Short Introduction","authors":["Michael Howard"],"year":2007,"container":"Oxford University Press","language":"eng","cover_id":14606475} +``` + +A book that arrives as a file often has no identifier to ask about, only a +title and perhaps a name. This asks which books a register has under those, +and prints one record per candidate: Open Library's index first, and Google +Books only when Open Library has nothing. + +Each candidate is one edition. Open Library's index answers with works, and +names the edition of each that best matches the search; what belongs to an +edition — its ISBN, publisher, language, year and cover — is that edition's, +and what belongs to the work — the title, the authors, the subjects and the +length — is the work's. Google Books answers with volumes, which are editions +already. + +| Field | When the candidate has an ISBN | When it has none | +|---|---|---| +| `id` | The first ISBN whose check digit agrees, in thirteen digits; one written in thirteen is taken over one written in ten | The register's own key: an Open Library id such as `OL37863121M` for the edition, or `OL267171W` for the work when no edition was named; a Google Books volume id | +| `kind` | `isbn` | `olid` for Open Library, `googlebooks` for Google Books | + +An ISBN whose check digit is wrong is passed over rather than corrected, and +Google's shelfmarks, typed `OTHER`, are not read as ISBNs even when their +digits would pass for one. A record of kind `isbn` can be handed straight back +to `bibfetch` for everything the registers say about that edition; `olid` and +`googlebooks` are names no lookup here answers about. + +`cover_id` is the same thing it is in a lookup record: the first cover in +Open Library's record of the edition the record names, by that edition's ISBN +or its key, so `bibfetch cover --cover-id` fetches that edition's cover and no +other. The cover the index picked for the whole work is never asked for, +because it is often another edition's. A work the index names no edition of +gives the first ISBN it lists that reads, which may be any edition's, and no +cover id; Google Books knows nothing of Open Library's covers and gives none. + +The candidates are what the register said, in the order it said them. This +makes no judgement about which one is the book in hand, and keeps one that is +plainly another book, because a title is not an identifier: *The Hamlet* comes +back for *Hamlet*. Deciding whether a candidate is the same work is a +comparison of titles, and [`bibclean`](../bibclean/README.md)'s `same_work` +is one. + +```sh +bibfetch search --title 'Anathem' --author Stephenson \ + | jq -r 'select(.kind == "isbn") | .id' | head -1 +``` + +Open Library answering that it has nothing is an answer, and it stands when +Google Books cannot be reached; a register being down is a failure only when +both are. Each is asked the same way a lookup asks, retried with the same +waits. Open Library is sent the title and the author as written; Google Books +is sent each word as its own `intitle:` or `inauthor:` term, so nothing in a +title is read as search syntax. + +| Option | Effect | +|---|---| +| `--title ` | The title, or as much of it as you have | +| `--author <AUTHOR>` | An author's name, or part of one | +| `--limit <COUNT>` | The most candidates to print, `1` to `40`. Default `10` | + +Exit `0` at least one candidate was printed, `1` no register has the title, +`65` the title was empty, `69` no register could be reached, `74` the output +could not be written. + +From Rust, `bibfetch::search(&agent, title, author, limit, attempts)` returns +the candidates, empty when no register has the title. + ## Reaching the network This and [`emb`](../emb/README.md) are the only two tools here that do. Both @@ -156,10 +229,11 @@ something that happens on the way to something else — which is the condition under which [the contract](../../docs/contract.md) allows it at all. Nothing is cached, nothing is stored but a cover you asked to have written, -and nothing is sent but the identifier and the address you pass to -`--polite`. +and nothing is sent but the identifier, or the title and author searched for, +and the address you pass to `--polite`. ## See also - [`idnorm`](../idnorm/README.md) — reading the identifier in the first place. -- [`bibclean`](../bibclean/README.md) — the rest of a catalogue record. +- [`bibclean`](../bibclean/README.md) — the rest of a catalogue record, and + whether a candidate is the same work. diff --git a/crates/bibfetch/src/ask.rs b/crates/bibfetch/src/ask.rs index bececa2..b1c3015 100644 --- a/crates/bibfetch/src/ask.rs +++ b/crates/bibfetch/src/ask.rs @@ -89,10 +89,6 @@ pub fn lookup( } /// Ask a register, retrying a failure that might not be one. -/// -/// Waits a second before the second attempt and two before the third: long -/// enough for a register that is rate-limiting, short enough that one which -/// is simply down does not hold up a pipeline. fn ask( agent: &ureq::Agent, source: Source, @@ -100,39 +96,48 @@ fn ask( kind: &str, attempts: u32, ) -> Result<crate::Record, Error> { - let mut last = Error::Unreachable("no attempt was made".into()); + match get(agent, &source.url(id), attempts)? { + Some(body) => source.read(id, kind, &body), + // A register that does not have it says so with a status, and that is + // an answer rather than a failure. + None => Ok(crate::missing(id, kind, source)), + } +} - for attempt in 0..attempts.max(1) { - match fetch(agent, source, id, kind) { - Ok(record) => return Ok(record), - // A shape this cannot read will not read any better next time. - Err(e @ Error::Shape(_)) => return Err(e), +/// The body at an address, or `None` when the register says it has no such +/// thing; a failure is asked again. +/// +/// Waits a second before the second attempt and two before the third: long +/// enough for a register that is rate-limiting, short enough that one which +/// is simply down does not hold up a pipeline. Reading what came back is left +/// to the caller, because an answer this cannot read will not read any better +/// next time. +pub(crate) fn get(agent: &ureq::Agent, url: &str, attempts: u32) -> Result<Option<String>, Error> { + let attempts = attempts.max(1); + let mut last = Error::Unreachable("no attempt was made".into()); + for attempt in 0..attempts { + match once(agent, url) { + Ok(answer) => return Ok(answer), Err(e) => last = e, } - std::thread::sleep(Duration::from_secs(1 << attempt)); + if attempt + 1 < attempts { + std::thread::sleep(Duration::from_secs(1 << attempt.min(5))); + } } Err(last) } -fn fetch( - agent: &ureq::Agent, - source: Source, - id: &str, - kind: &str, -) -> Result<crate::Record, Error> { - let response = agent.get(source.url(id)).call(); - let mut response = match response { +fn once(agent: &ureq::Agent, url: &str) -> Result<Option<String>, Error> { + let mut response = match agent.get(url).call() { Ok(response) => response, - // A register that does not have it says so with a status, and that is - // an answer rather than a failure. - Err(ureq::Error::StatusCode(404)) => return Ok(crate::missing(id, kind, source)), + Err(ureq::Error::StatusCode(404)) => return Ok(None), Err(e) => return Err(Error::Unreachable(e.to_string())), }; - let body = response + response .body_mut() .read_to_string() - .map_err(|e| Error::Unreachable(e.to_string()))?; - source.read(id, kind, &body) + .map(Some) + .map_err(|e| Error::Unreachable(e.to_string())) } /// Who is asking. diff --git a/crates/bibfetch/src/lib.rs b/crates/bibfetch/src/lib.rs index 8c84ce5..0587677 100644 --- a/crates/bibfetch/src/lib.rs +++ b/crates/bibfetch/src/lib.rs @@ -12,9 +12,14 @@ //! them: Crossref for a DOI, the arXiv for a preprint, OpenLibrary for a //! book. There is no register for an ISSN that answers without a key, so an //! ISSN is looked up as the journal of a work rather than on its own. +//! +//! A book with no identifier at all can still be looked for by its title, +//! with [`search()`]: Open Library's index, then Google Books, each candidate +//! a record of one edition, in the order the register gave them. pub mod ask; pub mod cover; +pub mod search; pub mod source; use std::time::Duration; @@ -23,6 +28,7 @@ use schemars::JsonSchema; use serde::{Deserialize, Serialize}; pub use cover::{Cover, CoverSize, cover}; +pub use search::search; pub use source::Source; /// How long to wait on a register before giving up. diff --git a/crates/bibfetch/src/main.rs b/crates/bibfetch/src/main.rs index 069e691..3dd3f93 100644 --- a/crates/bibfetch/src/main.rs +++ b/crates/bibfetch/src/main.rs @@ -30,7 +30,8 @@ Example: idnorm refs.txt | bibfetch | jq -r 'select(.found) | .title' idnorm --kind doi refs.txt | bibfetch --polite you@example.org - bibfetch cover 978-0-306-40615-7 -o cover.jpg" + bibfetch cover 978-0-306-40615-7 -o cover.jpg + bibfetch search --title 'The First World War' --author Howard" )] struct Args { #[command(subcommand)] @@ -74,6 +75,26 @@ enum Command { #[arg(long, value_name = "ID")] cover_id: Option<u64>, }, + /// Find the books a register has under a title, and an author. + /// + /// Prints one record per candidate edition, in the order the register + /// gave them: Open Library's index, or Google Books when that has + /// nothing. Which one is the book in hand is left to you. + /// + /// Exit codes: 0 at least one candidate, 1 none, 65 no title, + /// 69 no register could be reached, 74 the output could not be written. + Search { + /// The title, or as much of it as you have. + #[arg(long)] + title: String, + /// An author's name, or part of one. + #[arg(long)] + author: Option<String>, + /// The most candidates to print. + #[arg(long, default_value_t = 10, value_name = "COUNT", + value_parser = clap::value_parser!(u16).range(1..=bibfetch::search::MAX_LIMIT as i64))] + limit: u16, + }, } #[derive(Clone, Copy, clap::ValueEnum)] @@ -96,14 +117,27 @@ fn main() -> ExitCode { } .agent(); - if let Some(Command::Cover { - isbn, - output, - size, - cover_id, - }) = &args.command - { - return cover(&agent, args.attempts, isbn, output, *size, *cover_id); + match &args.command { + Some(Command::Cover { + isbn, + output, + size, + cover_id, + }) => return cover(&agent, args.attempts, isbn, output, *size, *cover_id), + Some(Command::Search { + title, + author, + limit, + }) => { + return search( + &agent, + args.attempts, + title, + author.as_deref(), + usize::from(*limit), + ); + } + None => {} } let text = match catena_cli::read_text(args.path.as_deref()) { @@ -197,6 +231,36 @@ fn cover( } } +/// Search by title and print the candidates. +fn search( + agent: &ureq::Agent, + attempts: u32, + title: &str, + author: Option<&str>, + limit: usize, +) -> ExitCode { + match bibfetch::search(agent, title, author, limit, attempts) { + Ok(found) => { + let wrote = catena_cli::stream("bibfetch", &found); + if format!("{wrote:?}") != format!("{:?}", ExitCode::from(exit::OK)) { + return wrote; + } + if found.is_empty() { + eprintln!("bibfetch: no register has {title:?}"); + return ExitCode::from(exit::NO); + } + ExitCode::from(exit::OK) + } + Err(why) => { + eprintln!("bibfetch: {why}"); + ExitCode::from(match why { + bibfetch::Error::Invalid(_) => exit::DATA, + _ => exit::UNAVAILABLE, + }) + } + } +} + /// The identifier and its kind out of one record. fn read_identifier(line: &str) -> Result<(String, String), String> { let value: serde_json::Value = diff --git a/crates/bibfetch/src/search.rs b/crates/bibfetch/src/search.rs new file mode 100644 index 0000000..f662c63 --- /dev/null +++ b/crates/bibfetch/src/search.rs @@ -0,0 +1,780 @@ +//! Books by their title, for when there is no identifier to ask about. +//! +//! A book that arrives as a file often carries no ISBN at all, only a title +//! and perhaps a name. A register can still be asked which books it has under +//! those, and what it answers is a list of candidates: each an edition the +//! register holds, with whatever identifier, publisher and cover it gives. +//! +//! Which candidate is the book in hand is not decided here. The candidates +//! are what the register said, in the order it said them; deciding whether +//! one is the same work is a comparison of titles, and that belongs to +//! whoever holds the book. + +use serde::Deserialize; + +use crate::source::{self, GoogleIdentifier, GoogleReply, SUBJECT_LIMIT, Source}; +use crate::{Error, Record}; + +/// The most candidates one search asks for. Google Books answers with no more +/// than forty at a time, and a title that needs more than forty candidates to +/// find the right one needs a better title. +pub const MAX_LIMIT: usize = 40; + +/// What Open Library's index is asked to say about each work, and about the +/// one edition of it that best matches the search. +/// +/// Not the work's own `cover_i` or `cover_edition_key`: the cover the index +/// chose for a work is often another edition's. The edition's `cover_i` is +/// the first cover in that edition's record, which is what a lookup by ISBN +/// gives as `cover_id`. +const OPEN_LIBRARY_FIELDS: &str = "key,title,subtitle,author_name,first_publish_year,publisher,\ +isbn,language,subject,number_of_pages_median,editions,editions.key,editions.isbn,\ +editions.publisher,editions.language,editions.publish_date,editions.cover_i"; + +/// Ask which books a register has under a title, and by an author if one is +/// named. +/// +/// Open Library's index first, and Google Books only when that answered with +/// nothing. A register being down is a failure only when every register +/// asked is: Open Library answering that it has nothing is an answer, and it +/// stands when Google Books cannot be reached. Asks nothing when `limit` is +/// zero; a `limit` over [`MAX_LIMIT`] is taken as that. +pub fn search( + agent: &ureq::Agent, + title: &str, + author: Option<&str>, + limit: usize, + attempts: u32, +) -> Result<Vec<Record>, Error> { + search_with(title, author, limit, |url| { + crate::ask::get(agent, url, attempts) + }) +} + +/// [`search`], with how to fetch an address passed in. +fn search_with( + title: &str, + author: Option<&str>, + limit: usize, + mut get: impl FnMut(&str) -> Result<Option<String>, Error>, +) -> Result<Vec<Record>, Error> { + let title = title.trim(); + if title.is_empty() { + return Err(Error::Invalid("no title to search for".into())); + } + let author = author.map(str::trim).filter(|author| !author.is_empty()); + let limit = limit.min(MAX_LIMIT); + if limit == 0 { + return Ok(Vec::new()); + } + + let mut fetch = |url: String| match get(&url)? { + Some(body) => Ok(body), + // A search that matches nothing answers with an empty list. A 404 is + // the address being wrong, not the book being absent. + None => Err(Error::Unreachable(format!("{url}: answered 404"))), + }; + + let open_library = + fetch(open_library_url(title, author, limit)).and_then(|body| open_library(&body, limit)); + if open_library.as_ref().is_ok_and(|found| !found.is_empty()) { + return open_library; + } + + let google = match google_url(title, author, limit) { + Some(url) => fetch(url).and_then(|body| google_books(&body, limit)), + // Nothing in the title Google could be asked by. + None => Ok(Vec::new()), + }; + match (open_library, google) { + (_, Ok(found)) if !found.is_empty() => Ok(found), + (Ok(none), _) => Ok(none), + (Err(why), _) => Err(why), + } +} + +/// Where to ask Open Library. +fn open_library_url(title: &str, author: Option<&str>, limit: usize) -> String { + let author = author + .map(|author| format!("&author={}", source::encode(author))) + .unwrap_or_default(); + format!( + "https://openlibrary.org/search.json?title={}{author}&fields={OPEN_LIBRARY_FIELDS}&limit={limit}", + source::encode(title), + ) +} + +/// Where to ask Google Books, or `None` when there is nothing to ask by. +/// +/// Every word is its own `intitle:` or `inauthor:` term, so no character of +/// the title can open a phrase, negate a term or name another field: a word is +/// the letters, digits and apostrophes in it, and anything else separates +/// words. +fn google_url(title: &str, author: Option<&str>, limit: usize) -> Option<String> { + let terms: Vec<String> = words(title) + .map(|word| format!("intitle:{word}")) + .chain( + author + .into_iter() + .flat_map(words) + .map(|word| format!("inauthor:{word}")), + ) + .collect(); + if !terms.iter().any(|term| term.starts_with("intitle:")) { + return None; + } + Some(format!( + "https://www.googleapis.com/books/v1/volumes?q={}&maxResults={limit}", + source::encode(&terms.join(" ")) + )) +} + +fn words(text: &str) -> impl Iterator<Item = &str> { + text.split(|c: char| !(c.is_alphanumeric() || c == '\'')) + .map(|word| word.trim_matches('\'')) + .filter(|word| !word.is_empty()) +} + +/// The first ISBN in a list whose check digit agrees, in its thirteen-digit +/// form. One written in thirteen digits is taken over one written in ten. +fn first_isbn<'a>(isbns: impl Iterator<Item = &'a str> + Clone) -> Option<String> { + isbns + .clone() + .filter(|isbn| idnorm::isbn::digits(isbn).len() == 13) + .find_map(idnorm::isbn::to_13) + .or_else(|| isbns.into_iter().find_map(idnorm::isbn::to_13)) +} + +#[derive(Deserialize)] +struct OpenLibraryReply { + #[serde(default)] + docs: Vec<Work>, +} + +#[derive(Deserialize)] +struct Work { + #[serde(default)] + key: Option<String>, + #[serde(default)] + title: Option<String>, + #[serde(default)] + subtitle: Option<String>, + #[serde(default)] + author_name: Vec<String>, + #[serde(default)] + first_publish_year: Option<i64>, + #[serde(default)] + publisher: Vec<String>, + #[serde(default)] + isbn: Vec<String>, + #[serde(default)] + language: Vec<String>, + #[serde(default)] + subject: Vec<String>, + #[serde(default)] + number_of_pages_median: Option<u32>, + #[serde(default)] + editions: Option<Editions>, +} + +#[derive(Deserialize)] +struct Editions { + #[serde(default)] + docs: Vec<Edition>, +} + +#[derive(Deserialize)] +struct Edition { + #[serde(default)] + key: Option<String>, + #[serde(default)] + isbn: Vec<String>, + #[serde(default)] + publisher: Vec<String>, + #[serde(default)] + language: Vec<String>, + #[serde(default)] + publish_date: Vec<String>, + #[serde(default)] + cover_i: Option<i64>, +} + +/// The last segment of an Open Library key: `OL3585423M` out of +/// `/books/OL3585423M`. The letter at the end says what it names, `M` an +/// edition and `W` a work. +fn olid(key: &str) -> Option<String> { + key.rsplit('/') + .next() + .filter(|id| !id.is_empty()) + .map(str::to_string) +} + +/// Read Open Library's answer into one candidate per work, in its order. +/// +/// What belongs to an edition — its ISBN, publisher, language, year and +/// cover — is taken from the edition the index chose as matching best; what +/// belongs to the work — its title, authors, subjects and length — from the +/// work. A work the index names no edition of gives its first ISBN that reads, +/// which may be any edition's, and no cover id. +fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> { + let reply: OpenLibraryReply = + serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?; + Ok(reply + .docs + .into_iter() + .filter_map(candidate) + .take(limit) + .collect()) +} + +fn candidate(work: Work) -> Option<Record> { + let edition = work.editions.and_then(|e| e.docs.into_iter().next()); + + // The cover is kept only when the record names the edition it is the + // cover of, by that edition's ISBN or its key. + let (id, kind, edition_named) = match &edition { + Some(edition) => match first_isbn(edition.isbn.iter().map(String::as_str)) { + Some(isbn) => (isbn, "isbn", true), + None => match edition.key.as_deref().and_then(olid) { + Some(key) => (key, "olid", true), + None => (work.key.as_deref().and_then(olid)?, "olid", false), + }, + }, + None => match first_isbn(work.isbn.iter().map(String::as_str)) { + Some(isbn) => (isbn, "isbn", false), + // A work with no key cannot be named, so it is not a candidate. + None => (work.key.as_deref().and_then(olid)?, "olid", false), + }, + }; + + let mut record = source::blank(&id, kind, Source::OpenLibrarySearch); + record.title = match (work.title, work.subtitle) { + (Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")), + (title, _) => title, + }; + record.authors = work.author_name; + record.subjects = work.subject; + record.subjects.truncate(SUBJECT_LIMIT); + record.pages = work.number_of_pages_median; + + let (publisher, language, year) = match edition { + Some(edition) => { + if edition_named { + record.cover_id = edition + .cover_i + .and_then(|id| u64::try_from(id).ok()) + .filter(|id| *id > 0); + } + ( + edition.publisher.into_iter().next(), + edition.language.into_iter().next(), + edition + .publish_date + .iter() + .find_map(|date| source::year_in(date)), + ) + } + None => (None, None, None), + }; + record.container = publisher.or_else(|| work.publisher.into_iter().next()); + record.language = language.or_else(|| work.language.into_iter().next()); + record.year = year.or(work.first_publish_year); + Some(record) +} + +/// Read Google Books' answer into one candidate per volume, in its order. +fn google_books(body: &str, limit: usize) -> Result<Vec<Record>, Error> { + let reply: GoogleReply = serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?; + Ok(reply + .items + .into_iter() + .filter_map(|item| { + // Only what Google types as an ISBN: its `OTHER` identifiers are + // library shelfmarks, some of which are ten digits long. + let isbns = item + .volume_info + .identifiers + .iter() + .filter(|GoogleIdentifier { kind, .. }| kind == "ISBN_13" || kind == "ISBN_10") + .map(|identifier| identifier.identifier.as_str()); + let (id, kind) = match first_isbn(isbns) { + Some(isbn) => (isbn, "isbn"), + None => (item.id.filter(|id| !id.is_empty())?, "googlebooks"), + }; + Some(source::google_volume(&id, kind, item.volume_info)) + }) + .take(limit) + .collect()) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Open Library's answer to `title=the first world war&author=howard`, + /// recorded 2026-11-13 with the fields above and `limit=2`, the ISBN and + /// subject lists shortened. + const HOWARD: &str = r#"{"numFound": 9, "start": 0, "numFoundExact": true, "num_found": 9, + "documentation_url": "https://openlibrary.org/dev/docs/api/search", "q": "", "offset": null, + "docs": [ + {"author_name": ["Michael Eliot Howard"], "first_publish_year": 2002, + "isbn": ["9780192853622", "0192853627"], "key": "/works/OL1382726W", + "language": ["eng"], "number_of_pages_median": 154, + "publisher": ["Oxford University Press"], "title": "The First World War", + "subject": ["World War, 1914-1918"], + "editions": {"numFound": 1, "start": 0, "numFoundExact": true, "docs": [ + {"key": "/books/OL3585423M", "cover_i": 1123124, "language": ["eng"], + "publisher": ["Oxford University Press"], "publish_date": ["2002"], + "isbn": ["0192853627", "9780192853622"]}]}}, + {"author_name": ["Michael Howard"], "first_publish_year": 2007, + "isbn": ["0199205590", "9780199205592"], "key": "/works/OL4282399W", + "language": ["eng"], "publisher": ["Oxford University Press"], + "subtitle": "A Very Short Introduction", "title": "The First World War", + "editions": {"numFound": 1, "start": 0, "numFoundExact": true, "docs": [ + {"key": "/books/OL7403004M", "cover_i": 14606475, "language": ["eng"], + "publisher": ["Oxford University Press"], "publish_date": ["20 May 2007"], + "isbn": ["9780199205592", "0199205590"]}]}, + "subject": []}]}"#; + + /// Open Library's answer to `title=war and peace&author=tolstoy`, recorded + /// 2026-11-13, first work only. The edition the index matched best is a + /// 1917 printing that has no ISBN, though other editions of the work do. + const TOLSTOY: &str = r#"{"numFound": 146, "docs": [ + {"author_name": ["Leo Tolstoy"], "first_publish_year": 1867, + "isbn": ["9781524881993", "1645178889"], "key": "/works/OL267171W", + "title": "War and Peace", + "editions": {"numFound": 487, "docs": [ + {"key": "/books/OL37863121M", "cover_i": 12736340, "language": ["eng"], + "publisher": ["Thomas Y. Crowell"], "publish_date": ["1917"]}]}}]}"#; + + /// Open Library's answer when nothing has the title. + const NOTHING: &str = r#"{"numFound": 0, "start": 0, "numFoundExact": true, "num_found": 0, + "documentation_url": "https://openlibrary.org/dev/docs/api/search", "q": "", + "offset": null, "docs": []}"#; + + /// Google Books' answer for a volume, in the shape its API documents. + const GOOGLE: &str = r#"{"kind": "books#volumes", "totalItems": 2, "items": [ + {"kind": "books#volume", "id": "zyTCAlFPjgYC", "volumeInfo": { + "title": "The Google Story", "authors": ["David A. Vise", "Mark Malseed"], + "publisher": "Random House Digital, Inc.", "publishedDate": "2005-11-15", + "industryIdentifiers": [ + {"type": "ISBN_10", "identifier": "055380457X"}, + {"type": "ISBN_13", "identifier": "9780553804577"}], + "pageCount": 207, "categories": ["Browsers (Computer programs)"], + "language": "en"}}, + {"kind": "books#volume", "id": "a1b2C3d4", "volumeInfo": { + "title": "The Google Story", "subtitle": "A Dissertation", + "publishedDate": "2009", + "industryIdentifiers": [ + {"type": "OTHER", "identifier": "UOM:0306406152"}]}}]}"#; + + /// Fetch from a table of recorded bodies, keeping every address asked. + fn recorded<'a>( + asked: &'a mut Vec<String>, + bodies: &'a [(&'a str, Result<&'a str, Error>)], + ) -> impl FnMut(&str) -> Result<Option<String>, Error> + 'a { + move |url| { + asked.push(url.to_string()); + let (_, body) = bodies + .iter() + .find(|(host, _)| url.contains(host)) + .unwrap_or_else(|| panic!("asked an address nothing was recorded for: {url}")); + body.clone().map(|body| Some(body.to_string())) + } + } + + fn down() -> Error { + Error::Unreachable("connection refused".into()) + } + + #[test] + fn each_work_open_library_has_is_a_candidate_named_by_its_isbn() { + let mut asked = Vec::new(); + let bodies = [("openlibrary.org", Ok(HOWARD))]; + let found = search_with( + "The First World War", + Some("Howard"), + 5, + recorded(&mut asked, &bodies), + ) + .expect("candidates"); + assert_eq!( + asked.len(), + 1, + "Google is not asked when Open Library answers" + ); + + assert_eq!(found.len(), 2); + let first = &found[0]; + assert!(first.found); + assert_eq!(first.v, corpus_core::V); + assert_eq!(first.source, "openlibrary"); + // Written ten-digit first by the edition, given in thirteen. + assert_eq!( + (first.id.as_str(), first.kind.as_str()), + ("9780192853622", "isbn") + ); + assert_eq!(first.title.as_deref(), Some("The First World War")); + assert_eq!(first.authors, ["Michael Eliot Howard"]); + assert_eq!(first.year, Some(2002)); + assert_eq!(first.container.as_deref(), Some("Oxford University Press")); + assert_eq!(first.language.as_deref(), Some("eng")); + assert_eq!(first.subjects, ["World War, 1914-1918"]); + assert_eq!(first.pages, Some(154)); + assert_eq!(first.cover_id, Some(1123124)); + + let second = &found[1]; + assert_eq!(second.id, "9780199205592"); + assert_eq!( + second.title.as_deref(), + Some("The First World War: A Very Short Introduction") + ); + // The edition's year, not the work's first. + assert_eq!(second.year, Some(2007)); + // The first cover in OL7403004M's own record; the work's cover is + // 1137779, which is not. + assert_eq!(second.cover_id, Some(14606475)); + } + + #[test] + fn the_candidates_stay_in_the_order_the_register_gave_them() { + let mut asked = Vec::new(); + let bodies = [("openlibrary.org", Ok(HOWARD))]; + let found = + search_with("first world war", None, 5, recorded(&mut asked, &bodies)).expect("ok"); + let ids: Vec<&str> = found.iter().map(|r| r.id.as_str()).collect(); + assert_eq!(ids, ["9780192853622", "9780199205592"]); + let found = + search_with("first world war", None, 1, recorded(&mut asked, &bodies)).expect("ok"); + assert_eq!(found.len(), 1); + assert_eq!(found[0].id, "9780192853622"); + } + + #[test] + fn an_edition_with_no_isbn_is_named_by_its_key_and_keeps_its_cover() { + let mut asked = Vec::new(); + let bodies = [("openlibrary.org", Ok(TOLSTOY))]; + let found = search_with( + "War and Peace", + Some("Tolstoy"), + 5, + recorded(&mut asked, &bodies), + ) + .expect("candidates"); + let only = &found[0]; + // Not 9781524881993: that is another edition's, and the publisher, + // year and cover here are this one's. + assert_eq!( + (only.id.as_str(), only.kind.as_str()), + ("OL37863121M", "olid") + ); + assert_eq!(only.container.as_deref(), Some("Thomas Y. Crowell")); + assert_eq!(only.year, Some(1917)); + assert_eq!(only.cover_id, Some(12736340)); + } + + #[test] + fn a_work_with_no_edition_named_gives_its_isbn_but_no_cover() { + let body = r#"{"docs": [{"key": "/works/OL1W", "title": "A Book", + "isbn": ["0-306-40615-2"], "publisher": ["Plenum"], "first_publish_year": 1981}]}"#; + let found = open_library(body, 5).expect("candidates"); + assert_eq!( + (found[0].id.as_str(), found[0].kind.as_str()), + ("9780306406157", "isbn") + ); + assert_eq!(found[0].container.as_deref(), Some("Plenum")); + assert_eq!(found[0].year, Some(1981)); + assert_eq!(found[0].cover_id, None); + } + + #[test] + fn a_work_with_no_isbn_and_no_edition_is_named_by_the_works_key() { + let body = r#"{"docs": [{"key": "/works/OL267171W", "title": "War and Peace"}]}"#; + let found = open_library(body, 5).expect("candidates"); + assert_eq!( + (found[0].id.as_str(), found[0].kind.as_str()), + ("OL267171W", "olid") + ); + assert_eq!(found[0].cover_id, None); + } + + #[test] + fn isbns_whose_check_digit_is_wrong_are_passed_over() { + // 9780306406158 and 0306406153 are 9780306406157 mistyped. + let body = r#"{"docs": [ + {"key": "/works/OL1W", "title": "A", "editions": {"docs": [ + {"key": "/books/OL1M", "isbn": ["9780306406158", "0306406153"], "cover_i": 7}]}}, + {"key": "/works/OL2W", "title": "B", "editions": {"docs": [ + {"key": "/books/OL2M", "isbn": ["9780306406158", "0306406152"]}]}}]}"#; + let found = open_library(body, 5).expect("candidates"); + assert_eq!( + (found[0].id.as_str(), found[0].kind.as_str()), + ("OL1M", "olid") + ); + assert_eq!(found[0].cover_id, Some(7)); + // The ten-digit one reads, and is given in thirteen. + assert_eq!( + (found[1].id.as_str(), found[1].kind.as_str()), + ("9780306406157", "isbn") + ); + } + + #[test] + fn thirteen_digits_are_taken_over_ten() { + // Two different ISBNs on one edition: the thirteen-digit one wins + // although the ten-digit one is listed first. + let isbns = ["0306406152", "9780199205592"]; + assert_eq!( + first_isbn(isbns.into_iter()).as_deref(), + Some("9780199205592") + ); + assert_eq!(first_isbn(["nonsense"].into_iter()), None); + } + + #[test] + fn nothing_from_open_library_is_asked_of_google_books() { + let mut asked = Vec::new(); + let bodies = [ + ("openlibrary.org", Ok(NOTHING)), + ("googleapis.com", Ok(GOOGLE)), + ]; + let found = search_with( + "The Google Story", + Some("Vise"), + 5, + recorded(&mut asked, &bodies), + ) + .expect("candidates"); + assert_eq!(asked.len(), 2); + assert!(asked[1].starts_with("https://www.googleapis.com/books/v1/volumes?q=")); + + assert_eq!(found.len(), 2); + let first = &found[0]; + assert_eq!(first.source, "googlebooks"); + assert_eq!( + (first.id.as_str(), first.kind.as_str()), + ("9780553804577", "isbn") + ); + assert_eq!(first.authors, ["David A. Vise", "Mark Malseed"]); + assert_eq!(first.year, Some(2005)); + assert_eq!( + first.container.as_deref(), + Some("Random House Digital, Inc.") + ); + assert_eq!(first.pages, Some(207)); + // Google Books knows nothing of Open Library's covers. + assert_eq!(first.cover_id, None); + + // A shelfmark is not an ISBN, though its digits would pass for one. + let second = &found[1]; + assert_eq!( + (second.id.as_str(), second.kind.as_str()), + ("a1b2C3d4", "googlebooks") + ); + assert_eq!( + second.title.as_deref(), + Some("The Google Story: A Dissertation") + ); + } + + #[test] + fn nothing_from_either_register_is_no_candidates() { + let mut asked = Vec::new(); + let bodies = [ + ("openlibrary.org", Ok(NOTHING)), + ( + "googleapis.com", + Ok(r#"{"kind": "books#volumes", "totalItems": 0}"#), + ), + ]; + let found = + search_with("Nothing", None, 5, recorded(&mut asked, &bodies)).expect("an answer"); + assert!(found.is_empty()); + } + + #[test] + fn open_library_having_nothing_stands_when_google_books_is_down() { + let mut asked = Vec::new(); + let bodies = [ + ("openlibrary.org", Ok(NOTHING)), + ("googleapis.com", Err(down())), + ]; + let found = + search_with("Nothing", None, 5, recorded(&mut asked, &bodies)).expect("an answer"); + assert!(found.is_empty()); + } + + #[test] + fn open_library_down_is_asked_of_google_books() { + let mut asked = Vec::new(); + let bodies = [ + ("openlibrary.org", Err(down())), + ("googleapis.com", Ok(GOOGLE)), + ]; + let found = + search_with("The Google Story", None, 5, recorded(&mut asked, &bodies)).expect("ok"); + assert_eq!(found[0].source, "googlebooks"); + } + + #[test] + fn every_register_down_is_a_failure() { + let mut asked = Vec::new(); + let bodies = [ + ("openlibrary.org", Err(down())), + ("googleapis.com", Err(down())), + ]; + let failed = search_with("A Book", None, 5, recorded(&mut asked, &bodies)); + assert!(matches!(failed, Err(Error::Unreachable(_))), "{failed:?}"); + } + + #[test] + fn an_answer_that_cannot_be_read_says_so() { + assert!(matches!(open_library("<html>", 5), Err(Error::Shape(_)))); + assert!(matches!(google_books("<html>", 5), Err(Error::Shape(_)))); + } + + #[test] + fn the_author_is_left_out_when_there_is_none() { + for author in [None, Some(""), Some(" ")] { + let url = open_library_url( + "Anathem", + author.map(str::trim).filter(|a| !a.is_empty()), + 5, + ); + assert!(!url.contains("author="), "{url}"); + } + let mut asked = Vec::new(); + let bodies = [ + ("openlibrary.org", Ok(NOTHING)), + ("googleapis.com", Ok(GOOGLE)), + ]; + search_with("Anathem", Some(" "), 5, recorded(&mut asked, &bodies)).expect("ok"); + assert!(!asked[0].contains("author="), "{}", asked[0]); + assert!(!asked[1].contains("inauthor"), "{}", asked[1]); + assert!(asked[1].contains("q=intitle%3AAnathem&"), "{}", asked[1]); + } + + #[test] + fn the_author_narrows_the_search_when_there_is_one() { + let url = open_library_url("Anathem", Some("Neal Stephenson"), 5); + assert!( + url.contains("?title=Anathem&author=Neal%20Stephenson&fields="), + "{url}" + ); + let url = google_url("Anathem", Some("Neal Stephenson"), 5).expect("a url"); + assert!( + url.contains("q=intitle%3AAnathem%20inauthor%3ANeal%20inauthor%3AStephenson&"), + "{url}" + ); + } + + #[test] + fn a_title_cannot_change_the_shape_of_the_address() { + let hostile = r#"Fish & Chips: "A History" #2?=x/y+z"#; + for url in [ + open_library_url(hostile, Some("O'Brien & Sons"), 5), + google_url(hostile, Some("O'Brien & Sons"), 5).expect("a url"), + ] { + assert_eq!(url.matches('?').count(), 1, "{url}"); + assert_eq!(url.matches('#').count(), 0, "{url}"); + assert_eq!(url.matches('"').count(), 0, "{url}"); + assert_eq!(url.matches('+').count(), 0, "{url}"); + let query = url.split_once('?').expect("a query").1; + let names: Vec<&str> = query + .split('&') + .map(|pair| pair.split_once('=').expect("name=value").0) + .collect(); + assert!( + names == ["title", "author", "fields", "limit"] || names == ["q", "maxResults"], + "{url}" + ); + } + // Open Library is asked the title as written. + let url = open_library_url(hostile, None, 5); + assert!( + url.contains("title=Fish%20%26%20Chips%3A%20%22A%20History%22%20%232%3F%3Dx%2Fy%2Bz&"), + "{url}" + ); + // Google is asked word by word, so nothing in a title is syntax. + let url = google_url(hostile, Some("O'Brien & Sons"), 5).expect("a url"); + let q = "intitle:Fish intitle:Chips intitle:A intitle:History intitle:2 intitle:x \ +intitle:y intitle:z inauthor:O'Brien inauthor:Sons"; + assert!(url.contains(&format!("q={}&", source::encode(q))), "{url}"); + } + + #[test] + fn a_title_outside_ascii_is_sent_as_utf8() { + let url = open_library_url("Преступление и наказание", Some("Dostoevský"), 5); + assert!( + url.contains("title=%D0%9F%D1%80%D0%B5%D1%81%D1%82%D1%83%D0%BF"), + "{url}" + ); + assert!(url.contains("author=Dostoevsk%C3%BD&"), "{url}"); + let url = google_url("Преступление и наказание", None, 5).expect("a url"); + // Letters in any script are words. + assert!(url.contains("intitle%3A%D0%B8%20"), "{url}"); + } + + #[test] + fn a_title_with_no_words_is_not_asked_of_google_books() { + assert_eq!(google_url("?!", Some("Someone"), 5), None); + let mut asked = Vec::new(); + let bodies = [("openlibrary.org", Ok(NOTHING))]; + let found = search_with("?!", Some("Someone"), 5, recorded(&mut asked, &bodies)) + .expect("an answer"); + assert!(found.is_empty()); + assert_eq!(asked.len(), 1); + } + + #[test] + fn no_title_is_refused_without_asking() { + let mut asked = Vec::new(); + let failed = search_with(" ", Some("Someone"), 5, recorded(&mut asked, &[])); + assert!(matches!(failed, Err(Error::Invalid(_))), "{failed:?}"); + assert!(asked.is_empty()); + } + + #[test] + fn the_limit_is_asked_for_and_held_to() { + let mut asked = Vec::new(); + let bodies = [ + ("openlibrary.org", Ok(NOTHING)), + ("googleapis.com", Ok(GOOGLE)), + ]; + let found = + search_with("The Google Story", None, 1, recorded(&mut asked, &bodies)).expect("ok"); + assert_eq!(found.len(), 1); + assert!(asked[0].ends_with("&limit=1"), "{}", asked[0]); + assert!(asked[1].ends_with("&maxResults=1"), "{}", asked[1]); + + let mut asked = Vec::new(); + search_with("The Google Story", None, 500, recorded(&mut asked, &bodies)).expect("ok"); + assert!(asked[0].ends_with("&limit=40"), "{}", asked[0]); + assert!(asked[1].ends_with("&maxResults=40"), "{}", asked[1]); + + let mut asked = Vec::new(); + let found = + search_with("The Google Story", None, 0, recorded(&mut asked, &bodies)).expect("ok"); + assert!(found.is_empty()); + assert!(asked.is_empty()); + } + + #[test] + fn the_index_is_not_asked_for_the_works_cover() { + let url = open_library_url("A", None, 5); + let fields = url + .split('&') + .find_map(|pair| pair.strip_prefix("fields=")) + .expect("fields"); + let fields: Vec<&str> = fields.split(',').collect(); + assert!(fields.contains(&"editions.cover_i")); + assert!(!fields.contains(&"cover_i")); + assert!(!fields.contains(&"cover_edition_key")); + } + + #[test] + fn a_404_from_a_search_is_a_failure_and_not_an_absence() { + let failed = search_with("A Book", None, 5, |_| Ok(None)); + assert!(matches!(failed, Err(Error::Unreachable(_))), "{failed:?}"); + } +} diff --git a/crates/bibfetch/src/source.rs b/crates/bibfetch/src/source.rs index 0ab0127..c7b1c96 100644 --- a/crates/bibfetch/src/source.rs +++ b/crates/bibfetch/src/source.rs @@ -117,7 +117,7 @@ number_of_pages_median&limit=1", /// /// A DOI suffix may hold a slash, a question mark or a hash, and every one of /// those means something else in a URL. -fn encode(text: &str) -> String { +pub(crate) fn encode(text: &str) -> String { let mut out = String::with_capacity(text.len()); for byte in text.bytes() { match byte { @@ -131,7 +131,7 @@ fn encode(text: &str) -> String { } /// A record with the fields nothing found fills in. -fn blank(id: &str, kind: &str, source: Source) -> Record { +pub(crate) fn blank(id: &str, kind: &str, source: Source) -> Record { let mut record = crate::missing(id, kind, source); record.found = true; record @@ -320,7 +320,7 @@ struct SearchDoc { /// The index lists every heading every edition has ever been given, which on a /// well-known book is hundreds and on Hamlet is thousands. A shelf is filed /// from a handful, and the first are the broadest. -const SUBJECT_LIMIT: usize = 12; +pub(crate) const SUBJECT_LIMIT: usize = 12; /// A search that matched nothing answers with no documents rather than a /// status, so an empty reply is the absence and not a failure. @@ -348,19 +348,22 @@ fn open_library_search(id: &str, kind: &str, body: &str) -> Result<Record, Error } #[derive(Deserialize)] -struct GoogleReply { +pub(crate) struct GoogleReply { #[serde(default)] - items: Vec<GoogleItem>, + pub(crate) items: Vec<GoogleItem>, } #[derive(Deserialize)] -struct GoogleItem { +pub(crate) struct GoogleItem { + /// Google's own key for the volume. + #[serde(default)] + pub(crate) id: Option<String>, #[serde(rename = "volumeInfo", default)] - volume_info: GoogleVolume, + pub(crate) volume_info: GoogleVolume, } #[derive(Deserialize, Default)] -struct GoogleVolume { +pub(crate) struct GoogleVolume { #[serde(default)] title: Option<String>, #[serde(default)] @@ -379,6 +382,18 @@ struct GoogleVolume { language: Option<String>, #[serde(rename = "pageCount", default)] page_count: Option<u32>, + /// `[{"type": "ISBN_13", "identifier": "…"}]`, alongside library + /// shelfmarks typed `OTHER`. + #[serde(rename = "industryIdentifiers", default)] + pub(crate) identifiers: Vec<GoogleIdentifier>, +} + +#[derive(Deserialize)] +pub(crate) struct GoogleIdentifier { + #[serde(rename = "type", default)] + pub(crate) kind: String, + #[serde(default)] + pub(crate) identifier: String, } /// A search that matched nothing answers with no items rather than a status, @@ -388,8 +403,11 @@ fn google_books(id: &str, kind: &str, body: &str) -> Result<Record, Error> { let Some(item) = reply.items.into_iter().next() else { return Ok(crate::missing(id, kind, Source::GoogleBooks)); }; - let volume = item.volume_info; + Ok(google_volume(id, kind, item.volume_info)) +} +/// One Google Books volume as a record. +pub(crate) fn google_volume(id: &str, kind: &str, volume: GoogleVolume) -> Record { let mut record = blank(id, kind, Source::GoogleBooks); record.title = match (volume.title, volume.subtitle) { (Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")), @@ -402,11 +420,11 @@ fn google_books(id: &str, kind: &str, body: &str) -> Result<Record, Error> { record.subjects = volume.categories; record.language = volume.language; record.pages = volume.page_count; - Ok(record) + record } /// The first four-digit year in a string. -fn year_in(text: &str) -> Option<i64> { +pub(crate) fn year_in(text: &str) -> Option<i64> { let bytes: Vec<char> = text.chars().collect(); for window in bytes.windows(4) { if window.iter().all(char::is_ascii_digit) { diff --git a/docs/contract.md b/docs/contract.md index 6e009d6..4367db1 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -64,7 +64,7 @@ No network, no database, no configuration file, no state kept between runs. Five exceptions, each declared in `--help`: - [`emb`](../crates/emb/README.md) makes an HTTP call, which is the whole of what it does. -- [`bibfetch`](../crates/bibfetch/README.md) asks a register about an identifier, likewise. +- [`bibfetch`](../crates/bibfetch/README.md) asks a register about an identifier or a title, likewise. - [`pdftriage`](../crates/pdftriage/README.md) runs `pdftotext` and `pdfinfo`. - [`pdfgutter`](../crates/pdfgutter/README.md) runs `pdftoppm` to look at a page. - [`avplan`](../crates/avplan/README.md) runs `ffprobe` to read a file's streams, and