Merge pull request 'bibfetch: the books a register has under a title' (#9) from bibfetch-search into main

This commit is contained in:
funkpower committed 2026-11-13 15:59:31 -05:00
commit 58573399e7
8 files changed
+997 -50

No files matched your search

+1 -1
View File
@@ -34,7 +34,7 @@ Each tool installs by name, so you can take one without taking the rest.
| [`vocabmerge`](crates/bibclean/README.md#vocabmerge--the-other-command) | The near-duplicate headings in a catalogue's vocabulary |
| [`seriesof`](crates/seriesof/README.md) | The series a book belongs to, and which volume this is |
| [`idnorm`](crates/idnorm/README.md) | DOIs, ISSNs, ISBNs and arXiv ids, read and checked |
| [`bibfetch`](crates/bibfetch/README.md) | What a register says about an identifier |
| [`bibfetch`](crates/bibfetch/README.md) | What a register says about an identifier, and which books it has under a title |
| [`citefind`](crates/citefind/README.md) | The references in a bibliography or a page of notes |
| [`mdscrub`](crates/mdscrub/README.md) | Converted markdown in, clean markdown out |
| [`mdsplit`](crates/mdsplit/README.md) | One markdown file into the works inside it |
+77 -3
View File
@@ -20,6 +20,8 @@ So this asks, and that is the only thing it does.
- **Filling in a reference** you have only the identifier for.
- **Before citing something**, to check the year and the authors are what you
think they are.
- **Cataloguing a book that came with no identifier**, to find the editions a
register has under its title, and an ISBN to file it by.
## Install
@@ -148,6 +150,77 @@ From Rust, `bibfetch::cover(&agent, isbn, cover_id, size)` returns
`Ok(None)` when there is no cover, and a [`Cover`](src/cover.rs) with the
bytes, the media type and the address asked when there is.
## Searching by title
```
$ bibfetch search --title 'The First World War' --author Howard --limit 2
{"v":1,"id":"9780192853622","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War","authors":["Michael Eliot Howard"],"year":2002,"container":"Oxford University Press","subjects":["World War, 1914-1918"],"language":"eng","pages":154,"cover_id":1123124}
{"v":1,"id":"9780199205592","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War: A Very Short Introduction","authors":["Michael Howard"],"year":2007,"container":"Oxford University Press","language":"eng","cover_id":14606475}
```
A book that arrives as a file often has no identifier to ask about, only a
title and perhaps a name. This asks which books a register has under those,
and prints one record per candidate: Open Library's index first, and Google
Books only when Open Library has nothing.
Each candidate is one edition. Open Library's index answers with works, and
names the edition of each that best matches the search; what belongs to an
edition — its ISBN, publisher, language, year and cover — is that edition's,
and what belongs to the work — the title, the authors, the subjects and the
length — is the work's. Google Books answers with volumes, which are editions
already.
| Field | When the candidate has an ISBN | When it has none |
|---|---|---|
| `id` | The first ISBN whose check digit agrees, in thirteen digits; one written in thirteen is taken over one written in ten | The register's own key: an Open Library id such as `OL37863121M` for the edition, or `OL267171W` for the work when no edition was named; a Google Books volume id |
| `kind` | `isbn` | `olid` for Open Library, `googlebooks` for Google Books |
An ISBN whose check digit is wrong is passed over rather than corrected, and
Google's shelfmarks, typed `OTHER`, are not read as ISBNs even when their
digits would pass for one. A record of kind `isbn` can be handed straight back
to `bibfetch` for everything the registers say about that edition; `olid` and
`googlebooks` are names no lookup here answers about.
`cover_id` is the same thing it is in a lookup record: the first cover in
Open Library's record of the edition the record names, by that edition's ISBN
or its key, so `bibfetch cover --cover-id` fetches that edition's cover and no
other. The cover the index picked for the whole work is never asked for,
because it is often another edition's. A work the index names no edition of
gives the first ISBN it lists that reads, which may be any edition's, and no
cover id; Google Books knows nothing of Open Library's covers and gives none.
The candidates are what the register said, in the order it said them. This
makes no judgement about which one is the book in hand, and keeps one that is
plainly another book, because a title is not an identifier: *The Hamlet* comes
back for *Hamlet*. Deciding whether a candidate is the same work is a
comparison of titles, and [`bibclean`](../bibclean/README.md)'s `same_work`
is one.
```sh
bibfetch search --title 'Anathem' --author Stephenson \
| jq -r 'select(.kind == "isbn") | .id' | head -1
```
Open Library answering that it has nothing is an answer, and it stands when
Google Books cannot be reached; a register being down is a failure only when
both are. Each is asked the same way a lookup asks, retried with the same
waits. Open Library is sent the title and the author as written; Google Books
is sent each word as its own `intitle:` or `inauthor:` term, so nothing in a
title is read as search syntax.
| Option | Effect |
|---|---|
| `--title <TITLE>` | The title, or as much of it as you have |
| `--author <AUTHOR>` | An author's name, or part of one |
| `--limit <COUNT>` | The most candidates to print, `1` to `40`. Default `10` |
Exit `0` at least one candidate was printed, `1` no register has the title,
`65` the title was empty, `69` no register could be reached, `74` the output
could not be written.
From Rust, `bibfetch::search(&agent, title, author, limit, attempts)` returns
the candidates, empty when no register has the title.
## Reaching the network
This and [`emb`](../emb/README.md) are the only two tools here that do. Both
@@ -156,10 +229,11 @@ something that happens on the way to something else — which is the condition
under which [the contract](../../docs/contract.md) allows it at all.
Nothing is cached, nothing is stored but a cover you asked to have written,
and nothing is sent but the identifier and the address you pass to
`--polite`.
and nothing is sent but the identifier, or the title and author searched for,
and the address you pass to `--polite`.
## See also
- [`idnorm`](../idnorm/README.md) — reading the identifier in the first place.
- [`bibclean`](../bibclean/README.md) — the rest of a catalogue record.
- [`bibclean`](../bibclean/README.md) — the rest of a catalogue record, and
whether a candidate is the same work.
+30 -25
View File
@@ -89,10 +89,6 @@ pub fn lookup(
}
/// Ask a register, retrying a failure that might not be one.
///
/// Waits a second before the second attempt and two before the third: long
/// enough for a register that is rate-limiting, short enough that one which
/// is simply down does not hold up a pipeline.
fn ask(
agent: &ureq::Agent,
source: Source,
@@ -100,39 +96,48 @@ fn ask(
kind: &str,
attempts: u32,
) -> Result<crate::Record, Error> {
let mut last = Error::Unreachable("no attempt was made".into());
match get(agent, &source.url(id), attempts)? {
Some(body) => source.read(id, kind, &body),
// A register that does not have it says so with a status, and that is
// an answer rather than a failure.
None => Ok(crate::missing(id, kind, source)),
}
}
for attempt in 0..attempts.max(1) {
match fetch(agent, source, id, kind) {
Ok(record) => return Ok(record),
// A shape this cannot read will not read any better next time.
Err(e @ Error::Shape(_)) => return Err(e),
/// The body at an address, or `None` when the register says it has no such
/// thing; a failure is asked again.
///
/// Waits a second before the second attempt and two before the third: long
/// enough for a register that is rate-limiting, short enough that one which
/// is simply down does not hold up a pipeline. Reading what came back is left
/// to the caller, because an answer this cannot read will not read any better
/// next time.
pub(crate) fn get(agent: &ureq::Agent, url: &str, attempts: u32) -> Result<Option<String>, Error> {
let attempts = attempts.max(1);
let mut last = Error::Unreachable("no attempt was made".into());
for attempt in 0..attempts {
match once(agent, url) {
Ok(answer) => return Ok(answer),
Err(e) => last = e,
}
std::thread::sleep(Duration::from_secs(1 << attempt));
if attempt + 1 < attempts {
std::thread::sleep(Duration::from_secs(1 << attempt.min(5)));
}
}
Err(last)
}
fn fetch(
agent: &ureq::Agent,
source: Source,
id: &str,
kind: &str,
) -> Result<crate::Record, Error> {
let response = agent.get(source.url(id)).call();
let mut response = match response {
fn once(agent: &ureq::Agent, url: &str) -> Result<Option<String>, Error> {
let mut response = match agent.get(url).call() {
Ok(response) => response,
// A register that does not have it says so with a status, and that is
// an answer rather than a failure.
Err(ureq::Error::StatusCode(404)) => return Ok(crate::missing(id, kind, source)),
Err(ureq::Error::StatusCode(404)) => return Ok(None),
Err(e) => return Err(Error::Unreachable(e.to_string())),
};
let body = response
response
.body_mut()
.read_to_string()
.map_err(|e| Error::Unreachable(e.to_string()))?;
source.read(id, kind, &body)
.map(Some)
.map_err(|e| Error::Unreachable(e.to_string()))
}
/// Who is asking.
+6
View File
@@ -12,9 +12,14 @@
//! them: Crossref for a DOI, the arXiv for a preprint, OpenLibrary for a
//! book. There is no register for an ISSN that answers without a key, so an
//! ISSN is looked up as the journal of a work rather than on its own.
//!
//! A book with no identifier at all can still be looked for by its title,
//! with [`search()`]: Open Library's index, then Google Books, each candidate
//! a record of one edition, in the order the register gave them.
pub mod ask;
pub mod cover;
pub mod search;
pub mod source;
use std::time::Duration;
@@ -23,6 +28,7 @@ use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
pub use cover::{Cover, CoverSize, cover};
pub use search::search;
pub use source::Source;
/// How long to wait on a register before giving up.
+73 -9
View File
@@ -30,7 +30,8 @@ Example:
idnorm refs.txt | bibfetch | jq -r 'select(.found) | .title'
idnorm --kind doi refs.txt | bibfetch --polite [email protected]
bibfetch cover 978-0-306-40615-7 -o cover.jpg"
bibfetch cover 978-0-306-40615-7 -o cover.jpg
bibfetch search --title 'The First World War' --author Howard"
)]
struct Args {
#[command(subcommand)]
@@ -74,6 +75,26 @@ enum Command {
#[arg(long, value_name = "ID")]
cover_id: Option<u64>,
},
/// Find the books a register has under a title, and an author.
///
/// Prints one record per candidate edition, in the order the register
/// gave them: Open Library's index, or Google Books when that has
/// nothing. Which one is the book in hand is left to you.
///
/// Exit codes: 0 at least one candidate, 1 none, 65 no title,
/// 69 no register could be reached, 74 the output could not be written.
Search {
/// The title, or as much of it as you have.
#[arg(long)]
title: String,
/// An author's name, or part of one.
#[arg(long)]
author: Option<String>,
/// The most candidates to print.
#[arg(long, default_value_t = 10, value_name = "COUNT",
value_parser = clap::value_parser!(u16).range(1..=bibfetch::search::MAX_LIMIT as i64))]
limit: u16,
},
}
#[derive(Clone, Copy, clap::ValueEnum)]
@@ -96,14 +117,27 @@ fn main() -> ExitCode {
}
.agent();
if let Some(Command::Cover {
isbn,
output,
size,
cover_id,
}) = &args.command
{
return cover(&agent, args.attempts, isbn, output, *size, *cover_id);
match &args.command {
Some(Command::Cover {
isbn,
output,
size,
cover_id,
}) => return cover(&agent, args.attempts, isbn, output, *size, *cover_id),
Some(Command::Search {
title,
author,
limit,
}) => {
return search(
&agent,
args.attempts,
title,
author.as_deref(),
usize::from(*limit),
);
}
None => {}
}
let text = match catena_cli::read_text(args.path.as_deref()) {
@@ -197,6 +231,36 @@ fn cover(
}
}
/// Search by title and print the candidates.
fn search(
agent: &ureq::Agent,
attempts: u32,
title: &str,
author: Option<&str>,
limit: usize,
) -> ExitCode {
match bibfetch::search(agent, title, author, limit, attempts) {
Ok(found) => {
let wrote = catena_cli::stream("bibfetch", &found);
if format!("{wrote:?}") != format!("{:?}", ExitCode::from(exit::OK)) {
return wrote;
}
if found.is_empty() {
eprintln!("bibfetch: no register has {title:?}");
return ExitCode::from(exit::NO);
}
ExitCode::from(exit::OK)
}
Err(why) => {
eprintln!("bibfetch: {why}");
ExitCode::from(match why {
bibfetch::Error::Invalid(_) => exit::DATA,
_ => exit::UNAVAILABLE,
})
}
}
}
/// The identifier and its kind out of one record.
fn read_identifier(line: &str) -> Result<(String, String), String> {
let value: serde_json::Value =
+780
View File
@@ -0,0 +1,780 @@
//! Books by their title, for when there is no identifier to ask about.
//!
//! A book that arrives as a file often carries no ISBN at all, only a title
//! and perhaps a name. A register can still be asked which books it has under
//! those, and what it answers is a list of candidates: each an edition the
//! register holds, with whatever identifier, publisher and cover it gives.
//!
//! Which candidate is the book in hand is not decided here. The candidates
//! are what the register said, in the order it said them; deciding whether
//! one is the same work is a comparison of titles, and that belongs to
//! whoever holds the book.
use serde::Deserialize;
use crate::source::{self, GoogleIdentifier, GoogleReply, SUBJECT_LIMIT, Source};
use crate::{Error, Record};
/// The most candidates one search asks for. Google Books answers with no more
/// than forty at a time, and a title that needs more than forty candidates to
/// find the right one needs a better title.
pub const MAX_LIMIT: usize = 40;
/// What Open Library's index is asked to say about each work, and about the
/// one edition of it that best matches the search.
///
/// Not the work's own `cover_i` or `cover_edition_key`: the cover the index
/// chose for a work is often another edition's. The edition's `cover_i` is
/// the first cover in that edition's record, which is what a lookup by ISBN
/// gives as `cover_id`.
const OPEN_LIBRARY_FIELDS: &str = "key,title,subtitle,author_name,first_publish_year,publisher,\
isbn,language,subject,number_of_pages_median,editions,editions.key,editions.isbn,\
editions.publisher,editions.language,editions.publish_date,editions.cover_i";
/// Ask which books a register has under a title, and by an author if one is
/// named.
///
/// Open Library's index first, and Google Books only when that answered with
/// nothing. A register being down is a failure only when every register
/// asked is: Open Library answering that it has nothing is an answer, and it
/// stands when Google Books cannot be reached. Asks nothing when `limit` is
/// zero; a `limit` over [`MAX_LIMIT`] is taken as that.
pub fn search(
agent: &ureq::Agent,
title: &str,
author: Option<&str>,
limit: usize,
attempts: u32,
) -> Result<Vec<Record>, Error> {
search_with(title, author, limit, |url| {
crate::ask::get(agent, url, attempts)
})
}
/// [`search`], with how to fetch an address passed in.
fn search_with(
title: &str,
author: Option<&str>,
limit: usize,
mut get: impl FnMut(&str) -> Result<Option<String>, Error>,
) -> Result<Vec<Record>, Error> {
let title = title.trim();
if title.is_empty() {
return Err(Error::Invalid("no title to search for".into()));
}
let author = author.map(str::trim).filter(|author| !author.is_empty());
let limit = limit.min(MAX_LIMIT);
if limit == 0 {
return Ok(Vec::new());
}
let mut fetch = |url: String| match get(&url)? {
Some(body) => Ok(body),
// A search that matches nothing answers with an empty list. A 404 is
// the address being wrong, not the book being absent.
None => Err(Error::Unreachable(format!("{url}: answered 404"))),
};
let open_library =
fetch(open_library_url(title, author, limit)).and_then(|body| open_library(&body, limit));
if open_library.as_ref().is_ok_and(|found| !found.is_empty()) {
return open_library;
}
let google = match google_url(title, author, limit) {
Some(url) => fetch(url).and_then(|body| google_books(&body, limit)),
// Nothing in the title Google could be asked by.
None => Ok(Vec::new()),
};
match (open_library, google) {
(_, Ok(found)) if !found.is_empty() => Ok(found),
(Ok(none), _) => Ok(none),
(Err(why), _) => Err(why),
}
}
/// Where to ask Open Library.
fn open_library_url(title: &str, author: Option<&str>, limit: usize) -> String {
let author = author
.map(|author| format!("&author={}", source::encode(author)))
.unwrap_or_default();
format!(
"https://openlibrary.org/search.json?title={}{author}&fields={OPEN_LIBRARY_FIELDS}&limit={limit}",
source::encode(title),
)
}
/// Where to ask Google Books, or `None` when there is nothing to ask by.
///
/// Every word is its own `intitle:` or `inauthor:` term, so no character of
/// the title can open a phrase, negate a term or name another field: a word is
/// the letters, digits and apostrophes in it, and anything else separates
/// words.
fn google_url(title: &str, author: Option<&str>, limit: usize) -> Option<String> {
let terms: Vec<String> = words(title)
.map(|word| format!("intitle:{word}"))
.chain(
author
.into_iter()
.flat_map(words)
.map(|word| format!("inauthor:{word}")),
)
.collect();
if !terms.iter().any(|term| term.starts_with("intitle:")) {
return None;
}
Some(format!(
"https://www.googleapis.com/books/v1/volumes?q={}&maxResults={limit}",
source::encode(&terms.join(" "))
))
}
fn words(text: &str) -> impl Iterator<Item = &str> {
text.split(|c: char| !(c.is_alphanumeric() || c == '\''))
.map(|word| word.trim_matches('\''))
.filter(|word| !word.is_empty())
}
/// The first ISBN in a list whose check digit agrees, in its thirteen-digit
/// form. One written in thirteen digits is taken over one written in ten.
fn first_isbn<'a>(isbns: impl Iterator<Item = &'a str> + Clone) -> Option<String> {
isbns
.clone()
.filter(|isbn| idnorm::isbn::digits(isbn).len() == 13)
.find_map(idnorm::isbn::to_13)
.or_else(|| isbns.into_iter().find_map(idnorm::isbn::to_13))
}
#[derive(Deserialize)]
struct OpenLibraryReply {
#[serde(default)]
docs: Vec<Work>,
}
#[derive(Deserialize)]
struct Work {
#[serde(default)]
key: Option<String>,
#[serde(default)]
title: Option<String>,
#[serde(default)]
subtitle: Option<String>,
#[serde(default)]
author_name: Vec<String>,
#[serde(default)]
first_publish_year: Option<i64>,
#[serde(default)]
publisher: Vec<String>,
#[serde(default)]
isbn: Vec<String>,
#[serde(default)]
language: Vec<String>,
#[serde(default)]
subject: Vec<String>,
#[serde(default)]
number_of_pages_median: Option<u32>,
#[serde(default)]
editions: Option<Editions>,
}
#[derive(Deserialize)]
struct Editions {
#[serde(default)]
docs: Vec<Edition>,
}
#[derive(Deserialize)]
struct Edition {
#[serde(default)]
key: Option<String>,
#[serde(default)]
isbn: Vec<String>,
#[serde(default)]
publisher: Vec<String>,
#[serde(default)]
language: Vec<String>,
#[serde(default)]
publish_date: Vec<String>,
#[serde(default)]
cover_i: Option<i64>,
}
/// The last segment of an Open Library key: `OL3585423M` out of
/// `/books/OL3585423M`. The letter at the end says what it names, `M` an
/// edition and `W` a work.
fn olid(key: &str) -> Option<String> {
key.rsplit('/')
.next()
.filter(|id| !id.is_empty())
.map(str::to_string)
}
/// Read Open Library's answer into one candidate per work, in its order.
///
/// What belongs to an edition — its ISBN, publisher, language, year and
/// cover — is taken from the edition the index chose as matching best; what
/// belongs to the work — its title, authors, subjects and length — from the
/// work. A work the index names no edition of gives its first ISBN that reads,
/// which may be any edition's, and no cover id.
fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
let reply: OpenLibraryReply =
serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?;
Ok(reply
.docs
.into_iter()
.filter_map(candidate)
.take(limit)
.collect())
}
fn candidate(work: Work) -> Option<Record> {
let edition = work.editions.and_then(|e| e.docs.into_iter().next());
// The cover is kept only when the record names the edition it is the
// cover of, by that edition's ISBN or its key.
let (id, kind, edition_named) = match &edition {
Some(edition) => match first_isbn(edition.isbn.iter().map(String::as_str)) {
Some(isbn) => (isbn, "isbn", true),
None => match edition.key.as_deref().and_then(olid) {
Some(key) => (key, "olid", true),
None => (work.key.as_deref().and_then(olid)?, "olid", false),
},
},
None => match first_isbn(work.isbn.iter().map(String::as_str)) {
Some(isbn) => (isbn, "isbn", false),
// A work with no key cannot be named, so it is not a candidate.
None => (work.key.as_deref().and_then(olid)?, "olid", false),
},
};
let mut record = source::blank(&id, kind, Source::OpenLibrarySearch);
record.title = match (work.title, work.subtitle) {
(Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")),
(title, _) => title,
};
record.authors = work.author_name;
record.subjects = work.subject;
record.subjects.truncate(SUBJECT_LIMIT);
record.pages = work.number_of_pages_median;
let (publisher, language, year) = match edition {
Some(edition) => {
if edition_named {
record.cover_id = edition
.cover_i
.and_then(|id| u64::try_from(id).ok())
.filter(|id| *id > 0);
}
(
edition.publisher.into_iter().next(),
edition.language.into_iter().next(),
edition
.publish_date
.iter()
.find_map(|date| source::year_in(date)),
)
}
None => (None, None, None),
};
record.container = publisher.or_else(|| work.publisher.into_iter().next());
record.language = language.or_else(|| work.language.into_iter().next());
record.year = year.or(work.first_publish_year);
Some(record)
}
/// Read Google Books' answer into one candidate per volume, in its order.
fn google_books(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
let reply: GoogleReply = serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?;
Ok(reply
.items
.into_iter()
.filter_map(|item| {
// Only what Google types as an ISBN: its `OTHER` identifiers are
// library shelfmarks, some of which are ten digits long.
let isbns = item
.volume_info
.identifiers
.iter()
.filter(|GoogleIdentifier { kind, .. }| kind == "ISBN_13" || kind == "ISBN_10")
.map(|identifier| identifier.identifier.as_str());
let (id, kind) = match first_isbn(isbns) {
Some(isbn) => (isbn, "isbn"),
None => (item.id.filter(|id| !id.is_empty())?, "googlebooks"),
};
Some(source::google_volume(&id, kind, item.volume_info))
})
.take(limit)
.collect())
}
#[cfg(test)]
mod tests {
use super::*;
/// Open Library's answer to `title=the first world war&author=howard`,
/// recorded 2026-11-13 with the fields above and `limit=2`, the ISBN and
/// subject lists shortened.
const HOWARD: &str = r#"{"numFound": 9, "start": 0, "numFoundExact": true, "num_found": 9,
"documentation_url": "https://openlibrary.org/dev/docs/api/search", "q": "", "offset": null,
"docs": [
{"author_name": ["Michael Eliot Howard"], "first_publish_year": 2002,
"isbn": ["9780192853622", "0192853627"], "key": "/works/OL1382726W",
"language": ["eng"], "number_of_pages_median": 154,
"publisher": ["Oxford University Press"], "title": "The First World War",
"subject": ["World War, 1914-1918"],
"editions": {"numFound": 1, "start": 0, "numFoundExact": true, "docs": [
{"key": "/books/OL3585423M", "cover_i": 1123124, "language": ["eng"],
"publisher": ["Oxford University Press"], "publish_date": ["2002"],
"isbn": ["0192853627", "9780192853622"]}]}},
{"author_name": ["Michael Howard"], "first_publish_year": 2007,
"isbn": ["0199205590", "9780199205592"], "key": "/works/OL4282399W",
"language": ["eng"], "publisher": ["Oxford University Press"],
"subtitle": "A Very Short Introduction", "title": "The First World War",
"editions": {"numFound": 1, "start": 0, "numFoundExact": true, "docs": [
{"key": "/books/OL7403004M", "cover_i": 14606475, "language": ["eng"],
"publisher": ["Oxford University Press"], "publish_date": ["20 May 2007"],
"isbn": ["9780199205592", "0199205590"]}]},
"subject": []}]}"#;
/// Open Library's answer to `title=war and peace&author=tolstoy`, recorded
/// 2026-11-13, first work only. The edition the index matched best is a
/// 1917 printing that has no ISBN, though other editions of the work do.
const TOLSTOY: &str = r#"{"numFound": 146, "docs": [
{"author_name": ["Leo Tolstoy"], "first_publish_year": 1867,
"isbn": ["9781524881993", "1645178889"], "key": "/works/OL267171W",
"title": "War and Peace",
"editions": {"numFound": 487, "docs": [
{"key": "/books/OL37863121M", "cover_i": 12736340, "language": ["eng"],
"publisher": ["Thomas Y. Crowell"], "publish_date": ["1917"]}]}}]}"#;
/// Open Library's answer when nothing has the title.
const NOTHING: &str = r#"{"numFound": 0, "start": 0, "numFoundExact": true, "num_found": 0,
"documentation_url": "https://openlibrary.org/dev/docs/api/search", "q": "",
"offset": null, "docs": []}"#;
/// Google Books' answer for a volume, in the shape its API documents.
const GOOGLE: &str = r#"{"kind": "books#volumes", "totalItems": 2, "items": [
{"kind": "books#volume", "id": "zyTCAlFPjgYC", "volumeInfo": {
"title": "The Google Story", "authors": ["David A. Vise", "Mark Malseed"],
"publisher": "Random House Digital, Inc.", "publishedDate": "2005-11-15",
"industryIdentifiers": [
{"type": "ISBN_10", "identifier": "055380457X"},
{"type": "ISBN_13", "identifier": "9780553804577"}],
"pageCount": 207, "categories": ["Browsers (Computer programs)"],
"language": "en"}},
{"kind": "books#volume", "id": "a1b2C3d4", "volumeInfo": {
"title": "The Google Story", "subtitle": "A Dissertation",
"publishedDate": "2009",
"industryIdentifiers": [
{"type": "OTHER", "identifier": "UOM:0306406152"}]}}]}"#;
/// Fetch from a table of recorded bodies, keeping every address asked.
fn recorded<'a>(
asked: &'a mut Vec<String>,
bodies: &'a [(&'a str, Result<&'a str, Error>)],
) -> impl FnMut(&str) -> Result<Option<String>, Error> + 'a {
move |url| {
asked.push(url.to_string());
let (_, body) = bodies
.iter()
.find(|(host, _)| url.contains(host))
.unwrap_or_else(|| panic!("asked an address nothing was recorded for: {url}"));
body.clone().map(|body| Some(body.to_string()))
}
}
fn down() -> Error {
Error::Unreachable("connection refused".into())
}
#[test]
fn each_work_open_library_has_is_a_candidate_named_by_its_isbn() {
let mut asked = Vec::new();
let bodies = [("openlibrary.org", Ok(HOWARD))];
let found = search_with(
"The First World War",
Some("Howard"),
5,
recorded(&mut asked, &bodies),
)
.expect("candidates");
assert_eq!(
asked.len(),
1,
"Google is not asked when Open Library answers"
);
assert_eq!(found.len(), 2);
let first = &found[0];
assert!(first.found);
assert_eq!(first.v, corpus_core::V);
assert_eq!(first.source, "openlibrary");
// Written ten-digit first by the edition, given in thirteen.
assert_eq!(
(first.id.as_str(), first.kind.as_str()),
("9780192853622", "isbn")
);
assert_eq!(first.title.as_deref(), Some("The First World War"));
assert_eq!(first.authors, ["Michael Eliot Howard"]);
assert_eq!(first.year, Some(2002));
assert_eq!(first.container.as_deref(), Some("Oxford University Press"));
assert_eq!(first.language.as_deref(), Some("eng"));
assert_eq!(first.subjects, ["World War, 1914-1918"]);
assert_eq!(first.pages, Some(154));
assert_eq!(first.cover_id, Some(1123124));
let second = &found[1];
assert_eq!(second.id, "9780199205592");
assert_eq!(
second.title.as_deref(),
Some("The First World War: A Very Short Introduction")
);
// The edition's year, not the work's first.
assert_eq!(second.year, Some(2007));
// The first cover in OL7403004M's own record; the work's cover is
// 1137779, which is not.
assert_eq!(second.cover_id, Some(14606475));
}
#[test]
fn the_candidates_stay_in_the_order_the_register_gave_them() {
let mut asked = Vec::new();
let bodies = [("openlibrary.org", Ok(HOWARD))];
let found =
search_with("first world war", None, 5, recorded(&mut asked, &bodies)).expect("ok");
let ids: Vec<&str> = found.iter().map(|r| r.id.as_str()).collect();
assert_eq!(ids, ["9780192853622", "9780199205592"]);
let found =
search_with("first world war", None, 1, recorded(&mut asked, &bodies)).expect("ok");
assert_eq!(found.len(), 1);
assert_eq!(found[0].id, "9780192853622");
}
#[test]
fn an_edition_with_no_isbn_is_named_by_its_key_and_keeps_its_cover() {
let mut asked = Vec::new();
let bodies = [("openlibrary.org", Ok(TOLSTOY))];
let found = search_with(
"War and Peace",
Some("Tolstoy"),
5,
recorded(&mut asked, &bodies),
)
.expect("candidates");
let only = &found[0];
// Not 9781524881993: that is another edition's, and the publisher,
// year and cover here are this one's.
assert_eq!(
(only.id.as_str(), only.kind.as_str()),
("OL37863121M", "olid")
);
assert_eq!(only.container.as_deref(), Some("Thomas Y. Crowell"));
assert_eq!(only.year, Some(1917));
assert_eq!(only.cover_id, Some(12736340));
}
#[test]
fn a_work_with_no_edition_named_gives_its_isbn_but_no_cover() {
let body = r#"{"docs": [{"key": "/works/OL1W", "title": "A Book",
"isbn": ["0-306-40615-2"], "publisher": ["Plenum"], "first_publish_year": 1981}]}"#;
let found = open_library(body, 5).expect("candidates");
assert_eq!(
(found[0].id.as_str(), found[0].kind.as_str()),
("9780306406157", "isbn")
);
assert_eq!(found[0].container.as_deref(), Some("Plenum"));
assert_eq!(found[0].year, Some(1981));
assert_eq!(found[0].cover_id, None);
}
#[test]
fn a_work_with_no_isbn_and_no_edition_is_named_by_the_works_key() {
let body = r#"{"docs": [{"key": "/works/OL267171W", "title": "War and Peace"}]}"#;
let found = open_library(body, 5).expect("candidates");
assert_eq!(
(found[0].id.as_str(), found[0].kind.as_str()),
("OL267171W", "olid")
);
assert_eq!(found[0].cover_id, None);
}
#[test]
fn isbns_whose_check_digit_is_wrong_are_passed_over() {
// 9780306406158 and 0306406153 are 9780306406157 mistyped.
let body = r#"{"docs": [
{"key": "/works/OL1W", "title": "A", "editions": {"docs": [
{"key": "/books/OL1M", "isbn": ["9780306406158", "0306406153"], "cover_i": 7}]}},
{"key": "/works/OL2W", "title": "B", "editions": {"docs": [
{"key": "/books/OL2M", "isbn": ["9780306406158", "0306406152"]}]}}]}"#;
let found = open_library(body, 5).expect("candidates");
assert_eq!(
(found[0].id.as_str(), found[0].kind.as_str()),
("OL1M", "olid")
);
assert_eq!(found[0].cover_id, Some(7));
// The ten-digit one reads, and is given in thirteen.
assert_eq!(
(found[1].id.as_str(), found[1].kind.as_str()),
("9780306406157", "isbn")
);
}
#[test]
fn thirteen_digits_are_taken_over_ten() {
// Two different ISBNs on one edition: the thirteen-digit one wins
// although the ten-digit one is listed first.
let isbns = ["0306406152", "9780199205592"];
assert_eq!(
first_isbn(isbns.into_iter()).as_deref(),
Some("9780199205592")
);
assert_eq!(first_isbn(["nonsense"].into_iter()), None);
}
#[test]
fn nothing_from_open_library_is_asked_of_google_books() {
let mut asked = Vec::new();
let bodies = [
("openlibrary.org", Ok(NOTHING)),
("googleapis.com", Ok(GOOGLE)),
];
let found = search_with(
"The Google Story",
Some("Vise"),
5,
recorded(&mut asked, &bodies),
)
.expect("candidates");
assert_eq!(asked.len(), 2);
assert!(asked[1].starts_with("https://www.googleapis.com/books/v1/volumes?q="));
assert_eq!(found.len(), 2);
let first = &found[0];
assert_eq!(first.source, "googlebooks");
assert_eq!(
(first.id.as_str(), first.kind.as_str()),
("9780553804577", "isbn")
);
assert_eq!(first.authors, ["David A. Vise", "Mark Malseed"]);
assert_eq!(first.year, Some(2005));
assert_eq!(
first.container.as_deref(),
Some("Random House Digital, Inc.")
);
assert_eq!(first.pages, Some(207));
// Google Books knows nothing of Open Library's covers.
assert_eq!(first.cover_id, None);
// A shelfmark is not an ISBN, though its digits would pass for one.
let second = &found[1];
assert_eq!(
(second.id.as_str(), second.kind.as_str()),
("a1b2C3d4", "googlebooks")
);
assert_eq!(
second.title.as_deref(),
Some("The Google Story: A Dissertation")
);
}
#[test]
fn nothing_from_either_register_is_no_candidates() {
let mut asked = Vec::new();
let bodies = [
("openlibrary.org", Ok(NOTHING)),
(
"googleapis.com",
Ok(r#"{"kind": "books#volumes", "totalItems": 0}"#),
),
];
let found =
search_with("Nothing", None, 5, recorded(&mut asked, &bodies)).expect("an answer");
assert!(found.is_empty());
}
#[test]
fn open_library_having_nothing_stands_when_google_books_is_down() {
let mut asked = Vec::new();
let bodies = [
("openlibrary.org", Ok(NOTHING)),
("googleapis.com", Err(down())),
];
let found =
search_with("Nothing", None, 5, recorded(&mut asked, &bodies)).expect("an answer");
assert!(found.is_empty());
}
#[test]
fn open_library_down_is_asked_of_google_books() {
let mut asked = Vec::new();
let bodies = [
("openlibrary.org", Err(down())),
("googleapis.com", Ok(GOOGLE)),
];
let found =
search_with("The Google Story", None, 5, recorded(&mut asked, &bodies)).expect("ok");
assert_eq!(found[0].source, "googlebooks");
}
#[test]
fn every_register_down_is_a_failure() {
let mut asked = Vec::new();
let bodies = [
("openlibrary.org", Err(down())),
("googleapis.com", Err(down())),
];
let failed = search_with("A Book", None, 5, recorded(&mut asked, &bodies));
assert!(matches!(failed, Err(Error::Unreachable(_))), "{failed:?}");
}
#[test]
fn an_answer_that_cannot_be_read_says_so() {
assert!(matches!(open_library("<html>", 5), Err(Error::Shape(_))));
assert!(matches!(google_books("<html>", 5), Err(Error::Shape(_))));
}
#[test]
fn the_author_is_left_out_when_there_is_none() {
for author in [None, Some(""), Some(" ")] {
let url = open_library_url(
"Anathem",
author.map(str::trim).filter(|a| !a.is_empty()),
5,
);
assert!(!url.contains("author="), "{url}");
}
let mut asked = Vec::new();
let bodies = [
("openlibrary.org", Ok(NOTHING)),
("googleapis.com", Ok(GOOGLE)),
];
search_with("Anathem", Some(" "), 5, recorded(&mut asked, &bodies)).expect("ok");
assert!(!asked[0].contains("author="), "{}", asked[0]);
assert!(!asked[1].contains("inauthor"), "{}", asked[1]);
assert!(asked[1].contains("q=intitle%3AAnathem&"), "{}", asked[1]);
}
#[test]
fn the_author_narrows_the_search_when_there_is_one() {
let url = open_library_url("Anathem", Some("Neal Stephenson"), 5);
assert!(
url.contains("?title=Anathem&author=Neal%20Stephenson&fields="),
"{url}"
);
let url = google_url("Anathem", Some("Neal Stephenson"), 5).expect("a url");
assert!(
url.contains("q=intitle%3AAnathem%20inauthor%3ANeal%20inauthor%3AStephenson&"),
"{url}"
);
}
#[test]
fn a_title_cannot_change_the_shape_of_the_address() {
let hostile = r#"Fish & Chips: "A History" #2?=x/y+z"#;
for url in [
open_library_url(hostile, Some("O'Brien & Sons"), 5),
google_url(hostile, Some("O'Brien & Sons"), 5).expect("a url"),
] {
assert_eq!(url.matches('?').count(), 1, "{url}");
assert_eq!(url.matches('#').count(), 0, "{url}");
assert_eq!(url.matches('"').count(), 0, "{url}");
assert_eq!(url.matches('+').count(), 0, "{url}");
let query = url.split_once('?').expect("a query").1;
let names: Vec<&str> = query
.split('&')
.map(|pair| pair.split_once('=').expect("name=value").0)
.collect();
assert!(
names == ["title", "author", "fields", "limit"] || names == ["q", "maxResults"],
"{url}"
);
}
// Open Library is asked the title as written.
let url = open_library_url(hostile, None, 5);
assert!(
url.contains("title=Fish%20%26%20Chips%3A%20%22A%20History%22%20%232%3F%3Dx%2Fy%2Bz&"),
"{url}"
);
// Google is asked word by word, so nothing in a title is syntax.
let url = google_url(hostile, Some("O'Brien & Sons"), 5).expect("a url");
let q = "intitle:Fish intitle:Chips intitle:A intitle:History intitle:2 intitle:x \
intitle:y intitle:z inauthor:O'Brien inauthor:Sons";
assert!(url.contains(&format!("q={}&", source::encode(q))), "{url}");
}
#[test]
fn a_title_outside_ascii_is_sent_as_utf8() {
let url = open_library_url("Преступление и наказание", Some("Dostoevský"), 5);
assert!(
url.contains("title=%D0%9F%D1%80%D0%B5%D1%81%D1%82%D1%83%D0%BF"),
"{url}"
);
assert!(url.contains("author=Dostoevsk%C3%BD&"), "{url}");
let url = google_url("Преступление и наказание", None, 5).expect("a url");
// Letters in any script are words.
assert!(url.contains("intitle%3A%D0%B8%20"), "{url}");
}
#[test]
fn a_title_with_no_words_is_not_asked_of_google_books() {
assert_eq!(google_url("?!", Some("Someone"), 5), None);
let mut asked = Vec::new();
let bodies = [("openlibrary.org", Ok(NOTHING))];
let found = search_with("?!", Some("Someone"), 5, recorded(&mut asked, &bodies))
.expect("an answer");
assert!(found.is_empty());
assert_eq!(asked.len(), 1);
}
#[test]
fn no_title_is_refused_without_asking() {
let mut asked = Vec::new();
let failed = search_with(" ", Some("Someone"), 5, recorded(&mut asked, &[]));
assert!(matches!(failed, Err(Error::Invalid(_))), "{failed:?}");
assert!(asked.is_empty());
}
#[test]
fn the_limit_is_asked_for_and_held_to() {
let mut asked = Vec::new();
let bodies = [
("openlibrary.org", Ok(NOTHING)),
("googleapis.com", Ok(GOOGLE)),
];
let found =
search_with("The Google Story", None, 1, recorded(&mut asked, &bodies)).expect("ok");
assert_eq!(found.len(), 1);
assert!(asked[0].ends_with("&limit=1"), "{}", asked[0]);
assert!(asked[1].ends_with("&maxResults=1"), "{}", asked[1]);
let mut asked = Vec::new();
search_with("The Google Story", None, 500, recorded(&mut asked, &bodies)).expect("ok");
assert!(asked[0].ends_with("&limit=40"), "{}", asked[0]);
assert!(asked[1].ends_with("&maxResults=40"), "{}", asked[1]);
let mut asked = Vec::new();
let found =
search_with("The Google Story", None, 0, recorded(&mut asked, &bodies)).expect("ok");
assert!(found.is_empty());
assert!(asked.is_empty());
}
#[test]
fn the_index_is_not_asked_for_the_works_cover() {
let url = open_library_url("A", None, 5);
let fields = url
.split('&')
.find_map(|pair| pair.strip_prefix("fields="))
.expect("fields");
let fields: Vec<&str> = fields.split(',').collect();
assert!(fields.contains(&"editions.cover_i"));
assert!(!fields.contains(&"cover_i"));
assert!(!fields.contains(&"cover_edition_key"));
}
#[test]
fn a_404_from_a_search_is_a_failure_and_not_an_absence() {
let failed = search_with("A Book", None, 5, |_| Ok(None));
assert!(matches!(failed, Err(Error::Unreachable(_))), "{failed:?}");
}
}
+29 -11
View File
@@ -117,7 +117,7 @@ number_of_pages_median&limit=1",
///
/// A DOI suffix may hold a slash, a question mark or a hash, and every one of
/// those means something else in a URL.
fn encode(text: &str) -> String {
pub(crate) fn encode(text: &str) -> String {
let mut out = String::with_capacity(text.len());
for byte in text.bytes() {
match byte {
@@ -131,7 +131,7 @@ fn encode(text: &str) -> String {
}
/// A record with the fields nothing found fills in.
fn blank(id: &str, kind: &str, source: Source) -> Record {
pub(crate) fn blank(id: &str, kind: &str, source: Source) -> Record {
let mut record = crate::missing(id, kind, source);
record.found = true;
record
@@ -320,7 +320,7 @@ struct SearchDoc {
/// The index lists every heading every edition has ever been given, which on a
/// well-known book is hundreds and on Hamlet is thousands. A shelf is filed
/// from a handful, and the first are the broadest.
const SUBJECT_LIMIT: usize = 12;
pub(crate) const SUBJECT_LIMIT: usize = 12;
/// A search that matched nothing answers with no documents rather than a
/// status, so an empty reply is the absence and not a failure.
@@ -348,19 +348,22 @@ fn open_library_search(id: &str, kind: &str, body: &str) -> Result<Record, Error
}
#[derive(Deserialize)]
struct GoogleReply {
pub(crate) struct GoogleReply {
#[serde(default)]
items: Vec<GoogleItem>,
pub(crate) items: Vec<GoogleItem>,
}
#[derive(Deserialize)]
struct GoogleItem {
pub(crate) struct GoogleItem {
/// Google's own key for the volume.
#[serde(default)]
pub(crate) id: Option<String>,
#[serde(rename = "volumeInfo", default)]
volume_info: GoogleVolume,
pub(crate) volume_info: GoogleVolume,
}
#[derive(Deserialize, Default)]
struct GoogleVolume {
pub(crate) struct GoogleVolume {
#[serde(default)]
title: Option<String>,
#[serde(default)]
@@ -379,6 +382,18 @@ struct GoogleVolume {
language: Option<String>,
#[serde(rename = "pageCount", default)]
page_count: Option<u32>,
/// `[{"type": "ISBN_13", "identifier": "…"}]`, alongside library
/// shelfmarks typed `OTHER`.
#[serde(rename = "industryIdentifiers", default)]
pub(crate) identifiers: Vec<GoogleIdentifier>,
}
#[derive(Deserialize)]
pub(crate) struct GoogleIdentifier {
#[serde(rename = "type", default)]
pub(crate) kind: String,
#[serde(default)]
pub(crate) identifier: String,
}
/// A search that matched nothing answers with no items rather than a status,
@@ -388,8 +403,11 @@ fn google_books(id: &str, kind: &str, body: &str) -> Result<Record, Error> {
let Some(item) = reply.items.into_iter().next() else {
return Ok(crate::missing(id, kind, Source::GoogleBooks));
};
let volume = item.volume_info;
Ok(google_volume(id, kind, item.volume_info))
}
/// One Google Books volume as a record.
pub(crate) fn google_volume(id: &str, kind: &str, volume: GoogleVolume) -> Record {
let mut record = blank(id, kind, Source::GoogleBooks);
record.title = match (volume.title, volume.subtitle) {
(Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")),
@@ -402,11 +420,11 @@ fn google_books(id: &str, kind: &str, body: &str) -> Result<Record, Error> {
record.subjects = volume.categories;
record.language = volume.language;
record.pages = volume.page_count;
Ok(record)
record
}
/// The first four-digit year in a string.
fn year_in(text: &str) -> Option<i64> {
pub(crate) fn year_in(text: &str) -> Option<i64> {
let bytes: Vec<char> = text.chars().collect();
for window in bytes.windows(4) {
if window.iter().all(char::is_ascii_digit) {
+1 -1
View File
@@ -64,7 +64,7 @@ No network, no database, no configuration file, no state kept between runs.
Five exceptions, each declared in `--help`:
- [`emb`](../crates/emb/README.md) makes an HTTP call, which is the whole of what it does.
- [`bibfetch`](../crates/bibfetch/README.md) asks a register about an identifier, likewise.
- [`bibfetch`](../crates/bibfetch/README.md) asks a register about an identifier or a title, likewise.
- [`pdftriage`](../crates/pdftriage/README.md) runs `pdftotext` and `pdfinfo`.
- [`pdfgutter`](../crates/pdfgutter/README.md) runs `pdftoppm` to look at a page.
- [`avplan`](../crates/avplan/README.md) runs `ffprobe` to read a file's streams, and