mirror of
https://git.pumpin.online/funkpower/catena.git
synced 2026-11-14 08:05:40 +00:00
Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
03c19c7293 | ||
|
|
ce8332d6d8 |
No files matched your search
@@ -34,7 +34,7 @@ Each tool installs by name, so you can take one without taking the rest.
|
|||||||
| [`vocabmerge`](crates/bibclean/README.md#vocabmerge--the-other-command) | The near-duplicate headings in a catalogue's vocabulary |
|
| [`vocabmerge`](crates/bibclean/README.md#vocabmerge--the-other-command) | The near-duplicate headings in a catalogue's vocabulary |
|
||||||
| [`seriesof`](crates/seriesof/README.md) | The series a book belongs to, and which volume this is |
|
| [`seriesof`](crates/seriesof/README.md) | The series a book belongs to, and which volume this is |
|
||||||
| [`idnorm`](crates/idnorm/README.md) | DOIs, ISSNs, ISBNs and arXiv ids, read and checked |
|
| [`idnorm`](crates/idnorm/README.md) | DOIs, ISSNs, ISBNs and arXiv ids, read and checked |
|
||||||
| [`bibfetch`](crates/bibfetch/README.md) | What a register says about an identifier |
|
| [`bibfetch`](crates/bibfetch/README.md) | What a register says about an identifier, and which books it has under a title |
|
||||||
| [`citefind`](crates/citefind/README.md) | The references in a bibliography or a page of notes |
|
| [`citefind`](crates/citefind/README.md) | The references in a bibliography or a page of notes |
|
||||||
| [`mdscrub`](crates/mdscrub/README.md) | Converted markdown in, clean markdown out |
|
| [`mdscrub`](crates/mdscrub/README.md) | Converted markdown in, clean markdown out |
|
||||||
| [`mdsplit`](crates/mdsplit/README.md) | One markdown file into the works inside it |
|
| [`mdsplit`](crates/mdsplit/README.md) | One markdown file into the works inside it |
|
||||||
|
|||||||
@@ -20,6 +20,8 @@ So this asks, and that is the only thing it does.
|
|||||||
- **Filling in a reference** you have only the identifier for.
|
- **Filling in a reference** you have only the identifier for.
|
||||||
- **Before citing something**, to check the year and the authors are what you
|
- **Before citing something**, to check the year and the authors are what you
|
||||||
think they are.
|
think they are.
|
||||||
|
- **Cataloguing a book that came with no identifier**, to find the editions a
|
||||||
|
register has under its title, and an ISBN to file it by.
|
||||||
|
|
||||||
## Install
|
## Install
|
||||||
|
|
||||||
@@ -148,6 +150,77 @@ From Rust, `bibfetch::cover(&agent, isbn, cover_id, size)` returns
|
|||||||
`Ok(None)` when there is no cover, and a [`Cover`](src/cover.rs) with the
|
`Ok(None)` when there is no cover, and a [`Cover`](src/cover.rs) with the
|
||||||
bytes, the media type and the address asked when there is.
|
bytes, the media type and the address asked when there is.
|
||||||
|
|
||||||
|
## Searching by title
|
||||||
|
|
||||||
|
```
|
||||||
|
$ bibfetch search --title 'The First World War' --author Howard --limit 2
|
||||||
|
{"v":1,"id":"9780192853622","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War","authors":["Michael Eliot Howard"],"year":2002,"container":"Oxford University Press","subjects":["World War, 1914-1918"],"language":"eng","pages":154,"cover_id":1123124}
|
||||||
|
{"v":1,"id":"9780199205592","kind":"isbn","found":true,"source":"openlibrary","title":"The First World War: A Very Short Introduction","authors":["Michael Howard"],"year":2007,"container":"Oxford University Press","language":"eng","cover_id":14606475}
|
||||||
|
```
|
||||||
|
|
||||||
|
A book that arrives as a file often has no identifier to ask about, only a
|
||||||
|
title and perhaps a name. This asks which books a register has under those,
|
||||||
|
and prints one record per candidate: Open Library's index first, and Google
|
||||||
|
Books only when Open Library has nothing.
|
||||||
|
|
||||||
|
Each candidate is one edition. Open Library's index answers with works, and
|
||||||
|
names the edition of each that best matches the search; what belongs to an
|
||||||
|
edition — its ISBN, publisher, language, year and cover — is that edition's,
|
||||||
|
and what belongs to the work — the title, the authors, the subjects and the
|
||||||
|
length — is the work's. Google Books answers with volumes, which are editions
|
||||||
|
already.
|
||||||
|
|
||||||
|
| Field | When the candidate has an ISBN | When it has none |
|
||||||
|
|---|---|---|
|
||||||
|
| `id` | The first ISBN whose check digit agrees, in thirteen digits; one written in thirteen is taken over one written in ten | The register's own key: an Open Library id such as `OL37863121M` for the edition, or `OL267171W` for the work when no edition was named; a Google Books volume id |
|
||||||
|
| `kind` | `isbn` | `olid` for Open Library, `googlebooks` for Google Books |
|
||||||
|
|
||||||
|
An ISBN whose check digit is wrong is passed over rather than corrected, and
|
||||||
|
Google's shelfmarks, typed `OTHER`, are not read as ISBNs even when their
|
||||||
|
digits would pass for one. A record of kind `isbn` can be handed straight back
|
||||||
|
to `bibfetch` for everything the registers say about that edition; `olid` and
|
||||||
|
`googlebooks` are names no lookup here answers about.
|
||||||
|
|
||||||
|
`cover_id` is the same thing it is in a lookup record: the first cover in
|
||||||
|
Open Library's record of the edition the record names, by that edition's ISBN
|
||||||
|
or its key, so `bibfetch cover --cover-id` fetches that edition's cover and no
|
||||||
|
other. The cover the index picked for the whole work is never asked for,
|
||||||
|
because it is often another edition's. A work the index names no edition of
|
||||||
|
gives the first ISBN it lists that reads, which may be any edition's, and no
|
||||||
|
cover id; Google Books knows nothing of Open Library's covers and gives none.
|
||||||
|
|
||||||
|
The candidates are what the register said, in the order it said them. This
|
||||||
|
makes no judgement about which one is the book in hand, and keeps one that is
|
||||||
|
plainly another book, because a title is not an identifier: *The Hamlet* comes
|
||||||
|
back for *Hamlet*. Deciding whether a candidate is the same work is a
|
||||||
|
comparison of titles, and [`bibclean`](../bibclean/README.md)'s `same_work`
|
||||||
|
is one.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
bibfetch search --title 'Anathem' --author Stephenson \
|
||||||
|
| jq -r 'select(.kind == "isbn") | .id' | head -1
|
||||||
|
```
|
||||||
|
|
||||||
|
Open Library answering that it has nothing is an answer, and it stands when
|
||||||
|
Google Books cannot be reached; a register being down is a failure only when
|
||||||
|
both are. Each is asked the same way a lookup asks, retried with the same
|
||||||
|
waits. Open Library is sent the title and the author as written; Google Books
|
||||||
|
is sent each word as its own `intitle:` or `inauthor:` term, so nothing in a
|
||||||
|
title is read as search syntax.
|
||||||
|
|
||||||
|
| Option | Effect |
|
||||||
|
|---|---|
|
||||||
|
| `--title <TITLE>` | The title, or as much of it as you have |
|
||||||
|
| `--author <AUTHOR>` | An author's name, or part of one |
|
||||||
|
| `--limit <COUNT>` | The most candidates to print, `1` to `40`. Default `10` |
|
||||||
|
|
||||||
|
Exit `0` at least one candidate was printed, `1` no register has the title,
|
||||||
|
`65` the title was empty, `69` no register could be reached, `74` the output
|
||||||
|
could not be written.
|
||||||
|
|
||||||
|
From Rust, `bibfetch::search(&agent, title, author, limit, attempts)` returns
|
||||||
|
the candidates, empty when no register has the title.
|
||||||
|
|
||||||
## Reaching the network
|
## Reaching the network
|
||||||
|
|
||||||
This and [`emb`](../emb/README.md) are the only two tools here that do. Both
|
This and [`emb`](../emb/README.md) are the only two tools here that do. Both
|
||||||
@@ -156,10 +229,11 @@ something that happens on the way to something else — which is the condition
|
|||||||
under which [the contract](../../docs/contract.md) allows it at all.
|
under which [the contract](../../docs/contract.md) allows it at all.
|
||||||
|
|
||||||
Nothing is cached, nothing is stored but a cover you asked to have written,
|
Nothing is cached, nothing is stored but a cover you asked to have written,
|
||||||
and nothing is sent but the identifier and the address you pass to
|
and nothing is sent but the identifier, or the title and author searched for,
|
||||||
`--polite`.
|
and the address you pass to `--polite`.
|
||||||
|
|
||||||
## See also
|
## See also
|
||||||
|
|
||||||
- [`idnorm`](../idnorm/README.md) — reading the identifier in the first place.
|
- [`idnorm`](../idnorm/README.md) — reading the identifier in the first place.
|
||||||
- [`bibclean`](../bibclean/README.md) — the rest of a catalogue record.
|
- [`bibclean`](../bibclean/README.md) — the rest of a catalogue record, and
|
||||||
|
whether a candidate is the same work.
|
||||||
+30
-25
@@ -89,10 +89,6 @@ pub fn lookup(
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Ask a register, retrying a failure that might not be one.
|
/// Ask a register, retrying a failure that might not be one.
|
||||||
///
|
|
||||||
/// Waits a second before the second attempt and two before the third: long
|
|
||||||
/// enough for a register that is rate-limiting, short enough that one which
|
|
||||||
/// is simply down does not hold up a pipeline.
|
|
||||||
fn ask(
|
fn ask(
|
||||||
agent: &ureq::Agent,
|
agent: &ureq::Agent,
|
||||||
source: Source,
|
source: Source,
|
||||||
@@ -100,39 +96,48 @@ fn ask(
|
|||||||
kind: &str,
|
kind: &str,
|
||||||
attempts: u32,
|
attempts: u32,
|
||||||
) -> Result<crate::Record, Error> {
|
) -> Result<crate::Record, Error> {
|
||||||
let mut last = Error::Unreachable("no attempt was made".into());
|
match get(agent, &source.url(id), attempts)? {
|
||||||
|
Some(body) => source.read(id, kind, &body),
|
||||||
|
// A register that does not have it says so with a status, and that is
|
||||||
|
// an answer rather than a failure.
|
||||||
|
None => Ok(crate::missing(id, kind, source)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
for attempt in 0..attempts.max(1) {
|
/// The body at an address, or `None` when the register says it has no such
|
||||||
match fetch(agent, source, id, kind) {
|
/// thing; a failure is asked again.
|
||||||
Ok(record) => return Ok(record),
|
///
|
||||||
// A shape this cannot read will not read any better next time.
|
/// Waits a second before the second attempt and two before the third: long
|
||||||
Err(e @ Error::Shape(_)) => return Err(e),
|
/// enough for a register that is rate-limiting, short enough that one which
|
||||||
|
/// is simply down does not hold up a pipeline. Reading what came back is left
|
||||||
|
/// to the caller, because an answer this cannot read will not read any better
|
||||||
|
/// next time.
|
||||||
|
pub(crate) fn get(agent: &ureq::Agent, url: &str, attempts: u32) -> Result<Option<String>, Error> {
|
||||||
|
let attempts = attempts.max(1);
|
||||||
|
let mut last = Error::Unreachable("no attempt was made".into());
|
||||||
|
for attempt in 0..attempts {
|
||||||
|
match once(agent, url) {
|
||||||
|
Ok(answer) => return Ok(answer),
|
||||||
Err(e) => last = e,
|
Err(e) => last = e,
|
||||||
}
|
}
|
||||||
std::thread::sleep(Duration::from_secs(1 << attempt));
|
if attempt + 1 < attempts {
|
||||||
|
std::thread::sleep(Duration::from_secs(1 << attempt.min(5)));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
Err(last)
|
Err(last)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn fetch(
|
fn once(agent: &ureq::Agent, url: &str) -> Result<Option<String>, Error> {
|
||||||
agent: &ureq::Agent,
|
let mut response = match agent.get(url).call() {
|
||||||
source: Source,
|
|
||||||
id: &str,
|
|
||||||
kind: &str,
|
|
||||||
) -> Result<crate::Record, Error> {
|
|
||||||
let response = agent.get(source.url(id)).call();
|
|
||||||
let mut response = match response {
|
|
||||||
Ok(response) => response,
|
Ok(response) => response,
|
||||||
// A register that does not have it says so with a status, and that is
|
Err(ureq::Error::StatusCode(404)) => return Ok(None),
|
||||||
// an answer rather than a failure.
|
|
||||||
Err(ureq::Error::StatusCode(404)) => return Ok(crate::missing(id, kind, source)),
|
|
||||||
Err(e) => return Err(Error::Unreachable(e.to_string())),
|
Err(e) => return Err(Error::Unreachable(e.to_string())),
|
||||||
};
|
};
|
||||||
let body = response
|
response
|
||||||
.body_mut()
|
.body_mut()
|
||||||
.read_to_string()
|
.read_to_string()
|
||||||
.map_err(|e| Error::Unreachable(e.to_string()))?;
|
.map(Some)
|
||||||
source.read(id, kind, &body)
|
.map_err(|e| Error::Unreachable(e.to_string()))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Who is asking.
|
/// Who is asking.
|
||||||
|
|||||||
@@ -12,9 +12,14 @@
|
|||||||
//! them: Crossref for a DOI, the arXiv for a preprint, OpenLibrary for a
|
//! them: Crossref for a DOI, the arXiv for a preprint, OpenLibrary for a
|
||||||
//! book. There is no register for an ISSN that answers without a key, so an
|
//! book. There is no register for an ISSN that answers without a key, so an
|
||||||
//! ISSN is looked up as the journal of a work rather than on its own.
|
//! ISSN is looked up as the journal of a work rather than on its own.
|
||||||
|
//!
|
||||||
|
//! A book with no identifier at all can still be looked for by its title,
|
||||||
|
//! with [`search()`]: Open Library's index, then Google Books, each candidate
|
||||||
|
//! a record of one edition, in the order the register gave them.
|
||||||
|
|
||||||
pub mod ask;
|
pub mod ask;
|
||||||
pub mod cover;
|
pub mod cover;
|
||||||
|
pub mod search;
|
||||||
pub mod source;
|
pub mod source;
|
||||||
|
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
@@ -23,6 +28,7 @@ use schemars::JsonSchema;
|
|||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
pub use cover::{Cover, CoverSize, cover};
|
pub use cover::{Cover, CoverSize, cover};
|
||||||
|
pub use search::search;
|
||||||
pub use source::Source;
|
pub use source::Source;
|
||||||
|
|
||||||
/// How long to wait on a register before giving up.
|
/// How long to wait on a register before giving up.
|
||||||
|
|||||||
@@ -30,7 +30,8 @@ Example:
|
|||||||
|
|
||||||
idnorm refs.txt | bibfetch | jq -r 'select(.found) | .title'
|
idnorm refs.txt | bibfetch | jq -r 'select(.found) | .title'
|
||||||
idnorm --kind doi refs.txt | bibfetch --polite you@example.org
|
idnorm --kind doi refs.txt | bibfetch --polite you@example.org
|
||||||
bibfetch cover 978-0-306-40615-7 -o cover.jpg"
|
bibfetch cover 978-0-306-40615-7 -o cover.jpg
|
||||||
|
bibfetch search --title 'The First World War' --author Howard"
|
||||||
)]
|
)]
|
||||||
struct Args {
|
struct Args {
|
||||||
#[command(subcommand)]
|
#[command(subcommand)]
|
||||||
@@ -74,6 +75,26 @@ enum Command {
|
|||||||
#[arg(long, value_name = "ID")]
|
#[arg(long, value_name = "ID")]
|
||||||
cover_id: Option<u64>,
|
cover_id: Option<u64>,
|
||||||
},
|
},
|
||||||
|
/// Find the books a register has under a title, and an author.
|
||||||
|
///
|
||||||
|
/// Prints one record per candidate edition, in the order the register
|
||||||
|
/// gave them: Open Library's index, or Google Books when that has
|
||||||
|
/// nothing. Which one is the book in hand is left to you.
|
||||||
|
///
|
||||||
|
/// Exit codes: 0 at least one candidate, 1 none, 65 no title,
|
||||||
|
/// 69 no register could be reached, 74 the output could not be written.
|
||||||
|
Search {
|
||||||
|
/// The title, or as much of it as you have.
|
||||||
|
#[arg(long)]
|
||||||
|
title: String,
|
||||||
|
/// An author's name, or part of one.
|
||||||
|
#[arg(long)]
|
||||||
|
author: Option<String>,
|
||||||
|
/// The most candidates to print.
|
||||||
|
#[arg(long, default_value_t = 10, value_name = "COUNT",
|
||||||
|
value_parser = clap::value_parser!(u16).range(1..=bibfetch::search::MAX_LIMIT as i64))]
|
||||||
|
limit: u16,
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Clone, Copy, clap::ValueEnum)]
|
#[derive(Clone, Copy, clap::ValueEnum)]
|
||||||
@@ -96,14 +117,27 @@ fn main() -> ExitCode {
|
|||||||
}
|
}
|
||||||
.agent();
|
.agent();
|
||||||
|
|
||||||
if let Some(Command::Cover {
|
match &args.command {
|
||||||
isbn,
|
Some(Command::Cover {
|
||||||
output,
|
isbn,
|
||||||
size,
|
output,
|
||||||
cover_id,
|
size,
|
||||||
}) = &args.command
|
cover_id,
|
||||||
{
|
}) => return cover(&agent, args.attempts, isbn, output, *size, *cover_id),
|
||||||
return cover(&agent, args.attempts, isbn, output, *size, *cover_id);
|
Some(Command::Search {
|
||||||
|
title,
|
||||||
|
author,
|
||||||
|
limit,
|
||||||
|
}) => {
|
||||||
|
return search(
|
||||||
|
&agent,
|
||||||
|
args.attempts,
|
||||||
|
title,
|
||||||
|
author.as_deref(),
|
||||||
|
usize::from(*limit),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
None => {}
|
||||||
}
|
}
|
||||||
|
|
||||||
let text = match catena_cli::read_text(args.path.as_deref()) {
|
let text = match catena_cli::read_text(args.path.as_deref()) {
|
||||||
@@ -197,6 +231,36 @@ fn cover(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Search by title and print the candidates.
|
||||||
|
fn search(
|
||||||
|
agent: &ureq::Agent,
|
||||||
|
attempts: u32,
|
||||||
|
title: &str,
|
||||||
|
author: Option<&str>,
|
||||||
|
limit: usize,
|
||||||
|
) -> ExitCode {
|
||||||
|
match bibfetch::search(agent, title, author, limit, attempts) {
|
||||||
|
Ok(found) => {
|
||||||
|
let wrote = catena_cli::stream("bibfetch", &found);
|
||||||
|
if format!("{wrote:?}") != format!("{:?}", ExitCode::from(exit::OK)) {
|
||||||
|
return wrote;
|
||||||
|
}
|
||||||
|
if found.is_empty() {
|
||||||
|
eprintln!("bibfetch: no register has {title:?}");
|
||||||
|
return ExitCode::from(exit::NO);
|
||||||
|
}
|
||||||
|
ExitCode::from(exit::OK)
|
||||||
|
}
|
||||||
|
Err(why) => {
|
||||||
|
eprintln!("bibfetch: {why}");
|
||||||
|
ExitCode::from(match why {
|
||||||
|
bibfetch::Error::Invalid(_) => exit::DATA,
|
||||||
|
_ => exit::UNAVAILABLE,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// The identifier and its kind out of one record.
|
/// The identifier and its kind out of one record.
|
||||||
fn read_identifier(line: &str) -> Result<(String, String), String> {
|
fn read_identifier(line: &str) -> Result<(String, String), String> {
|
||||||
let value: serde_json::Value =
|
let value: serde_json::Value =
|
||||||
|
|||||||
@@ -0,0 +1,780 @@
|
|||||||
|
//! Books by their title, for when there is no identifier to ask about.
|
||||||
|
//!
|
||||||
|
//! A book that arrives as a file often carries no ISBN at all, only a title
|
||||||
|
//! and perhaps a name. A register can still be asked which books it has under
|
||||||
|
//! those, and what it answers is a list of candidates: each an edition the
|
||||||
|
//! register holds, with whatever identifier, publisher and cover it gives.
|
||||||
|
//!
|
||||||
|
//! Which candidate is the book in hand is not decided here. The candidates
|
||||||
|
//! are what the register said, in the order it said them; deciding whether
|
||||||
|
//! one is the same work is a comparison of titles, and that belongs to
|
||||||
|
//! whoever holds the book.
|
||||||
|
|
||||||
|
use serde::Deserialize;
|
||||||
|
|
||||||
|
use crate::source::{self, GoogleIdentifier, GoogleReply, SUBJECT_LIMIT, Source};
|
||||||
|
use crate::{Error, Record};
|
||||||
|
|
||||||
|
/// The most candidates one search asks for. Google Books answers with no more
|
||||||
|
/// than forty at a time, and a title that needs more than forty candidates to
|
||||||
|
/// find the right one needs a better title.
|
||||||
|
pub const MAX_LIMIT: usize = 40;
|
||||||
|
|
||||||
|
/// What Open Library's index is asked to say about each work, and about the
|
||||||
|
/// one edition of it that best matches the search.
|
||||||
|
///
|
||||||
|
/// Not the work's own `cover_i` or `cover_edition_key`: the cover the index
|
||||||
|
/// chose for a work is often another edition's. The edition's `cover_i` is
|
||||||
|
/// the first cover in that edition's record, which is what a lookup by ISBN
|
||||||
|
/// gives as `cover_id`.
|
||||||
|
const OPEN_LIBRARY_FIELDS: &str = "key,title,subtitle,author_name,first_publish_year,publisher,\
|
||||||
|
isbn,language,subject,number_of_pages_median,editions,editions.key,editions.isbn,\
|
||||||
|
editions.publisher,editions.language,editions.publish_date,editions.cover_i";
|
||||||
|
|
||||||
|
/// Ask which books a register has under a title, and by an author if one is
|
||||||
|
/// named.
|
||||||
|
///
|
||||||
|
/// Open Library's index first, and Google Books only when that answered with
|
||||||
|
/// nothing. A register being down is a failure only when every register
|
||||||
|
/// asked is: Open Library answering that it has nothing is an answer, and it
|
||||||
|
/// stands when Google Books cannot be reached. Asks nothing when `limit` is
|
||||||
|
/// zero; a `limit` over [`MAX_LIMIT`] is taken as that.
|
||||||
|
pub fn search(
|
||||||
|
agent: &ureq::Agent,
|
||||||
|
title: &str,
|
||||||
|
author: Option<&str>,
|
||||||
|
limit: usize,
|
||||||
|
attempts: u32,
|
||||||
|
) -> Result<Vec<Record>, Error> {
|
||||||
|
search_with(title, author, limit, |url| {
|
||||||
|
crate::ask::get(agent, url, attempts)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// [`search`], with how to fetch an address passed in.
|
||||||
|
fn search_with(
|
||||||
|
title: &str,
|
||||||
|
author: Option<&str>,
|
||||||
|
limit: usize,
|
||||||
|
mut get: impl FnMut(&str) -> Result<Option<String>, Error>,
|
||||||
|
) -> Result<Vec<Record>, Error> {
|
||||||
|
let title = title.trim();
|
||||||
|
if title.is_empty() {
|
||||||
|
return Err(Error::Invalid("no title to search for".into()));
|
||||||
|
}
|
||||||
|
let author = author.map(str::trim).filter(|author| !author.is_empty());
|
||||||
|
let limit = limit.min(MAX_LIMIT);
|
||||||
|
if limit == 0 {
|
||||||
|
return Ok(Vec::new());
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut fetch = |url: String| match get(&url)? {
|
||||||
|
Some(body) => Ok(body),
|
||||||
|
// A search that matches nothing answers with an empty list. A 404 is
|
||||||
|
// the address being wrong, not the book being absent.
|
||||||
|
None => Err(Error::Unreachable(format!("{url}: answered 404"))),
|
||||||
|
};
|
||||||
|
|
||||||
|
let open_library =
|
||||||
|
fetch(open_library_url(title, author, limit)).and_then(|body| open_library(&body, limit));
|
||||||
|
if open_library.as_ref().is_ok_and(|found| !found.is_empty()) {
|
||||||
|
return open_library;
|
||||||
|
}
|
||||||
|
|
||||||
|
let google = match google_url(title, author, limit) {
|
||||||
|
Some(url) => fetch(url).and_then(|body| google_books(&body, limit)),
|
||||||
|
// Nothing in the title Google could be asked by.
|
||||||
|
None => Ok(Vec::new()),
|
||||||
|
};
|
||||||
|
match (open_library, google) {
|
||||||
|
(_, Ok(found)) if !found.is_empty() => Ok(found),
|
||||||
|
(Ok(none), _) => Ok(none),
|
||||||
|
(Err(why), _) => Err(why),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Where to ask Open Library.
|
||||||
|
fn open_library_url(title: &str, author: Option<&str>, limit: usize) -> String {
|
||||||
|
let author = author
|
||||||
|
.map(|author| format!("&author={}", source::encode(author)))
|
||||||
|
.unwrap_or_default();
|
||||||
|
format!(
|
||||||
|
"https://openlibrary.org/search.json?title={}{author}&fields={OPEN_LIBRARY_FIELDS}&limit={limit}",
|
||||||
|
source::encode(title),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Where to ask Google Books, or `None` when there is nothing to ask by.
|
||||||
|
///
|
||||||
|
/// Every word is its own `intitle:` or `inauthor:` term, so no character of
|
||||||
|
/// the title can open a phrase, negate a term or name another field: a word is
|
||||||
|
/// the letters, digits and apostrophes in it, and anything else separates
|
||||||
|
/// words.
|
||||||
|
fn google_url(title: &str, author: Option<&str>, limit: usize) -> Option<String> {
|
||||||
|
let terms: Vec<String> = words(title)
|
||||||
|
.map(|word| format!("intitle:{word}"))
|
||||||
|
.chain(
|
||||||
|
author
|
||||||
|
.into_iter()
|
||||||
|
.flat_map(words)
|
||||||
|
.map(|word| format!("inauthor:{word}")),
|
||||||
|
)
|
||||||
|
.collect();
|
||||||
|
if !terms.iter().any(|term| term.starts_with("intitle:")) {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
Some(format!(
|
||||||
|
"https://www.googleapis.com/books/v1/volumes?q={}&maxResults={limit}",
|
||||||
|
source::encode(&terms.join(" "))
|
||||||
|
))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn words(text: &str) -> impl Iterator<Item = &str> {
|
||||||
|
text.split(|c: char| !(c.is_alphanumeric() || c == '\''))
|
||||||
|
.map(|word| word.trim_matches('\''))
|
||||||
|
.filter(|word| !word.is_empty())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The first ISBN in a list whose check digit agrees, in its thirteen-digit
|
||||||
|
/// form. One written in thirteen digits is taken over one written in ten.
|
||||||
|
fn first_isbn<'a>(isbns: impl Iterator<Item = &'a str> + Clone) -> Option<String> {
|
||||||
|
isbns
|
||||||
|
.clone()
|
||||||
|
.filter(|isbn| idnorm::isbn::digits(isbn).len() == 13)
|
||||||
|
.find_map(idnorm::isbn::to_13)
|
||||||
|
.or_else(|| isbns.into_iter().find_map(idnorm::isbn::to_13))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct OpenLibraryReply {
|
||||||
|
#[serde(default)]
|
||||||
|
docs: Vec<Work>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct Work {
|
||||||
|
#[serde(default)]
|
||||||
|
key: Option<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
title: Option<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
subtitle: Option<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
author_name: Vec<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
first_publish_year: Option<i64>,
|
||||||
|
#[serde(default)]
|
||||||
|
publisher: Vec<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
isbn: Vec<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
language: Vec<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
subject: Vec<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
number_of_pages_median: Option<u32>,
|
||||||
|
#[serde(default)]
|
||||||
|
editions: Option<Editions>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct Editions {
|
||||||
|
#[serde(default)]
|
||||||
|
docs: Vec<Edition>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct Edition {
|
||||||
|
#[serde(default)]
|
||||||
|
key: Option<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
isbn: Vec<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
publisher: Vec<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
language: Vec<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
publish_date: Vec<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
cover_i: Option<i64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The last segment of an Open Library key: `OL3585423M` out of
|
||||||
|
/// `/books/OL3585423M`. The letter at the end says what it names, `M` an
|
||||||
|
/// edition and `W` a work.
|
||||||
|
fn olid(key: &str) -> Option<String> {
|
||||||
|
key.rsplit('/')
|
||||||
|
.next()
|
||||||
|
.filter(|id| !id.is_empty())
|
||||||
|
.map(str::to_string)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read Open Library's answer into one candidate per work, in its order.
|
||||||
|
///
|
||||||
|
/// What belongs to an edition — its ISBN, publisher, language, year and
|
||||||
|
/// cover — is taken from the edition the index chose as matching best; what
|
||||||
|
/// belongs to the work — its title, authors, subjects and length — from the
|
||||||
|
/// work. A work the index names no edition of gives its first ISBN that reads,
|
||||||
|
/// which may be any edition's, and no cover id.
|
||||||
|
fn open_library(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
|
||||||
|
let reply: OpenLibraryReply =
|
||||||
|
serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?;
|
||||||
|
Ok(reply
|
||||||
|
.docs
|
||||||
|
.into_iter()
|
||||||
|
.filter_map(candidate)
|
||||||
|
.take(limit)
|
||||||
|
.collect())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn candidate(work: Work) -> Option<Record> {
|
||||||
|
let edition = work.editions.and_then(|e| e.docs.into_iter().next());
|
||||||
|
|
||||||
|
// The cover is kept only when the record names the edition it is the
|
||||||
|
// cover of, by that edition's ISBN or its key.
|
||||||
|
let (id, kind, edition_named) = match &edition {
|
||||||
|
Some(edition) => match first_isbn(edition.isbn.iter().map(String::as_str)) {
|
||||||
|
Some(isbn) => (isbn, "isbn", true),
|
||||||
|
None => match edition.key.as_deref().and_then(olid) {
|
||||||
|
Some(key) => (key, "olid", true),
|
||||||
|
None => (work.key.as_deref().and_then(olid)?, "olid", false),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
None => match first_isbn(work.isbn.iter().map(String::as_str)) {
|
||||||
|
Some(isbn) => (isbn, "isbn", false),
|
||||||
|
// A work with no key cannot be named, so it is not a candidate.
|
||||||
|
None => (work.key.as_deref().and_then(olid)?, "olid", false),
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut record = source::blank(&id, kind, Source::OpenLibrarySearch);
|
||||||
|
record.title = match (work.title, work.subtitle) {
|
||||||
|
(Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")),
|
||||||
|
(title, _) => title,
|
||||||
|
};
|
||||||
|
record.authors = work.author_name;
|
||||||
|
record.subjects = work.subject;
|
||||||
|
record.subjects.truncate(SUBJECT_LIMIT);
|
||||||
|
record.pages = work.number_of_pages_median;
|
||||||
|
|
||||||
|
let (publisher, language, year) = match edition {
|
||||||
|
Some(edition) => {
|
||||||
|
if edition_named {
|
||||||
|
record.cover_id = edition
|
||||||
|
.cover_i
|
||||||
|
.and_then(|id| u64::try_from(id).ok())
|
||||||
|
.filter(|id| *id > 0);
|
||||||
|
}
|
||||||
|
(
|
||||||
|
edition.publisher.into_iter().next(),
|
||||||
|
edition.language.into_iter().next(),
|
||||||
|
edition
|
||||||
|
.publish_date
|
||||||
|
.iter()
|
||||||
|
.find_map(|date| source::year_in(date)),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
None => (None, None, None),
|
||||||
|
};
|
||||||
|
record.container = publisher.or_else(|| work.publisher.into_iter().next());
|
||||||
|
record.language = language.or_else(|| work.language.into_iter().next());
|
||||||
|
record.year = year.or(work.first_publish_year);
|
||||||
|
Some(record)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read Google Books' answer into one candidate per volume, in its order.
|
||||||
|
fn google_books(body: &str, limit: usize) -> Result<Vec<Record>, Error> {
|
||||||
|
let reply: GoogleReply = serde_json::from_str(body).map_err(|e| Error::Shape(e.to_string()))?;
|
||||||
|
Ok(reply
|
||||||
|
.items
|
||||||
|
.into_iter()
|
||||||
|
.filter_map(|item| {
|
||||||
|
// Only what Google types as an ISBN: its `OTHER` identifiers are
|
||||||
|
// library shelfmarks, some of which are ten digits long.
|
||||||
|
let isbns = item
|
||||||
|
.volume_info
|
||||||
|
.identifiers
|
||||||
|
.iter()
|
||||||
|
.filter(|GoogleIdentifier { kind, .. }| kind == "ISBN_13" || kind == "ISBN_10")
|
||||||
|
.map(|identifier| identifier.identifier.as_str());
|
||||||
|
let (id, kind) = match first_isbn(isbns) {
|
||||||
|
Some(isbn) => (isbn, "isbn"),
|
||||||
|
None => (item.id.filter(|id| !id.is_empty())?, "googlebooks"),
|
||||||
|
};
|
||||||
|
Some(source::google_volume(&id, kind, item.volume_info))
|
||||||
|
})
|
||||||
|
.take(limit)
|
||||||
|
.collect())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
/// Open Library's answer to `title=the first world war&author=howard`,
|
||||||
|
/// recorded 2026-11-13 with the fields above and `limit=2`, the ISBN and
|
||||||
|
/// subject lists shortened.
|
||||||
|
const HOWARD: &str = r#"{"numFound": 9, "start": 0, "numFoundExact": true, "num_found": 9,
|
||||||
|
"documentation_url": "https://openlibrary.org/dev/docs/api/search", "q": "", "offset": null,
|
||||||
|
"docs": [
|
||||||
|
{"author_name": ["Michael Eliot Howard"], "first_publish_year": 2002,
|
||||||
|
"isbn": ["9780192853622", "0192853627"], "key": "/works/OL1382726W",
|
||||||
|
"language": ["eng"], "number_of_pages_median": 154,
|
||||||
|
"publisher": ["Oxford University Press"], "title": "The First World War",
|
||||||
|
"subject": ["World War, 1914-1918"],
|
||||||
|
"editions": {"numFound": 1, "start": 0, "numFoundExact": true, "docs": [
|
||||||
|
{"key": "/books/OL3585423M", "cover_i": 1123124, "language": ["eng"],
|
||||||
|
"publisher": ["Oxford University Press"], "publish_date": ["2002"],
|
||||||
|
"isbn": ["0192853627", "9780192853622"]}]}},
|
||||||
|
{"author_name": ["Michael Howard"], "first_publish_year": 2007,
|
||||||
|
"isbn": ["0199205590", "9780199205592"], "key": "/works/OL4282399W",
|
||||||
|
"language": ["eng"], "publisher": ["Oxford University Press"],
|
||||||
|
"subtitle": "A Very Short Introduction", "title": "The First World War",
|
||||||
|
"editions": {"numFound": 1, "start": 0, "numFoundExact": true, "docs": [
|
||||||
|
{"key": "/books/OL7403004M", "cover_i": 14606475, "language": ["eng"],
|
||||||
|
"publisher": ["Oxford University Press"], "publish_date": ["20 May 2007"],
|
||||||
|
"isbn": ["9780199205592", "0199205590"]}]},
|
||||||
|
"subject": []}]}"#;
|
||||||
|
|
||||||
|
/// Open Library's answer to `title=war and peace&author=tolstoy`, recorded
|
||||||
|
/// 2026-11-13, first work only. The edition the index matched best is a
|
||||||
|
/// 1917 printing that has no ISBN, though other editions of the work do.
|
||||||
|
const TOLSTOY: &str = r#"{"numFound": 146, "docs": [
|
||||||
|
{"author_name": ["Leo Tolstoy"], "first_publish_year": 1867,
|
||||||
|
"isbn": ["9781524881993", "1645178889"], "key": "/works/OL267171W",
|
||||||
|
"title": "War and Peace",
|
||||||
|
"editions": {"numFound": 487, "docs": [
|
||||||
|
{"key": "/books/OL37863121M", "cover_i": 12736340, "language": ["eng"],
|
||||||
|
"publisher": ["Thomas Y. Crowell"], "publish_date": ["1917"]}]}}]}"#;
|
||||||
|
|
||||||
|
/// Open Library's answer when nothing has the title.
|
||||||
|
const NOTHING: &str = r#"{"numFound": 0, "start": 0, "numFoundExact": true, "num_found": 0,
|
||||||
|
"documentation_url": "https://openlibrary.org/dev/docs/api/search", "q": "",
|
||||||
|
"offset": null, "docs": []}"#;
|
||||||
|
|
||||||
|
/// Google Books' answer for a volume, in the shape its API documents.
|
||||||
|
const GOOGLE: &str = r#"{"kind": "books#volumes", "totalItems": 2, "items": [
|
||||||
|
{"kind": "books#volume", "id": "zyTCAlFPjgYC", "volumeInfo": {
|
||||||
|
"title": "The Google Story", "authors": ["David A. Vise", "Mark Malseed"],
|
||||||
|
"publisher": "Random House Digital, Inc.", "publishedDate": "2005-11-15",
|
||||||
|
"industryIdentifiers": [
|
||||||
|
{"type": "ISBN_10", "identifier": "055380457X"},
|
||||||
|
{"type": "ISBN_13", "identifier": "9780553804577"}],
|
||||||
|
"pageCount": 207, "categories": ["Browsers (Computer programs)"],
|
||||||
|
"language": "en"}},
|
||||||
|
{"kind": "books#volume", "id": "a1b2C3d4", "volumeInfo": {
|
||||||
|
"title": "The Google Story", "subtitle": "A Dissertation",
|
||||||
|
"publishedDate": "2009",
|
||||||
|
"industryIdentifiers": [
|
||||||
|
{"type": "OTHER", "identifier": "UOM:0306406152"}]}}]}"#;
|
||||||
|
|
||||||
|
/// Fetch from a table of recorded bodies, keeping every address asked.
|
||||||
|
fn recorded<'a>(
|
||||||
|
asked: &'a mut Vec<String>,
|
||||||
|
bodies: &'a [(&'a str, Result<&'a str, Error>)],
|
||||||
|
) -> impl FnMut(&str) -> Result<Option<String>, Error> + 'a {
|
||||||
|
move |url| {
|
||||||
|
asked.push(url.to_string());
|
||||||
|
let (_, body) = bodies
|
||||||
|
.iter()
|
||||||
|
.find(|(host, _)| url.contains(host))
|
||||||
|
.unwrap_or_else(|| panic!("asked an address nothing was recorded for: {url}"));
|
||||||
|
body.clone().map(|body| Some(body.to_string()))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn down() -> Error {
|
||||||
|
Error::Unreachable("connection refused".into())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn each_work_open_library_has_is_a_candidate_named_by_its_isbn() {
|
||||||
|
let mut asked = Vec::new();
|
||||||
|
let bodies = [("openlibrary.org", Ok(HOWARD))];
|
||||||
|
let found = search_with(
|
||||||
|
"The First World War",
|
||||||
|
Some("Howard"),
|
||||||
|
5,
|
||||||
|
recorded(&mut asked, &bodies),
|
||||||
|
)
|
||||||
|
.expect("candidates");
|
||||||
|
assert_eq!(
|
||||||
|
asked.len(),
|
||||||
|
1,
|
||||||
|
"Google is not asked when Open Library answers"
|
||||||
|
);
|
||||||
|
|
||||||
|
assert_eq!(found.len(), 2);
|
||||||
|
let first = &found[0];
|
||||||
|
assert!(first.found);
|
||||||
|
assert_eq!(first.v, corpus_core::V);
|
||||||
|
assert_eq!(first.source, "openlibrary");
|
||||||
|
// Written ten-digit first by the edition, given in thirteen.
|
||||||
|
assert_eq!(
|
||||||
|
(first.id.as_str(), first.kind.as_str()),
|
||||||
|
("9780192853622", "isbn")
|
||||||
|
);
|
||||||
|
assert_eq!(first.title.as_deref(), Some("The First World War"));
|
||||||
|
assert_eq!(first.authors, ["Michael Eliot Howard"]);
|
||||||
|
assert_eq!(first.year, Some(2002));
|
||||||
|
assert_eq!(first.container.as_deref(), Some("Oxford University Press"));
|
||||||
|
assert_eq!(first.language.as_deref(), Some("eng"));
|
||||||
|
assert_eq!(first.subjects, ["World War, 1914-1918"]);
|
||||||
|
assert_eq!(first.pages, Some(154));
|
||||||
|
assert_eq!(first.cover_id, Some(1123124));
|
||||||
|
|
||||||
|
let second = &found[1];
|
||||||
|
assert_eq!(second.id, "9780199205592");
|
||||||
|
assert_eq!(
|
||||||
|
second.title.as_deref(),
|
||||||
|
Some("The First World War: A Very Short Introduction")
|
||||||
|
);
|
||||||
|
// The edition's year, not the work's first.
|
||||||
|
assert_eq!(second.year, Some(2007));
|
||||||
|
// The first cover in OL7403004M's own record; the work's cover is
|
||||||
|
// 1137779, which is not.
|
||||||
|
assert_eq!(second.cover_id, Some(14606475));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_candidates_stay_in_the_order_the_register_gave_them() {
|
||||||
|
let mut asked = Vec::new();
|
||||||
|
let bodies = [("openlibrary.org", Ok(HOWARD))];
|
||||||
|
let found =
|
||||||
|
search_with("first world war", None, 5, recorded(&mut asked, &bodies)).expect("ok");
|
||||||
|
let ids: Vec<&str> = found.iter().map(|r| r.id.as_str()).collect();
|
||||||
|
assert_eq!(ids, ["9780192853622", "9780199205592"]);
|
||||||
|
let found =
|
||||||
|
search_with("first world war", None, 1, recorded(&mut asked, &bodies)).expect("ok");
|
||||||
|
assert_eq!(found.len(), 1);
|
||||||
|
assert_eq!(found[0].id, "9780192853622");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_edition_with_no_isbn_is_named_by_its_key_and_keeps_its_cover() {
|
||||||
|
let mut asked = Vec::new();
|
||||||
|
let bodies = [("openlibrary.org", Ok(TOLSTOY))];
|
||||||
|
let found = search_with(
|
||||||
|
"War and Peace",
|
||||||
|
Some("Tolstoy"),
|
||||||
|
5,
|
||||||
|
recorded(&mut asked, &bodies),
|
||||||
|
)
|
||||||
|
.expect("candidates");
|
||||||
|
let only = &found[0];
|
||||||
|
// Not 9781524881993: that is another edition's, and the publisher,
|
||||||
|
// year and cover here are this one's.
|
||||||
|
assert_eq!(
|
||||||
|
(only.id.as_str(), only.kind.as_str()),
|
||||||
|
("OL37863121M", "olid")
|
||||||
|
);
|
||||||
|
assert_eq!(only.container.as_deref(), Some("Thomas Y. Crowell"));
|
||||||
|
assert_eq!(only.year, Some(1917));
|
||||||
|
assert_eq!(only.cover_id, Some(12736340));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_work_with_no_edition_named_gives_its_isbn_but_no_cover() {
|
||||||
|
let body = r#"{"docs": [{"key": "/works/OL1W", "title": "A Book",
|
||||||
|
"isbn": ["0-306-40615-2"], "publisher": ["Plenum"], "first_publish_year": 1981}]}"#;
|
||||||
|
let found = open_library(body, 5).expect("candidates");
|
||||||
|
assert_eq!(
|
||||||
|
(found[0].id.as_str(), found[0].kind.as_str()),
|
||||||
|
("9780306406157", "isbn")
|
||||||
|
);
|
||||||
|
assert_eq!(found[0].container.as_deref(), Some("Plenum"));
|
||||||
|
assert_eq!(found[0].year, Some(1981));
|
||||||
|
assert_eq!(found[0].cover_id, None);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_work_with_no_isbn_and_no_edition_is_named_by_the_works_key() {
|
||||||
|
let body = r#"{"docs": [{"key": "/works/OL267171W", "title": "War and Peace"}]}"#;
|
||||||
|
let found = open_library(body, 5).expect("candidates");
|
||||||
|
assert_eq!(
|
||||||
|
(found[0].id.as_str(), found[0].kind.as_str()),
|
||||||
|
("OL267171W", "olid")
|
||||||
|
);
|
||||||
|
assert_eq!(found[0].cover_id, None);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn isbns_whose_check_digit_is_wrong_are_passed_over() {
|
||||||
|
// 9780306406158 and 0306406153 are 9780306406157 mistyped.
|
||||||
|
let body = r#"{"docs": [
|
||||||
|
{"key": "/works/OL1W", "title": "A", "editions": {"docs": [
|
||||||
|
{"key": "/books/OL1M", "isbn": ["9780306406158", "0306406153"], "cover_i": 7}]}},
|
||||||
|
{"key": "/works/OL2W", "title": "B", "editions": {"docs": [
|
||||||
|
{"key": "/books/OL2M", "isbn": ["9780306406158", "0306406152"]}]}}]}"#;
|
||||||
|
let found = open_library(body, 5).expect("candidates");
|
||||||
|
assert_eq!(
|
||||||
|
(found[0].id.as_str(), found[0].kind.as_str()),
|
||||||
|
("OL1M", "olid")
|
||||||
|
);
|
||||||
|
assert_eq!(found[0].cover_id, Some(7));
|
||||||
|
// The ten-digit one reads, and is given in thirteen.
|
||||||
|
assert_eq!(
|
||||||
|
(found[1].id.as_str(), found[1].kind.as_str()),
|
||||||
|
("9780306406157", "isbn")
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn thirteen_digits_are_taken_over_ten() {
|
||||||
|
// Two different ISBNs on one edition: the thirteen-digit one wins
|
||||||
|
// although the ten-digit one is listed first.
|
||||||
|
let isbns = ["0306406152", "9780199205592"];
|
||||||
|
assert_eq!(
|
||||||
|
first_isbn(isbns.into_iter()).as_deref(),
|
||||||
|
Some("9780199205592")
|
||||||
|
);
|
||||||
|
assert_eq!(first_isbn(["nonsense"].into_iter()), None);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn nothing_from_open_library_is_asked_of_google_books() {
|
||||||
|
let mut asked = Vec::new();
|
||||||
|
let bodies = [
|
||||||
|
("openlibrary.org", Ok(NOTHING)),
|
||||||
|
("googleapis.com", Ok(GOOGLE)),
|
||||||
|
];
|
||||||
|
let found = search_with(
|
||||||
|
"The Google Story",
|
||||||
|
Some("Vise"),
|
||||||
|
5,
|
||||||
|
recorded(&mut asked, &bodies),
|
||||||
|
)
|
||||||
|
.expect("candidates");
|
||||||
|
assert_eq!(asked.len(), 2);
|
||||||
|
assert!(asked[1].starts_with("https://www.googleapis.com/books/v1/volumes?q="));
|
||||||
|
|
||||||
|
assert_eq!(found.len(), 2);
|
||||||
|
let first = &found[0];
|
||||||
|
assert_eq!(first.source, "googlebooks");
|
||||||
|
assert_eq!(
|
||||||
|
(first.id.as_str(), first.kind.as_str()),
|
||||||
|
("9780553804577", "isbn")
|
||||||
|
);
|
||||||
|
assert_eq!(first.authors, ["David A. Vise", "Mark Malseed"]);
|
||||||
|
assert_eq!(first.year, Some(2005));
|
||||||
|
assert_eq!(
|
||||||
|
first.container.as_deref(),
|
||||||
|
Some("Random House Digital, Inc.")
|
||||||
|
);
|
||||||
|
assert_eq!(first.pages, Some(207));
|
||||||
|
// Google Books knows nothing of Open Library's covers.
|
||||||
|
assert_eq!(first.cover_id, None);
|
||||||
|
|
||||||
|
// A shelfmark is not an ISBN, though its digits would pass for one.
|
||||||
|
let second = &found[1];
|
||||||
|
assert_eq!(
|
||||||
|
(second.id.as_str(), second.kind.as_str()),
|
||||||
|
("a1b2C3d4", "googlebooks")
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
second.title.as_deref(),
|
||||||
|
Some("The Google Story: A Dissertation")
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn nothing_from_either_register_is_no_candidates() {
|
||||||
|
let mut asked = Vec::new();
|
||||||
|
let bodies = [
|
||||||
|
("openlibrary.org", Ok(NOTHING)),
|
||||||
|
(
|
||||||
|
"googleapis.com",
|
||||||
|
Ok(r#"{"kind": "books#volumes", "totalItems": 0}"#),
|
||||||
|
),
|
||||||
|
];
|
||||||
|
let found =
|
||||||
|
search_with("Nothing", None, 5, recorded(&mut asked, &bodies)).expect("an answer");
|
||||||
|
assert!(found.is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn open_library_having_nothing_stands_when_google_books_is_down() {
|
||||||
|
let mut asked = Vec::new();
|
||||||
|
let bodies = [
|
||||||
|
("openlibrary.org", Ok(NOTHING)),
|
||||||
|
("googleapis.com", Err(down())),
|
||||||
|
];
|
||||||
|
let found =
|
||||||
|
search_with("Nothing", None, 5, recorded(&mut asked, &bodies)).expect("an answer");
|
||||||
|
assert!(found.is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn open_library_down_is_asked_of_google_books() {
|
||||||
|
let mut asked = Vec::new();
|
||||||
|
let bodies = [
|
||||||
|
("openlibrary.org", Err(down())),
|
||||||
|
("googleapis.com", Ok(GOOGLE)),
|
||||||
|
];
|
||||||
|
let found =
|
||||||
|
search_with("The Google Story", None, 5, recorded(&mut asked, &bodies)).expect("ok");
|
||||||
|
assert_eq!(found[0].source, "googlebooks");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn every_register_down_is_a_failure() {
|
||||||
|
let mut asked = Vec::new();
|
||||||
|
let bodies = [
|
||||||
|
("openlibrary.org", Err(down())),
|
||||||
|
("googleapis.com", Err(down())),
|
||||||
|
];
|
||||||
|
let failed = search_with("A Book", None, 5, recorded(&mut asked, &bodies));
|
||||||
|
assert!(matches!(failed, Err(Error::Unreachable(_))), "{failed:?}");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_answer_that_cannot_be_read_says_so() {
|
||||||
|
assert!(matches!(open_library("<html>", 5), Err(Error::Shape(_))));
|
||||||
|
assert!(matches!(google_books("<html>", 5), Err(Error::Shape(_))));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_author_is_left_out_when_there_is_none() {
|
||||||
|
for author in [None, Some(""), Some(" ")] {
|
||||||
|
let url = open_library_url(
|
||||||
|
"Anathem",
|
||||||
|
author.map(str::trim).filter(|a| !a.is_empty()),
|
||||||
|
5,
|
||||||
|
);
|
||||||
|
assert!(!url.contains("author="), "{url}");
|
||||||
|
}
|
||||||
|
let mut asked = Vec::new();
|
||||||
|
let bodies = [
|
||||||
|
("openlibrary.org", Ok(NOTHING)),
|
||||||
|
("googleapis.com", Ok(GOOGLE)),
|
||||||
|
];
|
||||||
|
search_with("Anathem", Some(" "), 5, recorded(&mut asked, &bodies)).expect("ok");
|
||||||
|
assert!(!asked[0].contains("author="), "{}", asked[0]);
|
||||||
|
assert!(!asked[1].contains("inauthor"), "{}", asked[1]);
|
||||||
|
assert!(asked[1].contains("q=intitle%3AAnathem&"), "{}", asked[1]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_author_narrows_the_search_when_there_is_one() {
|
||||||
|
let url = open_library_url("Anathem", Some("Neal Stephenson"), 5);
|
||||||
|
assert!(
|
||||||
|
url.contains("?title=Anathem&author=Neal%20Stephenson&fields="),
|
||||||
|
"{url}"
|
||||||
|
);
|
||||||
|
let url = google_url("Anathem", Some("Neal Stephenson"), 5).expect("a url");
|
||||||
|
assert!(
|
||||||
|
url.contains("q=intitle%3AAnathem%20inauthor%3ANeal%20inauthor%3AStephenson&"),
|
||||||
|
"{url}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_title_cannot_change_the_shape_of_the_address() {
|
||||||
|
let hostile = r#"Fish & Chips: "A History" #2?=x/y+z"#;
|
||||||
|
for url in [
|
||||||
|
open_library_url(hostile, Some("O'Brien & Sons"), 5),
|
||||||
|
google_url(hostile, Some("O'Brien & Sons"), 5).expect("a url"),
|
||||||
|
] {
|
||||||
|
assert_eq!(url.matches('?').count(), 1, "{url}");
|
||||||
|
assert_eq!(url.matches('#').count(), 0, "{url}");
|
||||||
|
assert_eq!(url.matches('"').count(), 0, "{url}");
|
||||||
|
assert_eq!(url.matches('+').count(), 0, "{url}");
|
||||||
|
let query = url.split_once('?').expect("a query").1;
|
||||||
|
let names: Vec<&str> = query
|
||||||
|
.split('&')
|
||||||
|
.map(|pair| pair.split_once('=').expect("name=value").0)
|
||||||
|
.collect();
|
||||||
|
assert!(
|
||||||
|
names == ["title", "author", "fields", "limit"] || names == ["q", "maxResults"],
|
||||||
|
"{url}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// Open Library is asked the title as written.
|
||||||
|
let url = open_library_url(hostile, None, 5);
|
||||||
|
assert!(
|
||||||
|
url.contains("title=Fish%20%26%20Chips%3A%20%22A%20History%22%20%232%3F%3Dx%2Fy%2Bz&"),
|
||||||
|
"{url}"
|
||||||
|
);
|
||||||
|
// Google is asked word by word, so nothing in a title is syntax.
|
||||||
|
let url = google_url(hostile, Some("O'Brien & Sons"), 5).expect("a url");
|
||||||
|
let q = "intitle:Fish intitle:Chips intitle:A intitle:History intitle:2 intitle:x \
|
||||||
|
intitle:y intitle:z inauthor:O'Brien inauthor:Sons";
|
||||||
|
assert!(url.contains(&format!("q={}&", source::encode(q))), "{url}");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_title_outside_ascii_is_sent_as_utf8() {
|
||||||
|
let url = open_library_url("Преступление и наказание", Some("Dostoevský"), 5);
|
||||||
|
assert!(
|
||||||
|
url.contains("title=%D0%9F%D1%80%D0%B5%D1%81%D1%82%D1%83%D0%BF"),
|
||||||
|
"{url}"
|
||||||
|
);
|
||||||
|
assert!(url.contains("author=Dostoevsk%C3%BD&"), "{url}");
|
||||||
|
let url = google_url("Преступление и наказание", None, 5).expect("a url");
|
||||||
|
// Letters in any script are words.
|
||||||
|
assert!(url.contains("intitle%3A%D0%B8%20"), "{url}");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_title_with_no_words_is_not_asked_of_google_books() {
|
||||||
|
assert_eq!(google_url("?!", Some("Someone"), 5), None);
|
||||||
|
let mut asked = Vec::new();
|
||||||
|
let bodies = [("openlibrary.org", Ok(NOTHING))];
|
||||||
|
let found = search_with("?!", Some("Someone"), 5, recorded(&mut asked, &bodies))
|
||||||
|
.expect("an answer");
|
||||||
|
assert!(found.is_empty());
|
||||||
|
assert_eq!(asked.len(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn no_title_is_refused_without_asking() {
|
||||||
|
let mut asked = Vec::new();
|
||||||
|
let failed = search_with(" ", Some("Someone"), 5, recorded(&mut asked, &[]));
|
||||||
|
assert!(matches!(failed, Err(Error::Invalid(_))), "{failed:?}");
|
||||||
|
assert!(asked.is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_limit_is_asked_for_and_held_to() {
|
||||||
|
let mut asked = Vec::new();
|
||||||
|
let bodies = [
|
||||||
|
("openlibrary.org", Ok(NOTHING)),
|
||||||
|
("googleapis.com", Ok(GOOGLE)),
|
||||||
|
];
|
||||||
|
let found =
|
||||||
|
search_with("The Google Story", None, 1, recorded(&mut asked, &bodies)).expect("ok");
|
||||||
|
assert_eq!(found.len(), 1);
|
||||||
|
assert!(asked[0].ends_with("&limit=1"), "{}", asked[0]);
|
||||||
|
assert!(asked[1].ends_with("&maxResults=1"), "{}", asked[1]);
|
||||||
|
|
||||||
|
let mut asked = Vec::new();
|
||||||
|
search_with("The Google Story", None, 500, recorded(&mut asked, &bodies)).expect("ok");
|
||||||
|
assert!(asked[0].ends_with("&limit=40"), "{}", asked[0]);
|
||||||
|
assert!(asked[1].ends_with("&maxResults=40"), "{}", asked[1]);
|
||||||
|
|
||||||
|
let mut asked = Vec::new();
|
||||||
|
let found =
|
||||||
|
search_with("The Google Story", None, 0, recorded(&mut asked, &bodies)).expect("ok");
|
||||||
|
assert!(found.is_empty());
|
||||||
|
assert!(asked.is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_index_is_not_asked_for_the_works_cover() {
|
||||||
|
let url = open_library_url("A", None, 5);
|
||||||
|
let fields = url
|
||||||
|
.split('&')
|
||||||
|
.find_map(|pair| pair.strip_prefix("fields="))
|
||||||
|
.expect("fields");
|
||||||
|
let fields: Vec<&str> = fields.split(',').collect();
|
||||||
|
assert!(fields.contains(&"editions.cover_i"));
|
||||||
|
assert!(!fields.contains(&"cover_i"));
|
||||||
|
assert!(!fields.contains(&"cover_edition_key"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_404_from_a_search_is_a_failure_and_not_an_absence() {
|
||||||
|
let failed = search_with("A Book", None, 5, |_| Ok(None));
|
||||||
|
assert!(matches!(failed, Err(Error::Unreachable(_))), "{failed:?}");
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -117,7 +117,7 @@ number_of_pages_median&limit=1",
|
|||||||
///
|
///
|
||||||
/// A DOI suffix may hold a slash, a question mark or a hash, and every one of
|
/// A DOI suffix may hold a slash, a question mark or a hash, and every one of
|
||||||
/// those means something else in a URL.
|
/// those means something else in a URL.
|
||||||
fn encode(text: &str) -> String {
|
pub(crate) fn encode(text: &str) -> String {
|
||||||
let mut out = String::with_capacity(text.len());
|
let mut out = String::with_capacity(text.len());
|
||||||
for byte in text.bytes() {
|
for byte in text.bytes() {
|
||||||
match byte {
|
match byte {
|
||||||
@@ -131,7 +131,7 @@ fn encode(text: &str) -> String {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// A record with the fields nothing found fills in.
|
/// A record with the fields nothing found fills in.
|
||||||
fn blank(id: &str, kind: &str, source: Source) -> Record {
|
pub(crate) fn blank(id: &str, kind: &str, source: Source) -> Record {
|
||||||
let mut record = crate::missing(id, kind, source);
|
let mut record = crate::missing(id, kind, source);
|
||||||
record.found = true;
|
record.found = true;
|
||||||
record
|
record
|
||||||
@@ -320,7 +320,7 @@ struct SearchDoc {
|
|||||||
/// The index lists every heading every edition has ever been given, which on a
|
/// The index lists every heading every edition has ever been given, which on a
|
||||||
/// well-known book is hundreds and on Hamlet is thousands. A shelf is filed
|
/// well-known book is hundreds and on Hamlet is thousands. A shelf is filed
|
||||||
/// from a handful, and the first are the broadest.
|
/// from a handful, and the first are the broadest.
|
||||||
const SUBJECT_LIMIT: usize = 12;
|
pub(crate) const SUBJECT_LIMIT: usize = 12;
|
||||||
|
|
||||||
/// A search that matched nothing answers with no documents rather than a
|
/// A search that matched nothing answers with no documents rather than a
|
||||||
/// status, so an empty reply is the absence and not a failure.
|
/// status, so an empty reply is the absence and not a failure.
|
||||||
@@ -348,19 +348,22 @@ fn open_library_search(id: &str, kind: &str, body: &str) -> Result<Record, Error
|
|||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Deserialize)]
|
#[derive(Deserialize)]
|
||||||
struct GoogleReply {
|
pub(crate) struct GoogleReply {
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
items: Vec<GoogleItem>,
|
pub(crate) items: Vec<GoogleItem>,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Deserialize)]
|
#[derive(Deserialize)]
|
||||||
struct GoogleItem {
|
pub(crate) struct GoogleItem {
|
||||||
|
/// Google's own key for the volume.
|
||||||
|
#[serde(default)]
|
||||||
|
pub(crate) id: Option<String>,
|
||||||
#[serde(rename = "volumeInfo", default)]
|
#[serde(rename = "volumeInfo", default)]
|
||||||
volume_info: GoogleVolume,
|
pub(crate) volume_info: GoogleVolume,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Deserialize, Default)]
|
#[derive(Deserialize, Default)]
|
||||||
struct GoogleVolume {
|
pub(crate) struct GoogleVolume {
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
title: Option<String>,
|
title: Option<String>,
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
@@ -379,6 +382,18 @@ struct GoogleVolume {
|
|||||||
language: Option<String>,
|
language: Option<String>,
|
||||||
#[serde(rename = "pageCount", default)]
|
#[serde(rename = "pageCount", default)]
|
||||||
page_count: Option<u32>,
|
page_count: Option<u32>,
|
||||||
|
/// `[{"type": "ISBN_13", "identifier": "…"}]`, alongside library
|
||||||
|
/// shelfmarks typed `OTHER`.
|
||||||
|
#[serde(rename = "industryIdentifiers", default)]
|
||||||
|
pub(crate) identifiers: Vec<GoogleIdentifier>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
pub(crate) struct GoogleIdentifier {
|
||||||
|
#[serde(rename = "type", default)]
|
||||||
|
pub(crate) kind: String,
|
||||||
|
#[serde(default)]
|
||||||
|
pub(crate) identifier: String,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A search that matched nothing answers with no items rather than a status,
|
/// A search that matched nothing answers with no items rather than a status,
|
||||||
@@ -388,8 +403,11 @@ fn google_books(id: &str, kind: &str, body: &str) -> Result<Record, Error> {
|
|||||||
let Some(item) = reply.items.into_iter().next() else {
|
let Some(item) = reply.items.into_iter().next() else {
|
||||||
return Ok(crate::missing(id, kind, Source::GoogleBooks));
|
return Ok(crate::missing(id, kind, Source::GoogleBooks));
|
||||||
};
|
};
|
||||||
let volume = item.volume_info;
|
Ok(google_volume(id, kind, item.volume_info))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One Google Books volume as a record.
|
||||||
|
pub(crate) fn google_volume(id: &str, kind: &str, volume: GoogleVolume) -> Record {
|
||||||
let mut record = blank(id, kind, Source::GoogleBooks);
|
let mut record = blank(id, kind, Source::GoogleBooks);
|
||||||
record.title = match (volume.title, volume.subtitle) {
|
record.title = match (volume.title, volume.subtitle) {
|
||||||
(Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")),
|
(Some(title), Some(subtitle)) => Some(format!("{title}: {subtitle}")),
|
||||||
@@ -402,11 +420,11 @@ fn google_books(id: &str, kind: &str, body: &str) -> Result<Record, Error> {
|
|||||||
record.subjects = volume.categories;
|
record.subjects = volume.categories;
|
||||||
record.language = volume.language;
|
record.language = volume.language;
|
||||||
record.pages = volume.page_count;
|
record.pages = volume.page_count;
|
||||||
Ok(record)
|
record
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The first four-digit year in a string.
|
/// The first four-digit year in a string.
|
||||||
fn year_in(text: &str) -> Option<i64> {
|
pub(crate) fn year_in(text: &str) -> Option<i64> {
|
||||||
let bytes: Vec<char> = text.chars().collect();
|
let bytes: Vec<char> = text.chars().collect();
|
||||||
for window in bytes.windows(4) {
|
for window in bytes.windows(4) {
|
||||||
if window.iter().all(char::is_ascii_digit) {
|
if window.iter().all(char::is_ascii_digit) {
|
||||||
|
|||||||
+1
-1
@@ -64,7 +64,7 @@ No network, no database, no configuration file, no state kept between runs.
|
|||||||
Five exceptions, each declared in `--help`:
|
Five exceptions, each declared in `--help`:
|
||||||
|
|
||||||
- [`emb`](../crates/emb/README.md) makes an HTTP call, which is the whole of what it does.
|
- [`emb`](../crates/emb/README.md) makes an HTTP call, which is the whole of what it does.
|
||||||
- [`bibfetch`](../crates/bibfetch/README.md) asks a register about an identifier, likewise.
|
- [`bibfetch`](../crates/bibfetch/README.md) asks a register about an identifier or a title, likewise.
|
||||||
- [`pdftriage`](../crates/pdftriage/README.md) runs `pdftotext` and `pdfinfo`.
|
- [`pdftriage`](../crates/pdftriage/README.md) runs `pdftotext` and `pdfinfo`.
|
||||||
- [`pdfgutter`](../crates/pdfgutter/README.md) runs `pdftoppm` to look at a page.
|
- [`pdfgutter`](../crates/pdfgutter/README.md) runs `pdftoppm` to look at a page.
|
||||||
- [`avplan`](../crates/avplan/README.md) runs `ffprobe` to read a file's streams, and
|
- [`avplan`](../crates/avplan/README.md) runs `ffprobe` to read a file's streams, and
|
||||||
|
|||||||
Reference in new issue
Block a user