mirror of
https://git.pumpin.online/funkpower/catena.git
synced 2026-11-13 23:55:40 +00:00
funkpower
1abed4632d
Review id 3 on catena#2. P0: Record.cover_id came from the search index's cover_i, the cover chosen for the whole work, and the index is asked first, so cover() fell back to another edition's image (Penguin War and Peace, 9780140447934, got Standard Ebooks' Maude cover 12621906). cover_i is no longer asked for or read; cover_id is the first positive id in the edition record's covers, or None. An ISBN request that fails, as opposed to a 404, now ends the fetch with that failure instead of going on to the cover id. Redirects go only over HTTPS on the standard port to covers.openlibrary.org, archive.org or a host under archive.org, and never to an address by number; the covers store the fetch began at is still allowed. The 1 KB size floor is gone. With default=false a missing cover is a 404, and the stand-in without it is a 43-byte GIF the media type check refuses, so the floor could only drop real Small covers. Covers.deadline (DEADLINE, two minutes) bounds a whole fetch: every address, attempt, redirect, body and wait. It starts once the fetch holds the process lock. The timeout test no longer times a span that includes the lock wait.
166 lines
7.1 KiB
Markdown
166 lines
7.1 KiB
Markdown
# bibfetch
|
|
|
|
What a catalogue says about an identifier, asked of the body that assigns it.
|
|
|
|
Part of [catena](../../README.md).
|
|
|
|
## What it does
|
|
|
|
[`idnorm`](../idnorm/README.md) can tell you a DOI is well formed. It cannot
|
|
tell you the paper exists — that is not a fact about the string, it is a fact
|
|
about the world, and the only way to learn it is to ask somebody who keeps a
|
|
register.
|
|
|
|
So this asks, and that is the only thing it does.
|
|
|
|
## When you would use it
|
|
|
|
- **Auditing a bibliography.** A well-formed DOI that no register has is a
|
|
mistyped one, and a catalogue can carry it for years without noticing.
|
|
- **Filling in a reference** you have only the identifier for.
|
|
- **Before citing something**, to check the year and the authors are what you
|
|
think they are.
|
|
|
|
## Install
|
|
|
|
```sh
|
|
cargo install --git https://git.pumpin.online/funkpower/catena bibfetch
|
|
```
|
|
|
|
## Example
|
|
|
|
```
|
|
$ printf '10.1093/mind/LIX.236.433\narXiv:2401.01234\n978-0-306-40615-7\n' | idnorm | bibfetch --polite [email protected]
|
|
{"v":1,"id":"10.1093/mind/lix.236.433","kind":"doi","found":true,"source":"crossref","title":"I.—COMPUTING MACHINERY AND INTELLIGENCE","authors":["A. M. TURING"],"year":1950,"container":"Mind"}
|
|
{"v":1,"id":"2401.01234","kind":"arxiv","found":true,"source":"arxiv","title":"Mixture cure semiparametric additive hazard models under partly interval censoring -- a penalized likelihood approach","authors":["Jinqing Li","Jun Ma"],"year":2024,"container":"arXiv","summary":"Survival analysis can sometimes involve individuals who will not experience…"}
|
|
{"v":1,"id":"9780306406157","kind":"isbn","found":true,"source":"openlibrary","title":"Error-correction coding for digital communications","year":1981,"container":"Plenum Press"}
|
|
```
|
|
|
|
The ones nobody has:
|
|
|
|
```
|
|
$ printf '10.9999/definitely-not-a-real-doi\n' | idnorm | bibfetch
|
|
{"v":1,"id":"10.9999/definitely-not-a-real-doi","kind":"doi","found":false,"source":"crossref"}
|
|
$ echo $?
|
|
1
|
|
```
|
|
|
|
Absence is the useful answer here. It is what tells a catalogue that a DOI it
|
|
has been carrying for three years was mistyped on the way in.
|
|
|
|
Auditing a whole bibliography:
|
|
|
|
```sh
|
|
idnorm --quiet refs.txt | bibfetch --polite [email protected] \
|
|
| jq -r 'select(.found | not) | "missing: \(.id)"'
|
|
```
|
|
|
|
## Options
|
|
|
|
| Option | Effect |
|
|
|---|---|
|
|
| `--polite <EMAIL>` | An address to put in the request. The registers ask for one and answer faster when they get it |
|
|
| `--timeout <SECONDS>` | How long to wait on a register. Default `20` |
|
|
| `--attempts <COUNT>` | How many times to ask before believing a failure. Default `3` |
|
|
| `--man` | Write this page as a manual page, in roff |
|
|
|
|
## Exit codes
|
|
|
|
| Code | Meaning |
|
|
|---|---|
|
|
| `0` | Every identifier was found |
|
|
| `1` | At least one was not |
|
|
| `64` | The command line was wrong |
|
|
| `65` | The input was not identifier records |
|
|
| `69` | A register could not be reached |
|
|
| `74` | The file could not be read |
|
|
|
|
Exit `1` is an answer — the register replied and does not have it. Exit `69`
|
|
is a failure — nobody replied, and the identifier may be perfectly good.
|
|
|
|
## The registers
|
|
|
|
| Kind | Asked of | Why that one |
|
|
|---|---|---|
|
|
| `doi` | Crossref | It assigns most scholarly DOIs |
|
|
| `arxiv` | the arXiv | Its own register of preprints |
|
|
| `isbn` | OpenLibrary's edition record, then its index, then Google Books | Books, without a key |
|
|
|
|
One register is enough for a DOI or a preprint: the body that assigns the
|
|
identifier is the only one that has it. A book is different. An ISBN is
|
|
assigned by a publisher and recorded by whoever gets round to it, so an
|
|
answer that names the book but not who wrote it is common — OpenLibrary's
|
|
edition record names its authors by key rather than by name, and its index
|
|
names them. So the registers are asked in turn and the first complete answer
|
|
wins. An incomplete one is kept in case nothing better comes, and a register
|
|
being down is a failure only when every register is.
|
|
|
|
Google Books is asked last because it answers anonymous callers only until a
|
|
daily quota runs out, after which it replies `429` to everyone sharing the
|
|
address.
|
|
|
|
An **ISSN** has no register that answers without a key, so it is not looked up
|
|
on its own. A journal's ISSN comes back on the works published in it instead.
|
|
|
|
A failure is retried, waiting a second and then two: long enough for a
|
|
register that is rate-limiting, short enough that one which is simply down
|
|
does not hold up a pipeline. An answer this cannot read is not retried,
|
|
because it will not read any better next time.
|
|
|
|
## Covers
|
|
|
|
```sh
|
|
bibfetch cover 978-0-306-40615-7 -o cover.jpg
|
|
```
|
|
|
|
Fetches a book's cover from Open Library's covers store by its ISBN and
|
|
writes the image. The thirteen-digit form is asked first and the ten-digit
|
|
form second, because a book is often filed under only one of them; an ISBN
|
|
whose check digit is wrong is asked about as written, since converting it
|
|
would give another book's number. `--cover-id` names an Open Library cover id
|
|
to ask by last, which is what the `cover_id` field of a lookup record holds:
|
|
the first cover in Open Library's record of that edition, never the cover its
|
|
index picked for the whole work, which is often another edition's. The cover
|
|
id is asked only when every form of the ISBN asked was answered with no cover; if
|
|
asking by ISBN failed, that failure is the result.
|
|
|
|
What comes back is checked before it is written: a JPEG or PNG by its bytes
|
|
as well as by what the server says, no larger than ten megabytes, reached in at
|
|
most three redirects, each over HTTPS to `covers.openlibrary.org`,
|
|
`archive.org` or a host under `archive.org`, and never to an address by number.
|
|
One request is made at a time, a `429` or `503` is waited out for as long as
|
|
its `Retry-After` says, up to a minute, and the whole fetch, every attempt and
|
|
redirect included, stops after two minutes. The two minutes start once it is
|
|
the fetch's turn, so a fetch queued behind others in the same process can wait
|
|
that long again first.
|
|
|
|
| Option | Effect |
|
|
|---|---|
|
|
| `-o`, `--output <FILE>` | Where to write the image |
|
|
| `--size <SIZE>` | `small`, `medium` or `large`. Default `large` |
|
|
| `--cover-id <ID>` | An Open Library cover id to ask by when the ISBN has none |
|
|
|
|
Exit `0` the cover was written, `1` Open Library has none, `65` the argument
|
|
was not an ISBN or the answer was not an image, `69` Open Library could not be
|
|
reached, `74` the file could not be written.
|
|
|
|
From Rust, `bibfetch::cover(&agent, isbn, cover_id, size)` returns
|
|
`Ok(None)` when there is no cover, and a [`Cover`](src/cover.rs) with the
|
|
bytes, the media type and the address asked when there is.
|
|
|
|
## Reaching the network
|
|
|
|
This and [`emb`](../emb/README.md) are the only two tools here that do. Both
|
|
say so in `--help`, and for both it is the whole of what they do rather than
|
|
something that happens on the way to something else — which is the condition
|
|
under which [the contract](../../docs/contract.md) allows it at all.
|
|
|
|
Nothing is cached, nothing is stored but a cover you asked to have written,
|
|
and nothing is sent but the identifier and the address you pass to
|
|
`--polite`.
|
|
|
|
## See also
|
|
|
|
- [`idnorm`](../idnorm/README.md) — reading the identifier in the first place.
|
|
- [`bibclean`](../bibclean/README.md) — the rest of a catalogue record.
|