Author SHA1 Message Date
funkpower 48478fe84e booksniff: a zip of pictures is not a comic until something says so
A zip whose members are all images was called cbz from their names alone,
so a folder of holiday photographs came back as a comic. It is now cbz
only when it carries a ComicInfo.xml or is itself named .cbz or .cbr;
otherwise it is unknown, with a note that it is a zip of N images and
would be a comic book if its name said .cbz. Only the archive's directory
is read, so members are still judged by name, and the README says so,
for a wrapped FictionBook too.

{\rtf has to be the whole control word: the byte after it must be the
version digit, a space or the next control word, so {\rtfoo is not RTF.

A FictionBook saved as UTF-16 was unknown. A byte-order mark in either
order now has the front decoded, at most 2048 code units, before the same
walk to the root element.
2026-11-13 15:49:17 -05:00
funkpower de23a7abad bookmeta, bibfetch, bindings: what a caller needs to trust a cover
bookmeta: Cover::guessed says the file left the choice open — a Kindle
book's first image record, or an EPUB image picked because its name says
cover — so a caller with a register to ask can prefer that. A cover the
file names is never a guess. Cover::extension matches bibfetch's.

bibfetch: Open Library answers 403 to more than a hundred covers by ISBN
in five minutes from one address. That is now Error::Limited, not
Unreachable: the other form of the ISBN is not asked, the cover id still
is, since those are not limited, and the CLI exits 75. Covers asked by
ISBN are kept three seconds apart across the process (Covers::pace), which
stays under the limit however long a run goes. A deadline too long to add
to the clock no longer panics; it is no deadline.

bindings: fetch_cover and search_title for Python, and catena.Limited,
which fetch_identifier raises too when a register refuses for being asked
too often.

corpus-core: exit 75, a service refused for now.
2026-11-13 15:44:50 -05:00
funkpower 7aa1783a3e Merge remote-tracking branch 'origin/bibfetch-search' into fixes-040 2026-11-13 15:32:45 -05:00
funkpower 4d21cc5927 Merge remote-tracking branch 'origin/bookmeta-epub-cover' into fixes-040 2026-11-13 15:32:45 -05:00
funkpower 6510973850 Merge remote-tracking branch 'origin/booksniff-more-formats' into fixes-040 2026-11-13 15:32:45 -05:00
funkpower 897b2210b6 Merge remote-tracking branch 'origin/main' into bookmeta-epub-cover
# Conflicts:
#	crates/bookmeta/src/mobi.rs
2026-11-13 15:29:53 -05:00
funkpower 57ce9137e6 bookmeta: read an EPUB's cover, by the four routes a package can name one
The cover is the first image reached by EPUB 3's cover-image property,
EPUB 2's <meta name="cover"> (by id, or by address where a converter wrote
one), the guide's cover page and the first <img> or SVG <image> on it, and
last any manifest image whose id or address says cover. A route that names
something unusable leaves a note and the next is asked.

Addresses are resolved against the document they are written in, with the
fragment off and percent-escapes undone; one that climbs out of the archive
or carries a scheme is not followed. Every member is read through a bound,
checked against the size the archive declares and enforced on the read
itself, so an understated entry stops at the cap too.

The media type is read off the bytes, now WebP as well, and a manifest that
disagrees is a note; bytes that are not an image are no cover. MAX_COVER
(16 MiB) is the cap for both formats, so a Kindle cover record larger than
that is left out as well.
2026-11-13 15:26:49 -05:00
funkpower 5c4f770d76 booksniff: RTF, Microsoft Reader, FictionBook and comic archives
Four formats a library of ebooks holds and this had no name for, so each
came back as text, as HTML or as unknown.

RTF is its opening group, after any byte-order mark and blank space.
Microsoft Reader is its signature at the front. A FictionBook is XML
whose root element is FictionBook, found by stepping over the
declaration, comments and doctype in the first four kilobytes. That walk
is over bytes rather than text, because a great many FictionBooks are
windows-1251 and would otherwise be called binary; other XML is still
HTML, as before.

A zip whose only real member is one FictionBook is zip-wrapped-fb2, the
same way a zipped PDF is zip-wrapped-pdf: handed to a converter as
FictionBook it is still a zip. A zip whose every real member is a JPEG,
PNG, GIF or WebP is cbz, once directories, ComicInfo.xml and desktop
clutter are passed over. One PDF or one readme among the pages and it is
not a comic. Only the names in the archive's directory are read, as for
every other zip.

The kind vocabulary grows by five: lit, fb2, rtf, cbz and
zip-wrapped-fb2.
2026-11-13 15:25:33 -05:00
22 changed files with 1861 additions and 54 deletions

No files matched your search

+10 -1
View File
@@ -136,6 +136,14 @@ redirect included, stops after two minutes. The two minutes start once it is
the fetch's turn, so a fetch queued behind others in the same process can wait
that long again first.
Open Library allows a hundred covers by ISBN every five minutes from one
address, and refuses the rest with `403`. Covers asked by ISBN are kept three
seconds apart across the whole process (`Covers::pace`), which stays under that
limit however long a run goes. A refusal all the same is not mistaken for an
outage: the other form of the ISBN is not asked, the cover id is, because
Open Library does not limit those, and with no cover found that way the fetch
fails with `Error::Limited`, which means ask again later.
| Option | Effect |
|---|---|
| `-o`, `--output <FILE>` | Where to write the image |
@@ -144,7 +152,8 @@ that long again first.
Exit `0` the cover was written, `1` Open Library has none, `65` the argument
was not an ISBN or the answer was not an image, `69` Open Library could not be
reached, `74` the file could not be written.
reached, `74` the file could not be written, `75` Open Library refused for
being asked too often and will answer later.
From Rust, `bibfetch::cover(&agent, isbn, cover_id, size)` returns
`Ok(None)` when there is no cover, and a [`Cover`](src/cover.rs) with the
+126 -8
View File
@@ -46,6 +46,11 @@ const REDIRECT_HOSTS: [&str; 2] = ["covers.openlibrary.org", "archive.org"];
/// agent's timeout each, and a minute's `Retry-After` between attempts.
pub const DEADLINE: Duration = Duration::from_secs(120);
/// How long to leave between two covers asked by ISBN. Open Library allows a
/// hundred every five minutes from one address and refuses the rest with
/// `403`; one every three seconds stays under that however long a run goes.
pub const PACE: Duration = Duration::from_secs(3);
/// The longest a `Retry-After` is honoured for. A register asking for more
/// than this is better treated as down than waited on.
const MAX_WAIT: Duration = Duration::from_secs(60);
@@ -54,7 +59,10 @@ const MAX_WAIT: Duration = Duration::from_secs(60);
/// this, and limits covers asked by ISBN to a hundred every five minutes
/// from one address; threads asking at once would only meet the limit
/// sooner and all back off together.
static ONE_AT_A_TIME: Mutex<()> = Mutex::new(());
///
/// It holds when the last cover was asked for by ISBN, so that [`PACE`] is
/// kept across fetches and threads.
static ONE_AT_A_TIME: Mutex<Option<Instant>> = Mutex::new(None);
/// Which of Open Library's three sizes to ask for.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
@@ -127,6 +135,9 @@ pub struct Covers {
/// request is given whatever is left of it, or the agent's own timeout
/// if that is shorter, and a wait that would run past it is not waited.
pub deadline: Duration,
/// The least time between two covers asked by ISBN; [`PACE`] by default.
/// Covers asked by id are not limited and are not spaced out.
pub pace: Duration,
}
impl Default for Covers {
@@ -135,6 +146,7 @@ impl Default for Covers {
base: OPEN_LIBRARY_COVERS.to_string(),
attempts: crate::ATTEMPTS,
deadline: DEADLINE,
pace: PACE,
}
}
}
@@ -162,6 +174,12 @@ impl Covers {
/// of the ISBN does not stop the other being asked, but it does stop the
/// cover id being asked: the ISBN's own cover may exist, and is the one
/// wanted. The failure is returned when no ISBN form gave a cover.
///
/// Refusal for asking too often is the exception. It is
/// [`Error::Limited`], the other form is not asked, since it would only
/// be refused too, and the cover id is still asked, because Open Library
/// does not limit those. A cover found that way is returned; otherwise the
/// refusal is, so that the caller knows to ask again later.
pub fn fetch(
&self,
agent: &ureq::Agent,
@@ -186,31 +204,54 @@ impl Covers {
.map(|form| format!("{}/b/isbn/{form}-{letter}.jpg?default=false", self.base))
.collect();
let _turn = ONE_AT_A_TIME
let mut last_by_isbn = ONE_AT_A_TIME
.lock()
.unwrap_or_else(|held| held.into_inner());
// From here, not from the call: waiting for the lock is time another
// fetch spent, and counting it would let a busy process starve this
// one of the time its own requests need.
let until = Instant::now() + self.deadline;
// one of the time its own requests need. A deadline too long to add
// to the clock is no deadline at all.
let now = Instant::now();
let until = now
.checked_add(self.deadline)
.unwrap_or_else(|| now + Duration::from_secs(365 * 24 * 60 * 60));
let mut failed = None;
for address in &addresses {
match self.ask(agent, address, until) {
if let Some(ready) = last_by_isbn.and_then(|at| at.checked_add(self.pace)) {
let now = Instant::now();
if ready > now {
if ready >= until {
return Err(Error::Unreachable(format!(
"{address}: past the {:?} deadline, waiting to keep {:?} \
between covers asked by ISBN",
self.deadline, self.pace
)));
}
std::thread::sleep(ready - now);
}
}
let answer = self.ask(agent, address, until);
*last_by_isbn = Some(Instant::now());
match answer {
Ok(Answer::Image(cover)) => return Ok(Some(cover)),
Ok(Answer::Absent) => {}
Err(why @ Error::Limited(_)) => {
failed = Some(why);
break;
}
Err(why) => failed = Some(why),
}
}
if let Some(why) = failed {
if let Some(why) = failed.take_if(|why| !matches!(why, Error::Limited(_))) {
return Err(why);
}
let Some(id) = cover_id else {
return Ok(None);
return failed.map_or(Ok(None), Err);
};
let address = format!("{}/b/id/{id}-{letter}.jpg?default=false", self.base);
match self.ask(agent, &address, until)? {
Answer::Image(cover) => Ok(Some(cover)),
Answer::Absent => Ok(None),
Answer::Absent => failed.map_or(Ok(None), Err),
}
}
@@ -311,6 +352,15 @@ Open Library's covers or the Internet Archive"
// `default=false` makes a missing cover a 404 rather than a
// stand-in image.
404 | 410 => return Ok(Answer::Absent),
// Open Library's answer to more than a hundred covers asked by
// ISBN in five minutes from one address. Asking again at once
// would only be refused again.
403 => {
return Err(Fault::Final(Error::Limited(format!(
"{at}: refused (403); Open Library allows a hundred covers \
by ISBN every five minutes from one address"
))));
}
429 | 503 => {
let wait = header(&response, "retry-after")
.and_then(|value| value.trim().parse::<u64>().ok())
@@ -603,6 +653,7 @@ mod tests {
base: self.base.clone(),
attempts: 3,
deadline: DEADLINE,
pace: Duration::ZERO,
}
}
@@ -1052,6 +1103,73 @@ mod tests {
assert_eq!(stub.asked(), [ISBN13, ISBN10]);
}
#[test]
fn a_refusal_for_asking_too_often_still_asks_the_cover_id() {
let stub = Stub::start(|_, path| match path {
ISBN13 => Reply::status(403),
"/b/id/42-L.jpg?default=false" => Reply::image("image/jpeg", jpeg(4096)),
_ => Reply::status(404),
});
let cover = stub
.covers()
.fetch(&agent(), "9780306406157", Some(42), CoverSize::Large)
.expect("an answer")
.expect("a cover");
assert!(cover.url.ends_with("/b/id/42-L.jpg?default=false"));
// The ten-digit form would only have been refused too.
assert_eq!(stub.asked(), [ISBN13, "/b/id/42-L.jpg?default=false"]);
}
#[test]
fn a_refusal_for_asking_too_often_is_not_an_absence() {
let stub = Stub::start(|_, path| match path {
ISBN13 => Reply::status(403),
_ => Reply::status(404),
});
let found = stub
.covers()
.fetch(&agent(), "9780306406157", None, CoverSize::Large)
.map(|cover| cover.map(|cover| cover.url));
assert!(matches!(found, Err(Error::Limited(_))), "{found:?}");
assert_eq!(stub.asked(), [ISBN13]);
// With a cover id that has no cover either, the refusal still stands:
// the ISBN's own cover may be there once the limit passes.
let found = stub
.covers()
.fetch(&agent(), "9780306406157", Some(42), CoverSize::Large)
.map(|cover| cover.map(|cover| cover.url));
assert!(matches!(found, Err(Error::Limited(_))), "{found:?}");
}
#[test]
fn a_deadline_too_long_to_count_is_no_deadline() {
let stub = Stub::start(|_, _| Reply::image("image/jpeg", jpeg(2048)));
let covers = Covers {
deadline: Duration::MAX,
..stub.covers()
};
let cover = covers
.fetch(&agent(), "9780306406157", None, CoverSize::Large)
.expect("an answer");
assert!(cover.is_some());
}
#[test]
fn covers_asked_by_isbn_are_spaced_out() {
let stub = Stub::start(|_, _| Reply::status(404));
let covers = Covers {
pace: Duration::from_millis(300),
..stub.covers()
};
let started = Instant::now();
let found = covers.fetch(&agent(), "9780306406157", None, CoverSize::Large);
assert!(matches!(found, Ok(None)), "{found:?}");
// Two forms of the ISBN, so at least one pause between them.
assert!(started.elapsed() >= Duration::from_millis(300));
assert_eq!(stub.asked(), [ISBN13, ISBN10]);
}
#[test]
fn a_redirect_to_an_address_by_number_is_refused() {
// A listener standing in for a service on the LAN; it only counts.
+4
View File
@@ -112,6 +112,9 @@ pub enum Error {
NoRegister(String),
/// What was asked about is not an identifier of the kind it should be.
Invalid(String),
/// The register refused because it has been asked too often from this
/// address. It is not down, and asking again later will work.
Limited(String),
}
impl std::fmt::Display for Error {
@@ -123,6 +126,7 @@ impl std::fmt::Display for Error {
write!(f, "no register here answers about {kind} identifiers")
}
Error::Invalid(why) => write!(f, "{why}"),
Error::Limited(why) => write!(f, "{why}"),
}
}
}
+1
View File
@@ -225,6 +225,7 @@ fn cover(
eprintln!("bibfetch: {isbn}: {why}");
ExitCode::from(match why {
bibfetch::Error::Unreachable(_) => exit::UNAVAILABLE,
bibfetch::Error::Limited(_) => exit::TEMPFAIL,
_ => exit::DATA,
})
}
@@ -21,6 +21,8 @@ from ._catena import (
compose_subjects,
count_tokens,
cue_windows,
Limited,
fetch_cover,
fetch_identifier,
find_citations,
fuse_hits,
@@ -41,6 +43,7 @@ from ._catena import (
record_severity,
same_work,
scrub,
search_title,
segment_cues,
series_of,
shelve,
@@ -53,6 +56,7 @@ from ._catena import (
__all__ = [
"__version__",
"Limited",
"canonical_subject",
"chunk_markdown",
"citation_prompt",
@@ -62,6 +66,7 @@ __all__ = [
"compose_subjects",
"count_tokens",
"cue_windows",
"fetch_cover",
"fetch_identifier",
"find_citations",
"fuse_hits",
@@ -82,6 +87,7 @@ __all__ = [
"record_severity",
"same_work",
"scrub",
"search_title",
"segment_cues",
"series_of",
"shelve",
@@ -103,6 +103,40 @@ def fetch_identifier(
"""
...
class Limited(Exception):
"""A register refused because it has been asked too often; ask again later."""
def fetch_cover(
isbn: str,
cover_id: int | None = ...,
size: str = ...,
polite: str | None = ...,
timeout: int = ...,
deadline: int = ...,
) -> dict[str, Any] | None:
"""The cover Open Library has for a book, by ISBN, then by cover id.
A dict with `bytes`, `media_type`, `extension`, `source` and `url`, or
`None` when there is none. Raises `Limited` when Open Library refused for
being asked too often. Blocks on the network.
"""
...
def search_title(
title: str,
author: str | None = ...,
limit: int = ...,
polite: str | None = ...,
timeout: int = ...,
attempts: int = ...,
) -> list[dict[str, Any]]:
"""The editions the registers list for a title and author, in their order.
Records shaped like `fetch_identifier`'s. Use `same_work` to decide which
one is the book. Blocks on the network.
"""
...
def colophon_prompt(
record: dict[str, Any], text: str, pages: int = ...
) -> str | None:
+104 -2
View File
@@ -10,7 +10,7 @@
use pyo3::exceptions::PyValueError;
use pyo3::prelude::*;
use pyo3::types::{PyDict, PyList};
use pyo3::types::{PyBytes, PyDict, PyList};
use crate::{ChunkBudgets, CueBudgets};
@@ -434,6 +434,9 @@ fn _catena(module: &Bound<'_, PyModule>) -> PyResult<()> {
module.add_function(wrap_pyfunction!(read_colophon, module)?)?;
module.add_function(wrap_pyfunction!(shelve, module)?)?;
module.add_function(wrap_pyfunction!(fetch_identifier, module)?)?;
module.add_function(wrap_pyfunction!(fetch_cover, module)?)?;
module.add_function(wrap_pyfunction!(search_title, module)?)?;
module.add("Limited", module.py().get_type::<Limited>())?;
module.add_function(wrap_pyfunction!(colophon_prompt, module)?)?;
module.add_function(wrap_pyfunction!(merge_colophon, module)?)?;
Ok(())
@@ -512,13 +515,112 @@ fn fetch_identifier<'py>(
// would stop every other thread in the interpreter for the whole request.
let found = py
.detach(|| bibfetch::ask::lookup(&agent, id, kind, ask.attempts))
.map_err(|e| PyValueError::new_err(e.to_string()))?;
.map_err(register_error)?;
match found {
Some(record) => Ok(Some(to_py(py, &record)?)),
None => Ok(None),
}
}
pyo3::create_exception!(
_catena,
Limited,
pyo3::exceptions::PyException,
"A register refused because it has been asked too often; ask again later."
);
/// A register's error as the Python exception that says what to do about it.
fn register_error(error: bibfetch::Error) -> PyErr {
match error {
bibfetch::Error::Limited(why) => Limited::new_err(why),
other => PyValueError::new_err(other.to_string()),
}
}
/// The cover Open Library has for a book, by its ISBN, then by the cover id
/// a record named.
///
/// Returns a dict with `bytes`, `media_type` (`image/jpeg` or `image/png`),
/// `extension`, `source` and `url`, or `None` when there is no cover. Covers
/// asked by ISBN are kept three seconds apart across the process, and the
/// whole fetch stops after `deadline` seconds. Raises `catena.Limited` when
/// Open Library refused for being asked too often, and `ValueError` when the
/// ISBN is not one or the store could not be reached. Blocks on the network.
#[pyfunction]
#[pyo3(signature = (isbn, cover_id = None, size = "large", polite = None, timeout = 20, deadline = 120))]
fn fetch_cover<'py>(
py: Python<'py>,
isbn: &str,
cover_id: Option<u64>,
size: &str,
polite: Option<String>,
timeout: u64,
deadline: u64,
) -> PyResult<Option<Bound<'py, PyDict>>> {
let size = match size {
"small" => bibfetch::CoverSize::Small,
"medium" => bibfetch::CoverSize::Medium,
"large" => bibfetch::CoverSize::Large,
other => {
return Err(PyValueError::new_err(format!(
"size is small, medium or large, not {other:?}"
)));
}
};
let agent = bibfetch::ask::Ask {
polite,
timeout: std::time::Duration::from_secs(timeout.max(1)),
..Default::default()
}
.agent();
let covers = bibfetch::cover::Covers {
deadline: std::time::Duration::from_secs(deadline.max(1)),
..Default::default()
};
let found = py
.detach(|| covers.fetch(&agent, isbn, cover_id, size))
.map_err(register_error)?;
let Some(cover) = found else {
return Ok(None);
};
let out = PyDict::new(py);
out.set_item("bytes", PyBytes::new(py, &cover.bytes))?;
out.set_item("media_type", cover.media_type)?;
out.set_item("extension", cover.extension())?;
out.set_item("source", cover.source)?;
out.set_item("url", cover.url)?;
Ok(Some(out))
}
/// The editions the registers list for a title, and an author if one is known.
///
/// Open Library is asked first and Google Books only when it has nothing.
/// The candidates come back in the register's own order, as records shaped
/// like `fetch_identifier`'s; nothing here decides which is the book, which is
/// what `same_work` is for. Blocks on the network.
#[pyfunction]
#[pyo3(signature = (title, author = None, limit = 10, polite = None, timeout = 20, attempts = 3))]
fn search_title<'py>(
py: Python<'py>,
title: &str,
author: Option<&str>,
limit: usize,
polite: Option<String>,
timeout: u64,
attempts: u32,
) -> PyResult<Bound<'py, PyList>> {
let ask = bibfetch::ask::Ask {
polite,
timeout: std::time::Duration::from_secs(timeout.max(1)),
attempts: attempts.max(1),
};
let agent = ask.agent();
let found = py
.detach(|| bibfetch::search(&agent, title, author, limit, ask.attempts))
.map_err(register_error)?;
list_to_py(py, &found)
}
/// The question to put to a model about the fields a book's pages did not
/// settle, or `None` when they settled everything.
///
+43 -6
View File
@@ -99,8 +99,10 @@ A Kindle book's title is EXTH 503 where it is there, then the MOBI header's
full name, then the Palm database name (cut to 31 bytes, with a note). Authors
(100) and subjects (105) are every record of their kind, in order; publisher
(101), description (103), date (106) and language (524) the first. EXTH 104 is
an `isbn` when it is shaped like one and `other` when not, and 113 is an
`asin`. A contributor (108) is not an author, the same as an EPUB's
an `isbn` when it is shaped like one and `other` when not. 113 is filed by
its shape: a `uuid` when it is one (calibre writes its book id there), an
`asin` when it is ten letters and digits, and `other` when it is neither. A
contributor (108) is not an author, the same as an EPUB's
`dc:contributor`, and is not reported. With no EXTH language, the language
comes from the header's locale. Strings are decoded as the header says,
UTF-8 or Windows-1252. A book whose text is encrypted still has its header
@@ -112,10 +114,45 @@ an EXTH block that breaks off keeps the records before the break.
## Covers
`Metadata::cover` is the cover image a Kindle book carries (EXTH 201, or the
thumbnail, 202, when no cover is named), with its media type read off its
first bytes. It is for library callers and is not part of the JSON record,
which is a line of text.
`Metadata::cover` is the cover image the file carries, with its media type
read off its first bytes: JPEG, PNG, GIF, WebP or BMP. It is for library
callers and is not part of the JSON record, which is a line of text.
A Kindle book's cover is EXTH 201, or the thumbnail, 202, when no cover is
named. With neither, it is the record at the header's first image index, as
calibre does, and only that record: when it is not a JPEG, PNG or GIF there is
no cover, because the records after it can be illustrations, and kindlegen
sometimes puts a font there first.
An EPUB's is the first of these that reaches an image:
1. The manifest item whose `properties` include `cover-image` (EPUB 3).
2. The item `<meta name="cover" content="…">` names (EPUB 2). Some
converters write the image's address there rather than its id, and that is
followed too.
3. The guide's `<reference type="cover">`, a page, and the first `<img src>`
or SVG `<image href>` on it. A guide that points straight at an image is
taken at its word.
4. Any image in the manifest whose id or address says `cover`.
A route that names something unusable — not in the archive, not an image —
leaves a note, and the next route is asked. Addresses are resolved against the
document they are written in, with their fragment off and their
percent-escapes undone, and one that climbs out of the archive or names a
scheme (`http:`, `data:`) is not followed.
What the bytes say beats what the manifest says: a JPEG listed as
`image/png` comes back as `image/jpeg`, with a note. A cover larger than
`bookmeta::MAX_COVER` (16 MiB) is left out with a note, in either format, and
an EPUB's is never decompressed further than that, whatever size the archive
claims for it.
`Cover::guessed` says the file left the choice open: a Kindle book's first
image record, or an EPUB image picked because its name says `cover`. Either
can be an illustration from inside the book, so a caller that can get a cover
another way, from a register by ISBN, should prefer that one. A cover the file
names is never a guess. `Cover::extension` gives the usual file extension for
its media type.
## Asking a model
+731 -1
View File
@@ -4,10 +4,17 @@
//! document carries Dublin Core elements: title, creator, publisher, date,
//! language, identifier, subject. Calibre adds its own `<meta>` entries for
//! the series, which nothing in the standard has a place for.
//!
//! The cover is whichever image the package names, asked in the order the
//! standards came: EPUB 3's `cover-image` property, EPUB 2's
//! `<meta name="cover">`, the guide's cover page and the image on it, and
//! last any image whose name says cover. Every address is resolved inside the
//! archive and nowhere else, and every member is read no further than a fixed
//! bound, [`MAX_COVER`] for an image.
use std::io::Read;
use crate::{Metadata, Source};
use crate::{Cover, MAX_COVER, Metadata, Source, image_type};
/// How large a package document is worth reading.
///
@@ -38,6 +45,7 @@ pub fn read(path: &str, bytes: &[u8]) -> Metadata {
};
fill(&mut meta, &package);
meta.cover = cover(&mut archive, &package_path, &package, &mut meta);
if meta.is_empty() {
meta.note("a package document with nothing filled in");
}
@@ -248,6 +256,341 @@ fn unescape(text: &str) -> String {
.replace("&amp;", "&")
}
/// One entry of the package's manifest.
struct Item {
id: String,
href: String,
media_type: String,
properties: String,
/// Where `href` leads inside the archive, when it leads anywhere.
path: Option<String>,
}
/// The cover image, by the first of the four routes that reaches one.
///
/// A route that names something unusable leaves a note and the next route is
/// asked, so a book whose declared cover is missing can still have the one
/// its file name gives away.
fn cover<R: std::io::Read + std::io::Seek>(
archive: &mut zip::ZipArchive<R>,
package_path: &str,
package: &str,
meta: &mut Metadata,
) -> Option<Cover> {
let base = directory_of(package_path);
let items: Vec<Item> = tags(package)
.filter(|(name, _)| *name == "item")
.map(|(_, tag)| {
let href = attr(tag, "href").unwrap_or_default();
Item {
id: attr(tag, "id").unwrap_or_default(),
path: resolve(base, &href),
href,
media_type: attr(tag, "media-type").unwrap_or_default(),
properties: attr(tag, "properties").unwrap_or_default(),
}
})
.collect();
let mut look = Look {
archive,
items: &items,
tried: Vec::new(),
meta,
};
// EPUB 3: the manifest item whose properties say so.
for item in items.iter().filter(|item| {
item.properties
.split_ascii_whitespace()
.any(|property| property == "cover-image")
}) {
if let Some(found) = look.image(base, &item.href) {
return Some(found);
}
}
// EPUB 2: `<meta name="cover" content="…">` names an item by its id, or
// in some converters' hands by its address.
if let Some(named) = tags(package)
.filter(|(name, _)| *name == "meta")
.find(|(_, tag)| attr(tag, "name").as_deref() == Some("cover"))
.and_then(|(_, tag)| attr(tag, "content"))
.map(|content| content.trim().to_string())
.filter(|content| !content.is_empty())
{
let as_path = resolve(base, &named);
let href = items
.iter()
.find(|item| item.id == named)
.or_else(|| {
items
.iter()
.find(|item| item.path.is_some() && item.path == as_path)
})
.map_or(named.as_str(), |item| item.href.as_str());
if let Some(found) = look.image(base, href) {
return Some(found);
}
}
// The guide's cover page, and the first image on it.
if let Some(page) = tags(package)
.filter(|(name, _)| *name == "reference")
.find(|(_, tag)| attr(tag, "type").is_some_and(|kind| kind.eq_ignore_ascii_case("cover")))
.and_then(|(_, tag)| attr(tag, "href"))
{
if let Some(found) = look.page(base, &page) {
return Some(found);
}
}
// Last: an image whose id or address says cover.
for item in items.iter().filter(|item| {
item.media_type.starts_with("image/")
&& (item.id.to_lowercase().contains("cover")
|| item.href.to_lowercase().contains("cover"))
}) {
if let Some(mut found) = look.image(base, &item.href) {
found.guessed = true;
return Some(found);
}
}
None
}
/// The search for a cover, remembering what it has already looked at so
/// that two routes to the same member read it, and note it, once.
struct Look<'a, R> {
archive: &'a mut zip::ZipArchive<R>,
items: &'a [Item],
tried: Vec<String>,
meta: &'a mut Metadata,
}
impl<R: std::io::Read + std::io::Seek> Look<'_, R> {
/// Where an address leads, or a note saying it leads nowhere it may.
fn locate(&mut self, base: &str, href: &str) -> Option<String> {
let Some(path) = resolve(base, href) else {
self.meta.note(format!(
"the cover's address, {href}, is not a path inside the archive"
));
return None;
};
if self.tried.contains(&path) {
return None;
}
self.tried.push(path.clone());
Some(path)
}
/// An image at an address, if that is what is there.
fn image(&mut self, base: &str, href: &str) -> Option<Cover> {
let path = self.locate(base, href)?;
let bytes = self.fetch(&path, MAX_COVER, "cover")?;
self.check(&path, bytes)
}
/// The image a cover page shows, or the page itself where the guide
/// points straight at an image.
fn page(&mut self, base: &str, href: &str) -> Option<Cover> {
let path = self.locate(base, href)?;
let bytes = self.fetch(&path, MAX_PACKAGE as usize, "cover page")?;
if image_type(&bytes).is_some() {
return self.check(&path, bytes);
}
let text = String::from_utf8_lossy(&bytes);
let Some(src) = tags(&text).find_map(|(name, tag)| match name {
"img" => attr(tag, "src"),
"image" => attr(tag, "xlink:href").or_else(|| attr(tag, "href")),
_ => None,
}) else {
self.meta
.note(format!("the cover page, {path}, shows no image"));
return None;
};
self.image(directory_of(&path), &src)
}
/// One member's bytes, read no further than `cap`.
///
/// The size the archive declares is checked first and the read is
/// bounded anyway, because the declaration may be a lie.
fn fetch(&mut self, path: &str, cap: usize, what: &str) -> Option<Vec<u8>> {
let file = match self.archive.by_name(path) {
Ok(file) => file,
Err(zip::result::ZipError::FileNotFound) => {
self.meta
.note(format!("the {what}, {path}, is not in the archive"));
return None;
}
Err(_) => {
self.meta
.note(format!("the {what}, {path}, could not be read"));
return None;
}
};
let too_large = format!("the {what}, {path}, is larger than {}", crate::size(cap));
if file.size() > cap as u64 {
self.meta.note(too_large);
return None;
}
let mut bytes = Vec::new();
if file.take(cap as u64 + 1).read_to_end(&mut bytes).is_err() {
self.meta
.note(format!("the {what}, {path}, could not be read"));
return None;
}
if bytes.len() > cap {
self.meta.note(too_large);
return None;
}
Some(bytes)
}
/// The bytes as a cover when they are an image, typed by what they are
/// rather than by what the manifest says.
fn check(&mut self, path: &str, bytes: Vec<u8>) -> Option<Cover> {
let Some(media_type) = image_type(&bytes) else {
self.meta
.note(format!("the cover, {path}, is not an image"));
return None;
};
if let Some(item) = self
.items
.iter()
.find(|item| item.path.as_deref() == Some(path))
{
let declared = item.media_type.trim();
if !declared.is_empty() && !declared.eq_ignore_ascii_case(media_type) {
self.meta.note(format!(
"the manifest calls the cover, {path}, {declared}, but it is {media_type}"
));
}
}
Some(Cover {
media_type: media_type.to_string(),
bytes,
guessed: false,
})
}
}
/// The directory a member sits in, with no trailing slash.
fn directory_of(path: &str) -> &str {
path.rsplit_once('/').map_or("", |(dir, _)| dir)
}
/// Where an address written in a member in `base` leads inside the archive.
///
/// The fragment comes off and percent-escapes are undone. An address with a
/// scheme, or one that climbs above the root of the archive, leads nowhere
/// this will follow.
fn resolve(base: &str, href: &str) -> Option<String> {
let href = href.split('#').next().unwrap_or("").trim();
// `http:`, `data:`, `file:`: a scheme before the first slash.
if let Some(colon) = href.find(':') {
if !href[..colon].contains('/') {
return None;
}
}
let href = percent_decode(href)?;
let mut parts: Vec<&str> = if href.starts_with('/') {
Vec::new()
} else {
base.split('/').filter(|part| !part.is_empty()).collect()
};
for part in href.split('/') {
match part {
"" | "." => {}
".." => {
parts.pop()?;
}
part => parts.push(part),
}
}
// An address of nothing but the directory it is written in names no
// member at all.
if href.is_empty() || parts.is_empty() {
return None;
}
Some(parts.join("/"))
}
/// `%20` back to a space, and so on, where the result is still UTF-8.
fn percent_decode(text: &str) -> Option<String> {
let raw = text.as_bytes();
let mut out = Vec::with_capacity(raw.len());
let mut at = 0;
while at < raw.len() {
if raw[at] == b'%' {
let hex = std::str::from_utf8(raw.get(at + 1..at + 3)?).ok()?;
out.push(u8::from_str_radix(hex, 16).ok()?);
at += 3;
} else {
out.push(raw[at]);
at += 1;
}
}
String::from_utf8(out).ok()
}
/// Every start tag in a document, in order, as its local name and its text.
///
/// Comments, declarations, processing instructions and end tags are passed
/// over. The local name is what follows any namespace prefix, so
/// `<opf:item>` is an `item`.
fn tags(xml: &str) -> impl Iterator<Item = (&str, &str)> {
let mut rest = xml;
std::iter::from_fn(move || {
loop {
let at = rest.find('<')?;
rest = &rest[at..];
if let Some(after) = rest.strip_prefix("<!--") {
rest = after.find("-->").map_or("", |end| &after[end + 3..]);
continue;
}
let end = rest.find('>')?;
let tag = &rest[1..end];
rest = &rest[end + 1..];
if tag.starts_with(['/', '!', '?']) {
continue;
}
let name = tag
.split(|c: char| c.is_ascii_whitespace() || c == '/')
.next()
.unwrap_or("");
let local = name.rsplit_once(':').map_or(name, |(_, local)| local);
return Some((local, tag));
}
})
}
/// The value of one attribute in a tag's text, entities undone.
///
/// The name has to stand on its own — `id` is not the end of `xml:id` or
/// `data-id` — and may have space either side of its `=`.
fn attr(tag: &str, name: &str) -> Option<String> {
let mut from = 0;
while let Some(found) = tag[from..].find(name) {
let at = from + found;
from = at + name.len();
if !tag[..at].ends_with(|c: char| c.is_ascii_whitespace()) {
continue;
}
let Some(rest) = tag[from..].trim_start().strip_prefix('=') else {
continue;
};
let rest = rest.trim_start();
let Some(quote) = rest.chars().next().filter(|c| matches!(c, '"' | '\'')) else {
continue;
};
let value = &rest[1..];
let end = value.find(quote)?;
return Some(unescape(&value[..end]));
}
None
}
#[cfg(test)]
mod tests {
use super::*;
@@ -356,4 +699,391 @@ mod tests {
assert_eq!(meta.source, "none");
assert!(!meta.notes.is_empty());
}
// Covers. Each book is built here, a few members zipped in memory: the
// container, a package in `OEBPS/`, and whatever the case is about.
const JPEG: &[u8] = b"\xff\xd8\xff\xe0\x00\x10JFIF\x00jpeg body";
const PNG: &[u8] = b"\x89PNG\r\n\x1a\n\x00\x00\x00\x0dIHDR png body";
const GIF: &[u8] = b"GIF89a\x01\x00\x01\x00 gif body";
const WEBP: &[u8] = b"RIFF\x20\x00\x00\x00WEBPVP8 webp body";
const CONTAINER: &str = r#"<?xml version="1.0"?>
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
<rootfiles>
<rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/>
</rootfiles>
</container>"#;
/// A package with this manifest and these extra elements, around a title.
fn opf(manifest: &str, extra: &str) -> String {
format!(
r#"<?xml version="1.0"?>
<package xmlns="http://www.idpf.org/2007/opf" version="3.0">
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
<dc:title>Covered</dc:title>
</metadata>
<manifest>
<item id="text" href="text/one.xhtml" media-type="application/xhtml+xml"/>
{manifest}
</manifest>
{extra}
</package>"#
)
}
/// An EPUB of these members, after the container.
fn epub(members: &[(&str, &[u8])]) -> Vec<u8> {
use std::io::Write;
let mut zip = zip::ZipWriter::new(std::io::Cursor::new(Vec::new()));
let options = zip::write::SimpleFileOptions::default()
.compression_method(zip::CompressionMethod::Deflated);
let container: (&str, &[u8]) = ("META-INF/container.xml", CONTAINER.as_bytes());
for (name, data) in std::iter::once(&container).chain(members) {
zip.start_file(*name, options).unwrap();
zip.write_all(data).unwrap();
}
zip.finish().unwrap().into_inner()
}
fn book(package: &str, members: &[(&str, &[u8])]) -> Metadata {
let mut all = vec![("OEBPS/content.opf", package.as_bytes())];
all.extend_from_slice(members);
read("x.epub", &epub(&all))
}
#[test]
fn epub_3_names_its_cover_with_a_property() {
let package = opf(
r#"<item id="decoy-cover" href="images/cover-thumb.png" media-type="image/png"/>
<item id="c" href="images/front.jpg" media-type="image/jpeg" properties="svg cover-image"/>"#,
r#"<metadata><meta name="cover" content="decoy-cover"/></metadata>"#,
);
let meta = book(
&package,
&[
("OEBPS/images/front.jpg", JPEG),
("OEBPS/images/cover-thumb.png", PNG),
],
);
let cover = meta.cover.expect("a cover");
assert_eq!(cover.media_type, "image/jpeg");
assert_eq!(cover.bytes, JPEG);
assert!(!cover.guessed, "a declared cover is not a guess");
assert_eq!(cover.extension(), "jpg");
assert!(meta.notes.is_empty(), "{:?}", meta.notes);
}
#[test]
fn epub_2_names_its_cover_with_a_meta_holding_an_id() {
let package = opf(
r#"<opf:item id="cover-img" href="Images/Front%20Page.png#top" media-type="image/png"/>"#,
r#"<metadata><meta content="cover-img" name="cover" /></metadata>"#,
);
let meta = book(&package, &[("OEBPS/Images/Front Page.png", PNG)]);
let cover = meta.cover.expect("a cover");
assert_eq!(cover.media_type, "image/png");
assert_eq!(cover.bytes, PNG);
assert!(meta.notes.is_empty(), "{:?}", meta.notes);
}
#[test]
fn a_cover_meta_holding_an_address_is_followed_as_one() {
// Listed in the manifest under another id, and not listed at all.
let listed = opf(
r#"<item id="x1" href="../art/front.gif" media-type="image/gif"/>"#,
r#"<metadata><meta name="cover" content="../art/front.gif"/></metadata>"#,
);
let meta = book(&listed, &[("art/front.gif", GIF)]);
assert_eq!(meta.cover.expect("a cover").media_type, "image/gif");
let unlisted = opf(
"",
r#"<metadata><meta name="cover" content="front.webp"/></metadata>"#,
);
let meta = book(&unlisted, &[("OEBPS/front.webp", WEBP)]);
let cover = meta.cover.expect("a cover");
assert_eq!(cover.media_type, "image/webp");
assert_eq!(cover.bytes, WEBP);
}
#[test]
fn the_guide_cover_page_gives_the_first_image_on_it() {
let package = opf(
r#"<item id="page" href="text/cover.xhtml" media-type="application/xhtml+xml"/>
<item id="art" href="images/art.jpg" media-type="image/jpeg"/>"#,
r#"<guide><reference type="Cover" title="Cover" href="text/cover.xhtml#start"/></guide>"#,
);
let page = br#"<html><body><!-- <img src="nothing.png"/> -->
<img alt="cover" src="../images/art.jpg"/><img src="../images/later.png"/></body></html>"#;
let meta = book(
&package,
&[
("OEBPS/text/cover.xhtml", page),
("OEBPS/images/art.jpg", JPEG),
("OEBPS/images/later.png", PNG),
],
);
let cover = meta.cover.expect("a cover");
assert_eq!(cover.bytes, JPEG);
assert!(meta.notes.is_empty(), "{:?}", meta.notes);
}
#[test]
fn the_guide_cover_page_may_draw_its_image_in_svg() {
let package = opf(
"",
r#"<guide><reference type="cover" href="titlepage.xhtml"/></guide>"#,
);
for image in [
r#"<svg:image width="600" xlink:href="cover.png"/>"#,
r#"<image href="cover.png" />"#,
] {
let page = format!(r#"<html><body><svg:svg>{image}</svg:svg></body></html>"#);
let meta = book(
&package,
&[
("OEBPS/titlepage.xhtml", page.as_bytes()),
("OEBPS/cover.png", PNG),
],
);
assert_eq!(meta.cover.expect(image).bytes, PNG);
}
}
#[test]
fn last_an_image_whose_name_says_cover() {
let package = opf(
r#"<item id="pic1" href="images/plate.jpg" media-type="image/jpeg"/>
<item id="CoverImage" href="images/i2.gif" media-type="image/gif"/>"#,
"",
);
let meta = book(
&package,
&[
("OEBPS/images/plate.jpg", JPEG),
("OEBPS/images/i2.gif", GIF),
],
);
let cover = meta.cover.expect("a cover");
assert_eq!(cover.bytes, GIF);
assert!(cover.guessed, "a cover found by its name is a guess");
assert_eq!(cover.extension(), "gif");
let by_address = opf(
r#"<item id="i9" href="images/Book_COVER.jpeg" media-type="image/jpeg"/>"#,
"",
);
let meta = book(&by_address, &[("OEBPS/images/Book_COVER.jpeg", JPEG)]);
assert_eq!(meta.cover.expect("a cover").bytes, JPEG);
}
#[test]
fn a_book_that_names_no_cover_has_none_and_says_nothing() {
let package = opf(
r#"<item id="plate" href="images/plate.jpg" media-type="image/jpeg"/>"#,
"",
);
let meta = book(&package, &[("OEBPS/images/plate.jpg", JPEG)]);
assert_eq!(meta.cover, None);
assert!(meta.notes.is_empty(), "{:?}", meta.notes);
}
#[test]
fn a_mislabelled_cover_is_what_its_bytes_say() {
let package = opf(
r#"<item id="c" href="c.png" media-type="image/png" properties="cover-image"/>"#,
"",
);
let meta = book(&package, &[("OEBPS/c.png", JPEG)]);
assert_eq!(meta.cover.expect("a cover").media_type, "image/jpeg");
assert_eq!(
meta.notes,
["the manifest calls the cover, OEBPS/c.png, image/png, but it is image/jpeg"]
);
}
#[test]
fn a_cover_that_is_not_an_image_is_a_note_and_not_a_cover() {
let package = opf(
r#"<item id="c" href="c.jpg" media-type="image/jpeg" properties="cover-image"/>"#,
"",
);
let meta = book(&package, &[("OEBPS/c.jpg", b"<html>not an image</html>")]);
assert_eq!(meta.cover, None);
assert_eq!(meta.notes, ["the cover, OEBPS/c.jpg, is not an image"]);
}
#[test]
fn a_cover_larger_than_the_cap_is_left_out() {
let mut huge = JPEG.to_vec();
huge.resize(MAX_COVER + 1, 0);
let package = opf(
r#"<item id="c" href="c.jpg" media-type="image/jpeg" properties="cover-image"/>"#,
"",
);
let meta = book(&package, &[("OEBPS/c.jpg", &huge)]);
assert_eq!(meta.cover, None);
assert_eq!(
meta.notes,
["the cover, OEBPS/c.jpg, is larger than 16 MiB"]
);
}
/// Read one member through `Look::fetch` with a cap of `cap`.
fn fetch(bytes: Vec<u8>, cap: usize) -> (Option<usize>, Vec<String>) {
let mut meta = Metadata::empty("x.epub", Source::Opf);
let mut archive = zip::ZipArchive::new(std::io::Cursor::new(bytes)).unwrap();
let mut look = Look {
archive: &mut archive,
items: &[],
tried: Vec::new(),
meta: &mut meta,
};
let got = look.fetch("big.jpg", cap, "cover").map(|bytes| bytes.len());
(got, meta.notes)
}
#[test]
fn a_member_is_read_no_further_than_its_cap() {
let bytes = epub(&[("big.jpg", &[0u8; 4096])]);
assert_eq!(fetch(bytes.clone(), 4096).0, Some(4096));
assert_eq!(
fetch(bytes, 4095),
(
None,
vec!["the cover, big.jpg, is larger than 4095 bytes".to_string()]
)
);
}
#[test]
fn a_member_whose_archive_understates_its_size_is_still_bounded() {
let mut bytes = epub(&[("big.jpg", &[0u8; 4096])]);
// Declare 10 bytes uncompressed, in the local header and in the
// central directory, where 4096 are compressed in.
// The name comes 30 bytes into a local header and 46 into a central
// directory entry, and the uncompressed size 22 and 24.
let local = find(&bytes, b"big.jpg") - 30;
bytes[local + 22..local + 26].copy_from_slice(&10u32.to_le_bytes());
let central = local + 31 + find(&bytes[local + 31..], b"big.jpg") - 46;
assert_eq!(&bytes[central..central + 4], b"PK\x01\x02");
bytes[central + 24..central + 28].copy_from_slice(&10u32.to_le_bytes());
// The declaration passes the check, and the read stops at the cap.
let (got, notes) = fetch(bytes, 100);
assert_eq!(got, None);
assert_eq!(notes, ["the cover, big.jpg, is larger than 100 bytes"]);
}
fn find(haystack: &[u8], needle: &[u8]) -> usize {
haystack
.windows(needle.len())
.position(|window| window == needle)
.expect("present")
}
#[test]
fn a_cover_named_but_missing_is_a_note_and_the_next_route_is_asked() {
let package = opf(
r#"<item id="c" href="gone.jpg" media-type="image/jpeg" properties="cover-image"/>"#,
"",
);
let meta = book(&package, &[]);
assert_eq!(meta.cover, None);
assert_eq!(
meta.notes,
["the cover, OEBPS/gone.jpg, is not in the archive"]
);
// A missing page is said the same way.
let page = opf(
"",
r#"<guide><reference type="cover" href="none.xhtml"/></guide>"#,
);
assert_eq!(
book(&page, &[]).notes,
["the cover page, OEBPS/none.xhtml, is not in the archive"]
);
// The same file reached again by another route is not read twice,
// and an image the name gives away still stands in.
let package = opf(
r#"<item id="cover" href="gone.jpg" media-type="image/jpeg" properties="cover-image"/>
<item id="cover-small" href="small.png" media-type="image/png"/>"#,
r#"<metadata><meta name="cover" content="cover"/></metadata>"#,
);
let meta = book(&package, &[("OEBPS/small.png", PNG)]);
assert_eq!(meta.cover.expect("the stand-in").bytes, PNG);
assert_eq!(
meta.notes,
["the cover, OEBPS/gone.jpg, is not in the archive"]
);
}
#[test]
fn an_address_never_leaves_the_archive() {
assert_eq!(resolve("OEBPS", "../c.jpg").as_deref(), Some("c.jpg"));
assert_eq!(resolve("OEBPS", "/c.jpg").as_deref(), Some("c.jpg"));
assert_eq!(
resolve("OEBPS/text", "./a/../b.jpg").as_deref(),
Some("OEBPS/text/b.jpg")
);
assert_eq!(resolve("", "a%2Fb.jpg").as_deref(), Some("a/b.jpg"));
for hostile in [
"../../c.jpg",
"../../../etc/passwd",
"http://example.com/c.jpg",
"data:image/png;base64,AAAA",
"file:///etc/passwd",
"%ZZ.jpg",
"%ff.jpg",
"c%",
"#only-a-fragment",
"",
] {
assert_eq!(resolve("OEBPS", hostile), None, "{hostile}");
}
let package = opf(
r#"<item id="c" href="../../c.jpg" media-type="image/jpeg" properties="cover-image"/>"#,
"",
);
let meta = book(&package, &[("c.jpg", JPEG)]);
assert_eq!(meta.cover, None);
assert_eq!(
meta.notes,
["the cover's address, ../../c.jpg, is not a path inside the archive"]
);
}
#[test]
fn an_attribute_is_matched_by_its_whole_name() {
let tag = r#"item data-id="no" xml:id="no" id = 'yes' href="a&amp;b.jpg""#;
assert_eq!(attr(tag, "id").as_deref(), Some("yes"));
assert_eq!(attr(tag, "href").as_deref(), Some("a&b.jpg"));
assert_eq!(attr(r#"image xlink:href="x.png""#, "href"), None);
assert_eq!(attr(r#"item id="unterminated"#, "id"), None);
}
#[test]
fn every_truncation_of_a_book_with_a_cover_returns() {
let package = opf(
r#"<item id="c" href="c.jpg" media-type="image/jpeg" properties="cover-image"/>"#,
r#"<guide><reference type="cover" href="p.xhtml"/></guide>"#,
);
let bytes = epub(&[
("OEBPS/content.opf", package.as_bytes()),
("OEBPS/c.jpg", JPEG),
("OEBPS/p.xhtml", b"<img src=\"c.jpg\"/>"),
]);
for len in 0..=bytes.len() {
let _ = read("cut.epub", &bytes[..len]);
}
for at in 0..bytes.len() {
let mut broken = bytes.clone();
broken[at] ^= 0xff;
let _ = read("broken.epub", &broken);
}
}
}
+55 -2
View File
@@ -106,10 +106,63 @@ pub struct Metadata {
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Cover {
/// What kind of image it is, read off its first bytes: `image/jpeg`,
/// `image/png`, `image/gif` or `image/bmp`.
/// `image/png`, `image/gif`, `image/webp` or `image/bmp`.
pub media_type: String,
/// The image itself.
/// The image itself, never more than [`MAX_COVER`] bytes.
pub bytes: Vec<u8>,
/// Whether the file left the choice open. A MOBI that names no cover, or
/// an EPUB whose only cover is an image with "cover" in its name, has one
/// picked by position or by name, and that pick can be an illustration
/// from inside the book. A caller with a better source should prefer it.
pub guessed: bool,
}
impl Cover {
/// The usual file extension for the image, without the dot.
pub fn extension(&self) -> &'static str {
match self.media_type.as_str() {
"image/png" => "png",
"image/gif" => "gif",
"image/webp" => "webp",
"image/bmp" => "bmp",
_ => "jpg",
}
}
}
/// How large a cover is worth keeping.
///
/// A real one is a few hundred kilobytes. An EPUB's cover has to be
/// decompressed to be read, and this is as far as that goes: a larger one is
/// left out with a note, in every format, so a hostile file cannot make a
/// reader allocate whatever it likes.
pub const MAX_COVER: usize = 16 * 1024 * 1024;
/// A size, as a note says it.
pub(crate) fn size(bytes: usize) -> String {
const MIB: usize = 1024 * 1024;
if bytes >= MIB && bytes % MIB == 0 {
format!("{} MiB", bytes / MIB)
} else {
format!("{bytes} bytes")
}
}
/// What kind of image these bytes are, by their magic.
pub(crate) fn image_type(data: &[u8]) -> Option<&'static str> {
if data.starts_with(&[0xff, 0xd8, 0xff]) {
Some("image/jpeg")
} else if data.starts_with(b"\x89PNG\r\n\x1a\n") {
Some("image/png")
} else if data.starts_with(b"GIF87a") || data.starts_with(b"GIF89a") {
Some("image/gif")
} else if data.starts_with(b"RIFF") && data.get(8..12) == Some(b"WEBP") {
Some("image/webp")
} else if data.starts_with(b"BM") && data.len() > 14 {
Some("image/bmp")
} else {
None
}
}
fn version() -> u32 {
+13 -16
View File
@@ -13,7 +13,7 @@
//! panics or reads past the end.
use crate::epub::{digits_of, is_isbn, is_uuid};
use crate::{Cover, Metadata, Source};
use crate::{Cover, MAX_COVER, Metadata, Source, image_type};
/// Where a Palm database says what it holds.
const TYPE_AT: usize = 60;
@@ -332,9 +332,17 @@ fn cover(
meta.note("the cover record is not an image");
return None;
};
if data.len() > MAX_COVER {
meta.note(format!(
"the cover record is larger than {}",
crate::size(MAX_COVER)
));
return None;
}
Some(Cover {
media_type: media_type.to_string(),
bytes: data.to_vec(),
guessed: false,
})
}
@@ -347,10 +355,14 @@ fn cover(
fn first_image_record(bytes: &[u8], records: &[usize], first_image: Option<u32>) -> Option<Cover> {
let first = first_image.filter(|&index| index != NONE)?;
let data = record(bytes, records, first as usize)?;
if data.len() > MAX_COVER {
return None;
}
let media_type = image_type(data).filter(|kind| *kind != "image/bmp")?;
Some(Cover {
media_type: media_type.to_string(),
bytes: data.to_vec(),
guessed: true,
})
}
@@ -359,21 +371,6 @@ fn is_asin(value: &str) -> bool {
value.len() == 10 && value.chars().all(|c| c.is_ascii_alphanumeric())
}
/// What kind of image these bytes are, by their magic.
fn image_type(data: &[u8]) -> Option<&'static str> {
if data.starts_with(&[0xff, 0xd8, 0xff]) {
Some("image/jpeg")
} else if data.starts_with(b"\x89PNG\r\n\x1a\n") {
Some("image/png")
} else if data.starts_with(b"GIF87a") || data.starts_with(b"GIF89a") {
Some("image/gif")
} else if data.starts_with(b"BM") && data.len() > 14 {
Some("image/bmp")
} else {
None
}
}
/// The Palm database's name, the last place a title can come from.
///
/// Converters write the title there with spaces made underscores; it is
+2
View File
@@ -31,6 +31,7 @@ fn says_what_it_was_given(meta: &bookmeta::Metadata) {
let cover = meta.cover.as_ref().expect("a cover");
assert_eq!(cover.media_type, "image/jpeg");
assert!(!cover.guessed, "EXTH 201 names the cover");
let original = std::fs::read(fixture("lantern-cover.jpg")).expect("the cover fixture");
assert_eq!(cover.bytes, original);
}
@@ -84,6 +85,7 @@ fn a_file_naming_no_cover_gets_its_first_image_record() {
let meta = bookmeta::from_bytes("x.mobi", &lantern_naming_no_cover());
let cover = meta.cover.expect("the first image record");
assert_eq!(cover.media_type, "image/jpeg");
assert!(cover.guessed, "a cover taken by position is a guess");
let original = std::fs::read(fixture("lantern-cover.jpg")).expect("the cover fixture");
assert_eq!(cover.bytes, original);
}
+30 -2
View File
@@ -26,8 +26,8 @@ file too small to be a book, a name that says "sample".
cover images that ended up in a folder of PDFs.
A library accumulates files that lie: a DjVu named `.pdf`, a Word document
named `.epub`, an EPUB saved as `.zip`, a single PDF that somebody zipped and
renamed back to `.pdf`.
named `.epub`, an RTF saved as `.doc`, an EPUB saved as `.zip`, a single PDF
that somebody zipped and renamed back to `.pdf`.
## Install
@@ -89,7 +89,12 @@ one bad file does not stop a sweep.
| `ole2` | The compound-file header every binary Word document starts with |
| `epub` | A zip holding `mimetype` and `META-INF/container.xml` |
| `zip-wrapped-pdf` | A zip whose only real member is one PDF |
| `zip-wrapped-fb2` | A zip whose only real member is named `.fb2` — the `.fb2.zip` most FictionBooks are passed around as |
| `cbz` | A zip whose every real member is a page image (JPEG, PNG, GIF or WebP), and which carries a `ComicInfo.xml` or is itself named `.cbz` or `.cbr` |
| `azw3`, `mobi` | A Palm database tagged `BOOKMOBI`, told apart by the format version inside the first record |
| `lit` | `ITOLITLS`, Microsoft Reader's signature |
| `rtf` | `{\rtf` at the front, after any byte-order mark and blank space, followed by the version digit, a space or the next control word |
| `fb2` | XML whose root element is `FictionBook`, found by stepping over the declaration, comments and doctype in the first four kilobytes; UTF-16 with a byte-order mark is decoded first |
| `html` | Markup at the front, after any byte-order mark |
| `txt` | Text that is none of the above |
| `unknown` | Anything else, with a note saying what was seen |
@@ -98,6 +103,29 @@ An archive holding several PDFs is a collection, not a wrapped book: choosing
among them would be a guess, and naming one chapter as the whole work is worse
than refusing.
A zipped FictionBook is reported as wrapped rather than as `fb2`, the same way
a zipped PDF is: a converter handed it as FictionBook is handed a zip, and the
kind is there to stop exactly that.
A comic is the pages and nothing else. Directories, the `ComicInfo.xml` a
comic reader writes into the archive, and the desktop clutter every archive
picks up are passed over; anything more — one PDF among the pages, a readme —
and the zip is not a comic, and says what it holds instead.
A zip of pictures is as often a holiday as a comic, so images alone are not
enough. It is a comic when it carries a `ComicInfo.xml`, or when its own name
ends `.cbz` or `.cbr`. Otherwise it is `unknown`, with a note saying it is a
zip of so many images and would be a comic book if its name said `.cbz`.
Members are judged by name. Only the archive's directory is read, so no
member is ever unpacked and a page called `001.jpg` is taken to be an image
without its bytes being looked at. The same goes for a wrapped FictionBook:
the one member is taken to be a FictionBook because its name ends `.fb2`.
FictionBook is read as bytes rather than as text. A great many are written in
windows-1251, which spells its markup in ASCII but is not UTF-8, and would
otherwise be called binary.
## As a library
```rust
+261
View File
@@ -0,0 +1,261 @@
//! Telling a FictionBook from any other XML.
//!
//! A FictionBook is an XML document whose root element is `FictionBook`.
//! The root comes after the prolog — a declaration, comments, perhaps a
//! doctype — so it is found by stepping over those rather than by matching
//! at a fixed offset.
//!
//! The walk is over bytes, not text. A great many FictionBooks are written in
//! windows-1251, which is not UTF-8 but spells its markup in ASCII all the
//! same, and a reader that insisted on UTF-8 would call every one of them
//! binary.
//!
//! A FictionBook saved as UTF-16 is the one case where the markup is not
//! ASCII bytes. Its byte-order mark says so, and the front is decoded before
//! the same walk is made over it.
use crate::BOM;
/// The UTF-16 byte-order mark, little-endian.
const UTF16_LE: &[u8] = b"\xff\xfe";
/// The UTF-16 byte-order mark, big-endian.
const UTF16_BE: &[u8] = b"\xfe\xff";
/// Most UTF-16 code units decoded from the front: the same window, in
/// characters, as the bytes read for every other encoding.
const UTF16_UNITS: usize = 2048;
/// What the root element of a FictionBook is called.
const ROOT: &[u8] = b"FictionBook";
/// Whether the front of a file is the start of a FictionBook.
///
/// `head` is however much of the front the caller read. A root element that
/// does not begin inside it is not found, so the answer is no rather than a
/// guess.
pub fn is_fictionbook(head: &[u8]) -> bool {
match utf16(head) {
Some(decoded) => is_fictionbook_root(decoded.as_bytes()),
None => is_fictionbook_root(head),
}
}
fn is_fictionbook_root(head: &[u8]) -> bool {
root_element(head).is_some_and(|name| local_name(name) == ROOT)
}
/// The front of a UTF-16 file as UTF-8, if a byte-order mark says it is one.
///
/// Only a bounded prefix is decoded. A half character at the end, or a
/// surrogate without its pair, becomes a replacement character, which the
/// walk takes as part of a name and so never as the end of one.
fn utf16(head: &[u8]) -> Option<String> {
let (body, big_endian) = match head.strip_prefix(UTF16_LE) {
Some(body) => (body, false),
None => (head.strip_prefix(UTF16_BE)?, true),
};
let units = body.chunks_exact(2).take(UTF16_UNITS).map(|pair| {
let pair = [pair[0], pair[1]];
if big_endian {
u16::from_be_bytes(pair)
} else {
u16::from_le_bytes(pair)
}
});
Some(
char::decode_utf16(units)
.map(|c| c.unwrap_or(char::REPLACEMENT_CHARACTER))
.collect(),
)
}
/// The name of the first element, after the prolog.
fn root_element(head: &[u8]) -> Option<&[u8]> {
let mut rest = head.strip_prefix(BOM).unwrap_or(head);
loop {
rest = rest.trim_ascii_start();
if let Some(after) = rest.strip_prefix(b"<?") {
rest = after_marker(after, b"?>")?;
} else if let Some(after) = rest.strip_prefix(b"<!--") {
rest = after_marker(after, b"-->")?;
} else if let Some(after) = rest.strip_prefix(b"<!") {
rest = after_declaration(after)?;
} else {
// Anything but an element here is text before the root, which
// XML does not allow.
let after = rest.strip_prefix(b"<")?;
// A name that runs to the end of what was read may be the front
// of a longer one, so it is not taken.
let end = after.iter().position(|byte| !is_name_byte(*byte))?;
let ends_properly =
matches!(after[end], b'>' | b'/') || after[end].is_ascii_whitespace();
return (end > 0 && ends_properly).then_some(&after[..end]);
}
}
}
/// What follows the first `marker`, if it is there at all.
fn after_marker<'a>(rest: &'a [u8], marker: &[u8]) -> Option<&'a [u8]> {
rest.windows(marker.len())
.position(|window| window == marker)
.map(|at| &rest[at + marker.len()..])
}
/// What follows a doctype, whose internal subset may hold `>` of its own.
fn after_declaration(rest: &[u8]) -> Option<&[u8]> {
let mut depth = 0usize;
let mut quote = None;
for (at, &byte) in rest.iter().enumerate() {
match quote {
Some(open) if byte == open => quote = None,
Some(_) => {}
None => match byte {
b'"' | b'\'' => quote = Some(byte),
b'[' => depth += 1,
b']' => depth = depth.saturating_sub(1),
b'>' if depth == 0 => return Some(&rest[at + 1..]),
_ => {}
},
}
}
None
}
/// The name without its namespace prefix.
fn local_name(name: &[u8]) -> &[u8] {
match name.iter().rposition(|byte| *byte == b':') {
Some(at) => &name[at + 1..],
None => name,
}
}
fn is_name_byte(byte: u8) -> bool {
byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'-' | b'.' | b':') || byte >= 0x80
}
#[cfg(test)]
mod tests {
use super::*;
const BOOK: &[u8] = b"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n\
<FictionBook xmlns=\"http://www.gribuser.ru/xml/fictionbook/2.0\">\n\
<description><title-info><book-title>The Mill</book-title></title-info></description>";
#[test]
fn a_fictionbook_is_recognised() {
assert!(is_fictionbook(BOOK));
assert!(is_fictionbook(b"<FictionBook>"));
}
#[test]
fn the_prolog_is_stepped_over() {
let head = b"\xef\xbb\xbf <?xml version=\"1.0\"?>\r\n\
<!-- converted by hand -->\n\
<!DOCTYPE FictionBook [ <!ENTITY mill \"<b>the mill</b>\"> ]>\n\
<?xml-stylesheet href=\"fb2.css\"?>\n\
<FictionBook\n xmlns=\"http://www.gribuser.ru/xml/fictionbook/2.0\">";
assert!(is_fictionbook(head));
}
#[test]
fn a_prefixed_root_is_still_a_fictionbook() {
assert!(is_fictionbook(
b"<fb:FictionBook xmlns:fb=\"http://www.gribuser.ru/xml/fictionbook/2.0\">"
));
}
#[test]
fn markup_need_not_be_utf8() {
// windows-1251 for "Мельница" in the title.
let head = b"<?xml version=\"1.0\" encoding=\"windows-1251\"?>\
<FictionBook><description><book-title>\xcc\xe5\xeb\xfc\xed\xe8\xf6\xe0</book-title>";
assert!(is_fictionbook(head));
}
#[test]
fn other_xml_is_not_a_fictionbook() {
assert!(!is_fictionbook(
b"<?xml version=\"1.0\"?>\n<rss version=\"2.0\">"
));
assert!(!is_fictionbook(b"<?xml version=\"1.0\"?><FictionBookish>"));
assert!(!is_fictionbook(b"<html><FictionBook>"));
assert!(!is_fictionbook(b"text before <FictionBook>"));
assert!(!is_fictionbook(b""));
}
#[test]
fn a_root_the_window_cut_short_is_not_taken() {
assert!(!is_fictionbook(b"<?xml version=\"1.0\"?><FictionBook"));
assert!(!is_fictionbook(b"<?xml version=\"1.0\"?><Fiction"));
assert!(!is_fictionbook(b"<?xml version=\"1.0\""));
assert!(!is_fictionbook(b"<!-- never closed <FictionBook>"));
}
fn utf16(text: &str, big_endian: bool) -> Vec<u8> {
let mut out = if big_endian {
UTF16_BE.to_vec()
} else {
UTF16_LE.to_vec()
};
for unit in text.encode_utf16() {
out.extend_from_slice(&if big_endian {
unit.to_be_bytes()
} else {
unit.to_le_bytes()
});
}
out
}
const UTF16_BOOK: &str = "<?xml version=\"1.0\" encoding=\"UTF-16\"?>\r\n\
<FictionBook xmlns=\"http://www.gribuser.ru/xml/fictionbook/2.0\">\
<description><book-title>Мельница</book-title>";
#[test]
fn a_little_endian_utf16_fictionbook_is_recognised() {
assert!(is_fictionbook(&utf16(UTF16_BOOK, false)));
assert!(!is_fictionbook(&utf16(
"<?xml version=\"1.0\"?><rss>",
false
)));
}
#[test]
fn a_big_endian_utf16_fictionbook_is_recognised() {
assert!(is_fictionbook(&utf16(UTF16_BOOK, true)));
assert!(!is_fictionbook(&utf16(
"<?xml version=\"1.0\"?><rss>",
true
)));
}
#[test]
fn utf16_read_in_the_wrong_order_is_not_a_fictionbook() {
let mut swapped = utf16(UTF16_BOOK, false);
swapped[..2].copy_from_slice(UTF16_BE);
assert!(!is_fictionbook(&swapped));
}
#[test]
fn a_utf16_root_the_window_cut_short_is_not_taken() {
let head = utf16("<FictionBook>", false);
for cut in 0..head.len() {
assert!(!is_fictionbook(&head[..cut]), "cut at {cut}");
}
assert!(is_fictionbook(&head));
}
#[test]
fn a_truncated_or_mangled_prolog_does_not_panic() {
let head = b"\xef\xbb\xbf<?xml?><!--c--><!DOCTYPE x [ <!ENTITY a '>'> ] \"]\">\
<FictionBook>";
assert!(is_fictionbook(head));
for cut in 0..head.len() {
let _ = is_fictionbook(&head[..cut]);
}
for byte in 0..=255u8 {
let _ = is_fictionbook(&[b'<', byte]);
let _ = is_fictionbook(&[b'<', b'!', byte, b'>']);
}
}
}
+378 -13
View File
@@ -9,6 +9,7 @@
//! The containers are unambiguous, so they are read rather than guessed at.
pub mod bytes;
pub mod fb2;
pub mod mobi;
pub mod zip;
@@ -22,17 +23,35 @@ const DJVU: &[u8] = b"AT&TFORM";
const OLE2: &[u8] = &[0xd0, 0xcf, 0x11, 0xe0, 0xa1, 0xb1, 0x1a, 0xe1];
/// A PDF announces its version in its first bytes.
const PDF: &[u8] = b"%PDF-";
/// Microsoft Reader's signature, ahead of everything else in the file.
const LIT: &[u8] = b"ITOLITLS";
/// The group every RTF document is, opened with the control word that names
/// the format.
const RTF: &[u8] = br"{\rtf";
/// The UTF-8 byte-order mark.
const BOM: &[u8] = b"\xef\xbb\xbf";
/// Archive members that carry no content: resource forks and desktop clutter.
const ARCHIVE_NOISE: &[&str] = &["__macosx/", ".ds_store", "thumbs.db"];
/// What a comic book archive carries besides its pages: the catalogue entry
/// comic readers write into it.
const COMIC_CATALOGUE: &str = "comicinfo.xml";
/// The extensions that say an archive is a comic book.
const COMIC_EXTENSIONS: &[&str] = &["cbz", "cbr"];
/// The extensions of the images a comic book archive is made of.
const PAGE_IMAGES: &[&str] = &[".jpg", ".jpeg", ".png", ".gif", ".webp"];
/// Substrings in a name that mark a file as a sample rather than a book.
const SAMPLE_MARKERS: &[&str] = &["sample", "readme", "read me"];
/// Below this, a file is too small to be a book whatever it claims to be.
const SMALLEST_BOOK: u64 = 4096;
/// How much of the front is read to tell text from markup.
/// How much of the front is read to tell RTF, FictionBook, markup and text
/// apart.
const TEXT_WINDOW: usize = 4096;
/// Identify a file.
@@ -49,12 +68,17 @@ pub fn sniff(source: &impl Bytes, name: &str) -> Doc {
Kind::Pdf
} else if source.magic_at(0, OLE2) {
Kind::Ole2
} else if source.magic_at(0, LIT) {
Kind::Lit
} else if source.magic_at(0, zip::LOCAL_HEADER) {
archive(source, &mut doc)
} else if let Some(kind) = mobi::kind(source) {
kind
} else {
text(source, &mut doc)
match source.slice(0, TEXT_WINDOW) {
Ok(head) => front(&head, &mut doc),
Err(_) => Kind::Unknown,
}
};
if doc.kind == Kind::Unknown {
@@ -80,7 +104,8 @@ pub fn sniff(source: &impl Bytes, name: &str) -> Doc {
doc
}
/// What is inside a zip: an EPUB, a wrapped PDF, or a collection of pages.
/// What is inside a zip: an EPUB, a wrapped book, a comic, or a collection of
/// pages.
fn archive(source: &impl Bytes, doc: &mut Doc) -> Kind {
let Some(names) = zip::members(source) else {
doc.note("a zip whose directory could not be read");
@@ -99,6 +124,28 @@ fn archive(source: &impl Bytes, doc: &mut Doc) -> Kind {
doc.note(format!("a zip holding one PDF, {pdf:?}"));
return Kind::ZipWrappedPdf;
}
if let Some(fb2) = sole_member(&names, ".fb2") {
doc.note(format!("a zip holding one FictionBook, {fb2:?}"));
return Kind::ZipWrappedFb2;
}
if let Some(pages) = comic_pages(&names) {
// A zip of pictures is as often a holiday as a comic, and only the
// names have been read. It is a comic when something says so: the
// catalogue entry a comic reader writes, or a name that claims it.
let named_comic = doc
.declared_ext
.as_deref()
.is_some_and(|ext| COMIC_EXTENSIONS.contains(&ext));
if pages.catalogued || named_comic {
return Kind::Cbz;
}
doc.note(format!(
"a zip of {} image{}, which would be a comic book if its name said .cbz",
pages.count,
if pages.count == 1 { "" } else { "s" }
));
return Kind::Unknown;
}
if lower
.iter()
.any(|name| name.ends_with(".htm") || name.ends_with(".html") || name.ends_with(".xhtml"))
@@ -119,25 +166,94 @@ fn archive(source: &impl Bytes, doc: &mut Doc) -> Kind {
/// a collection, and choosing among them would be a guess — better to say it
/// is not a book than to name one chapter as the whole.
pub fn sole_pdf(names: &[String]) -> Option<&str> {
let real: Vec<&String> = names
sole_member(names, ".pdf")
}
/// The one member an archive exists to carry, if it is the only real member
/// and its name ends in `extension`.
fn sole_member<'a>(names: &'a [String], extension: &str) -> Option<&'a str> {
match real_members(names).as_slice() {
[only] if only.to_lowercase().ends_with(extension) => Some(only.as_str()),
_ => None,
}
}
/// The pages of an archive that is images and nothing else.
struct Pages {
/// How many images there are.
count: usize,
/// Whether a comic reader's catalogue entry is there beside them.
catalogued: bool,
}
/// The pages, if an archive is images and nothing else.
///
/// Every real member has to be an image. One PDF or one text file among the
/// pages makes it something else, and saying what would be a guess. Images
/// are known by their names alone: only the archive's directory is read.
fn comic_pages(names: &[String]) -> Option<Pages> {
let mut catalogued = false;
let pages: Vec<String> = real_members(names)
.into_iter()
.map(|name| name.to_lowercase())
.filter(|name| {
let file = name.rsplit('/').next().unwrap_or(name);
let catalogue = file == COMIC_CATALOGUE;
catalogued |= catalogue;
!catalogue
})
.collect();
let all_images = !pages.is_empty()
&& pages
.iter()
.all(|name| PAGE_IMAGES.iter().any(|image| name.ends_with(image)));
all_images.then_some(Pages {
count: pages.len(),
catalogued,
})
}
/// The members that carry content: no directories, no desktop clutter.
fn real_members(names: &[String]) -> Vec<&String> {
names
.iter()
.filter(|name| !name.ends_with('/'))
.filter(|name| {
let lower = name.to_lowercase();
!ARCHIVE_NOISE.iter().any(|noise| lower.contains(noise))
})
.collect();
match real.as_slice() {
[only] if only.to_lowercase().ends_with(".pdf") => Some(only.as_str()),
_ => None,
.collect()
}
/// What the front of a file says when it is not a container: RTF, a
/// FictionBook, markup, text, or nothing this knows.
fn front(head: &[u8], doc: &mut Doc) -> Kind {
// Both are recognised on bytes rather than text. RTF is seven-bit by
// design but not always in practice, and a FictionBook is as often
// windows-1251 as UTF-8.
let lead = head.strip_prefix(BOM).unwrap_or(head).trim_ascii_start();
if is_rtf(lead) {
return Kind::Rtf;
}
if fb2::is_fictionbook(head) {
return Kind::Fb2;
}
text(head, doc)
}
/// Whether `lead` opens an RTF document.
///
/// `{\rtf` has to be the whole control word: what follows it is the version,
/// normally `1`, or the space or next control word that ends it. `{\rtfoo` is
/// some other word that happens to start the same way.
fn is_rtf(lead: &[u8]) -> bool {
lead.strip_prefix(RTF)
.and_then(|rest| rest.first())
.is_some_and(|next| next.is_ascii_digit() || next.is_ascii_whitespace() || *next == b'\\')
}
/// Text, markup, or nothing this knows.
fn text(source: &impl Bytes, doc: &mut Doc) -> Kind {
let Ok(head) = source.slice(0, TEXT_WINDOW) else {
return Kind::Unknown;
};
fn text(head: &[u8], doc: &mut Doc) -> Kind {
if head.is_empty() {
doc.note("the file is empty");
return Kind::Unknown;
@@ -149,7 +265,7 @@ fn text(source: &impl Bytes, doc: &mut Doc) -> Kind {
}
// The window may have cut a character in half, which is not the same
// thing as the file not being text.
let head = match std::str::from_utf8(&head) {
let head = match std::str::from_utf8(head) {
Ok(text) => text,
Err(e) if e.error_len().is_none() => {
std::str::from_utf8(&head[..e.valid_up_to()]).expect("a valid prefix")
@@ -178,10 +294,15 @@ fn names_the_same_thing(extension: &str, kind: Kind) -> bool {
Kind::Djvu => matches!(extension, "djvu" | "djv"),
Kind::Azw3 => matches!(extension, "azw3" | "azw" | "kfx"),
Kind::Mobi => matches!(extension, "mobi" | "prc" | "azw"),
Kind::Lit => extension == "lit",
Kind::Fb2 => extension == "fb2",
Kind::Rtf => extension == "rtf",
Kind::Txt => matches!(extension, "txt" | "text" | "md" | "markdown"),
Kind::Html => matches!(extension, "html" | "htm" | "xhtml" | "xml"),
Kind::Ole2 => matches!(extension, "doc" | "xls" | "ppt"),
Kind::ZipWrappedPdf => extension == "zip",
Kind::ZipWrappedFb2 => matches!(extension, "zip" | "fbz"),
Kind::Cbz => extension == "cbz",
// Nothing was recognised, so there is nothing to disagree with.
Kind::Unknown => true,
}
@@ -194,10 +315,15 @@ fn spelling(kind: Kind) -> &'static str {
Kind::Djvu => "DjVu",
Kind::Azw3 => "AZW3",
Kind::Mobi => "MOBI",
Kind::Lit => "Microsoft Reader",
Kind::Fb2 => "FictionBook",
Kind::Rtf => "RTF",
Kind::Txt => "plain text",
Kind::Html => "HTML",
Kind::Ole2 => "an OLE2 document",
Kind::ZipWrappedPdf => "a zip holding one PDF",
Kind::ZipWrappedFb2 => "a zip holding one FictionBook",
Kind::Cbz => "a comic book archive",
Kind::Unknown => "nothing recognisable",
}
}
@@ -332,4 +458,243 @@ mod tests {
head.truncate(TEXT_WINDOW + 1);
assert_eq!(sniff(&head.as_slice(), "a.txt").kind, Kind::Txt);
}
fn archive_of(names: &[&str], name: &str) -> Doc {
sniff(&zip::tests::zip(names).as_slice(), name)
}
#[test]
fn rtf_is_recognised_by_its_opening_group() {
assert_eq!(kind_of(br"{\rtf1\ansi\deff0 {\fonttbl", "a.rtf"), Kind::Rtf);
assert_eq!(
kind_of(b"\xef\xbb\xbf\r\n {\\rtf1\\ansi", "a.rtf"),
Kind::Rtf
);
// Eight-bit text in the body does not stop it being RTF.
assert_eq!(kind_of(b"{\\rtf1 caf\xe9 \x00", "a.rtf"), Kind::Rtf);
}
#[test]
fn rtf_cut_short_of_its_control_word_is_not_rtf() {
assert_eq!(sniff(&br"{\rt".as_slice(), "a.rtf").kind, Kind::Txt);
assert_eq!(sniff(&br"{\rtf".as_slice(), "a.rtf").kind, Kind::Txt);
assert_eq!(kind_of(br"{\rtx1", "a.rtf"), Kind::Txt);
assert_eq!(kind_of(br"text {\rtf1", "a.rtf"), Kind::Txt);
}
#[test]
fn rtf_is_the_whole_control_word() {
assert_eq!(kind_of(br"{\rtf\ansi\deff0", "a.rtf"), Kind::Rtf);
assert_eq!(kind_of(b"{\\rtf \\ansi", "a.rtf"), Kind::Rtf);
assert_eq!(kind_of(b"{\\rtf\r\n\\ansi", "a.rtf"), Kind::Rtf);
assert_eq!(kind_of(br"{\rtf2", "a.rtf"), Kind::Rtf);
assert_eq!(kind_of(br"{\rtfoo bar}", "a.rtf"), Kind::Txt);
assert_eq!(kind_of(br"{\rtf}", "a.rtf"), Kind::Txt);
assert_eq!(kind_of(br"{\rtf-1", "a.rtf"), Kind::Txt);
}
#[test]
fn rtf_named_as_a_word_document_is_remarked_on() {
let doc = sniff(&padded(br"{\rtf1\ansi").as_slice(), "letter.doc");
assert_eq!(doc.kind, Kind::Rtf);
assert!(
doc.notes
.iter()
.any(|note| note == "named .doc, but the bytes say RTF"),
"{:?}",
doc.notes
);
}
#[test]
fn microsoft_reader_is_recognised_by_its_signature() {
let doc = sniff(&padded(b"ITOLITLS\x01\x00\x00\x00").as_slice(), "book.lit");
assert_eq!(doc.kind, Kind::Lit);
assert!(doc.notes.is_empty(), "{:?}", doc.notes);
assert_eq!(kind_of(b"ITOLITL", "book.lit"), Kind::Txt);
assert_eq!(kind_of(b" ITOLITLS", "book.lit"), Kind::Txt);
}
#[test]
fn a_fictionbook_is_recognised_and_other_xml_is_not() {
let book = b"<?xml version=\"1.0\" encoding=\"windows-1251\"?>\n\
<FictionBook xmlns=\"http://www.gribuser.ru/xml/fictionbook/2.0\">\
<book-title>\xcc\xe5\xeb\xfc\xed\xe8\xf6\xe0</book-title>";
let doc = sniff(&padded(book).as_slice(), "mill.fb2");
assert_eq!(doc.kind, Kind::Fb2);
assert!(doc.notes.is_empty(), "{:?}", doc.notes);
let feed = b"<?xml version=\"1.0\"?>\n<rss version=\"2.0\"><channel>";
assert_eq!(kind_of(feed, "feed.xml"), Kind::Html);
}
#[test]
fn a_utf16_fictionbook_is_recognised_in_either_byte_order() {
let book = "<?xml version=\"1.0\" encoding=\"UTF-16\"?>\n<FictionBook>";
for big_endian in [false, true] {
let mut bytes = if big_endian {
vec![0xfe, 0xff]
} else {
vec![0xff, 0xfe]
};
for unit in book.encode_utf16() {
bytes.extend_from_slice(&if big_endian {
unit.to_be_bytes()
} else {
unit.to_le_bytes()
});
}
let doc = sniff(&padded(&bytes).as_slice(), "mill.fb2");
assert_eq!(doc.kind, Kind::Fb2, "big-endian: {big_endian}");
assert!(doc.notes.is_empty(), "{:?}", doc.notes);
}
}
#[test]
fn a_fictionbook_named_as_something_else_is_remarked_on() {
let doc = sniff(&padded(b"<FictionBook>").as_slice(), "mill.txt");
assert_eq!(doc.kind, Kind::Fb2);
assert!(
doc.notes.iter().any(|note| note.contains("FictionBook")),
"{:?}",
doc.notes
);
}
#[test]
fn a_zip_holding_one_fictionbook_is_a_wrapped_one() {
let doc = archive_of(&["__MACOSX/._mill.fb2", "mill.fb2"], "mill.fb2.zip");
assert_eq!(doc.kind, Kind::ZipWrappedFb2);
assert_eq!(doc.declared_ext.as_deref(), Some("zip"));
assert!(
doc.notes
.iter()
.any(|note| note.contains("one FictionBook")),
"{:?}",
doc.notes
);
assert!(!doc.notes.iter().any(|note| note.contains("named .")));
let two = archive_of(&["one.fb2", "two.fb2"], "pair.zip");
assert_eq!(two.kind, Kind::Unknown);
}
#[test]
fn a_zip_of_page_images_is_a_comic() {
let doc = archive_of(
&[
"Issue 1/",
"Issue 1/001.jpg",
"Issue 1/002.JPEG",
"Issue 1/003.png",
"Issue 1/004.gif",
"Issue 1/005.webp",
"ComicInfo.xml",
"__MACOSX/Issue 1/._001.jpg",
"Issue 1/Thumbs.db",
".DS_Store",
],
"issue-1.cbz",
);
assert_eq!(doc.kind, Kind::Cbz);
assert!(
!doc.notes.iter().any(|note| note.contains("named .")),
"{:?}",
doc.notes
);
}
#[test]
fn a_zip_of_photographs_is_not_a_comic() {
let doc = archive_of(&["IMG_0001.jpg", "IMG_0002.jpg"], "holiday.zip");
assert_eq!(doc.kind, Kind::Unknown);
assert!(
doc.notes.iter().any(|note| note
== "a zip of 2 images, which would be a comic book if its name said .cbz"),
"{:?}",
doc.notes
);
let one = archive_of(&["IMG_0001.jpg"], "holiday.zip");
assert!(
one.notes
.iter()
.any(|note| note.starts_with("a zip of 1 image,")),
"{:?}",
one.notes
);
}
#[test]
fn the_same_images_named_as_a_comic_are_one() {
let doc = archive_of(&["IMG_0001.jpg", "IMG_0002.jpg"], "holiday.cbz");
assert_eq!(doc.kind, Kind::Cbz);
assert!(
!doc.notes.iter().any(|note| note.contains("named .")),
"{:?}",
doc.notes
);
// A comic zipped and named as if it were RAR is still a comic, and
// the name is remarked on.
let rar = archive_of(&["001.jpg", "002.jpg"], "issue-1.cbr");
assert_eq!(rar.kind, Kind::Cbz);
assert!(
rar.notes.iter().any(|note| note.contains("named .cbr")),
"{:?}",
rar.notes
);
}
#[test]
fn a_comic_saved_as_a_zip_is_known_by_its_catalogue_and_remarked_on() {
let doc = archive_of(&["001.jpg", "002.jpg", "ComicInfo.xml"], "issue-1.zip");
assert_eq!(doc.kind, Kind::Cbz);
assert!(
doc.notes
.iter()
.any(|note| note == "named .zip, but the bytes say a comic book archive"),
"{:?}",
doc.notes
);
}
#[test]
fn a_zip_with_anything_but_images_is_not_a_comic() {
let one_pdf = archive_of(&["001.jpg", "002.jpg", "extras.pdf"], "issue-1.cbz");
assert_eq!(one_pdf.kind, Kind::Unknown);
assert!(
one_pdf
.notes
.iter()
.any(|note| note.contains("none of them a book")),
"{:?}",
one_pdf.notes
);
let readme = archive_of(&["001.jpg", "readme.txt"], "issue-1.cbz");
assert_eq!(readme.kind, Kind::Unknown);
// A name that ends like an image without being one.
let lookalike = archive_of(&["001.jpg", "notes.jpg.txt"], "issue-1.cbz");
assert_eq!(lookalike.kind, Kind::Unknown);
}
#[test]
fn a_zip_with_no_pages_is_not_a_comic() {
let catalogue_only = archive_of(&["ComicInfo.xml", "pages/"], "issue-1.cbz");
assert_eq!(catalogue_only.kind, Kind::Unknown);
let empty = archive_of(&[], "issue-1.cbz");
assert_eq!(empty.kind, Kind::Unknown);
}
#[test]
fn a_truncated_comic_does_not_panic() {
let whole = zip::tests::zip(&["001.jpg", "002.png", "ComicInfo.xml"]);
for cut in 0..whole.len() {
let _ = sniff(&&whole[..cut], "issue-1.cbz");
}
}
}
+2 -2
View File
@@ -78,11 +78,11 @@ fn directory_start(source: &impl Bytes) -> Option<(usize, u64)> {
}
#[cfg(test)]
mod tests {
pub(crate) mod tests {
use super::*;
/// Build a zip holding `names`, each with empty stored content.
fn zip(names: &[&str]) -> Vec<u8> {
pub(crate) fn zip(names: &[&str]) -> Vec<u8> {
let mut out: Vec<u8> = Vec::new();
let mut offsets = Vec::new();
for name in names {
+24
View File
@@ -64,6 +64,30 @@ fn several_files_come_back_as_several_records() {
assert_eq!(kinds, ["\"pdf\"", "\"djvu\""]);
}
#[test]
fn the_formats_read_from_the_front_come_back_by_their_wire_names() {
let rtf = written("letter.rtf", &padded(br"{\rtf1\ansi"));
let lit = written("book.lit", &padded(b"ITOLITLS"));
let fb2 = written(
"mill.fb2",
&padded(b"<?xml version=\"1.0\"?>\n<FictionBook xmlns=\"http://www.gribuser.ru/xml/fictionbook/2.0\">"),
);
let out = run(&[
rtf.to_str().expect("utf-8 path"),
lit.to_str().expect("utf-8 path"),
fb2.to_str().expect("utf-8 path"),
]);
assert_eq!(out.status.code(), Some(0));
let kinds: Vec<String> = String::from_utf8(out.stdout)
.expect("utf-8")
.lines()
.map(|line| {
serde_json::from_str::<serde_json::Value>(line).expect("valid JSON")["kind"].to_string()
})
.collect();
assert_eq!(kinds, ["\"rtf\"", "\"lit\"", "\"fb2\""]);
}
#[test]
fn a_file_that_is_not_there_is_an_io_error_and_the_rest_still_run() {
let path = written("kept.pdf", &padded(b"%PDF-1.7"));
+15
View File
@@ -77,6 +77,12 @@ pub enum Kind {
Azw3,
/// Amazon MOBI.
Mobi,
/// Microsoft Reader.
Lit,
/// FictionBook, the XML ebook format.
Fb2,
/// Rich Text Format.
Rtf,
/// Plain text.
Txt,
/// HTML or XHTML.
@@ -85,6 +91,10 @@ pub enum Kind {
Ole2,
/// A zip archive whose only real content is a single PDF.
ZipWrappedPdf,
/// A zip archive whose only real content is a single FictionBook.
ZipWrappedFb2,
/// A comic book archive: a zip of page images.
Cbz,
/// Recognised as none of the above.
Unknown,
}
@@ -118,6 +128,11 @@ mod tests {
(Kind::Ole2, "\"ole2\""),
(Kind::Azw3, "\"azw3\""),
(Kind::ZipWrappedPdf, "\"zip-wrapped-pdf\""),
(Kind::ZipWrappedFb2, "\"zip-wrapped-fb2\""),
(Kind::Fb2, "\"fb2\""),
(Kind::Lit, "\"lit\""),
(Kind::Rtf, "\"rtf\""),
(Kind::Cbz, "\"cbz\""),
(Kind::Unknown, "\"unknown\""),
];
for (kind, wire) in cases {
+2
View File
@@ -30,6 +30,8 @@ pub const UNAVAILABLE: u8 = 69;
pub const SOFTWARE: u8 = 70;
/// Reading or writing failed.
pub const IO: u8 = 74;
/// A service refused for now, and asking again later will work.
pub const TEMPFAIL: u8 = 75;
/// The answer a predicate tool gives.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+1
View File
@@ -47,6 +47,7 @@ that was concatenated or written on Windows still reads.
| `69` | A program it needs is not installed |
| `70` | Its own fault |
| `74` | Reading or writing failed |
| `75` | A service refused for now; try again later |
Failures start where `sysexits.h` puts them, so "no" and "broke" are never the
same number. That is what makes this work:
+18
View File
@@ -20,6 +20,7 @@ to any of them is a change to a decision rather than to an accident.
- [pdfgutter — a page tree that does not add up is refused](#pdfgutter--a-page-tree-that-does-not-add-up-is-refused)
- [pdfgutter — a sheet on its side is cut the way it is read](#pdfgutter--a-sheet-on-its-side-is-cut-the-way-it-is-read)
- [pdfgutter — an unreadable file is an answer, not a false](#pdfgutter--an-unreadable-file-is-an-answer-not-a-false)
- [bookmeta — a cover is what its bytes say it is](#bookmeta--a-cover-is-what-its-bytes-say-it-is)
- [Every pattern — two regex engines, two Unicode tables](#every-pattern--two-regex-engines-two-unicode-tables)
## cuenorm — settings after the end time are the format, not a malformation
@@ -243,6 +244,23 @@ answers, and the [contract](contract.md) gives them three different numbers. A
caller wanting them collapsed treats 1 and 2 alike; a caller wanting to know
which files it failed on can now find out.
## bookmeta — a cover is what its bytes say it is
An EPUB's manifest gives every image a media type, and the cover's is not
used: the type comes from the image's first bytes, and a disagreement is a
note. Elsewhere `bookmeta` reports what the file says rather than what is
true, and this is the one place it does not.
The difference is that a title nobody believes is still a title, while a
media type that is wrong is an image that does not open. A caller handed
`image/png` for a JPEG writes `cover.png`, serves it as PNG, or hands it to a
decoder that refuses it. An image can be replaced by a converter or by hand
without its manifest entry changing, and the note keeps what the file said,
so nothing is lost by correcting it.
The same reasoning goes the other way: bytes that are not an image are not a
cover, whatever the manifest calls them.
## Every pattern — two regex engines, two Unicode tables
Patterns here spell their character classes explicitly rather than leaning on
+1 -1
View File
@@ -19,7 +19,7 @@ Part of [catena](../README.md).
| Field | Meaning |
|---|---|
| `path` | Where the file was read from, as given |
| `kind` | `pdf`, `epub`, `djvu`, `azw3`, `mobi`, `txt`, `html`, `ole2`, `zip-wrapped-pdf`, `unknown` |
| `kind` | `pdf`, `epub`, `djvu`, `azw3`, `mobi`, `lit`, `fb2`, `rtf`, `txt`, `html`, `ole2`, `cbz`, `zip-wrapped-pdf`, `zip-wrapped-fb2`, `unknown` |
| `declared_ext` | The extension the name claimed, lowercased, without the dot |
| `bytes` | Size on disk |
| `pages` | Page count, for paginated formats |