The last route took any image whose id or address contained "cover", so
images/discover-ch1.png listed before images/cover.jpg won, and
recovery-map.png alone became the cover. A name now says cover only with
it as a word of its own: words break at anything not a letter or digit
and where a lower case letter meets a capital, so CoverImage and
Book_COVER still do. An image named cover outright comes first, then
manifest order. It is still a guess.
The guide's cover page took its first image, which can be a logo. Where
the page has several, the one whose file name, alt, id or class says
cover is taken; where none does, the first is, with a note saying how
many there were.
Each member was read no further than MAX_COVER, but nothing bounded how
many: two hundred cover-named images of 16 MiB took seven seconds to
inflate and left two hundred notes. One search now reads at most eight
members and stops with one note.
A percent-escape takes two hex digits and nothing else. from_str_radix
also takes a sign, so a%+fb decoded to a, 0x0f, b.
bookmeta: Cover::guessed says the file left the choice open — a Kindle
book's first image record, or an EPUB image picked because its name says
cover — so a caller with a register to ask can prefer that. A cover the
file names is never a guess. Cover::extension matches bibfetch's.
bibfetch: Open Library answers 403 to more than a hundred covers by ISBN
in five minutes from one address. That is now Error::Limited, not
Unreachable: the other form of the ISBN is not asked, the cover id still
is, since those are not limited, and the CLI exits 75. Covers asked by
ISBN are kept three seconds apart across the process (Covers::pace), which
stays under the limit however long a run goes. A deadline too long to add
to the clock no longer panics; it is no deadline.
bindings: fetch_cover and search_title for Python, and catena.Limited,
which fetch_identifier raises too when a register refuses for being asked
too often.
corpus-core: exit 75, a service refused for now.
A book that arrives as a file often has no identifier to ask about, only
a title and perhaps a name. `bibfetch search --title T [--author A]
[--limit N]` asks Open Library's index which books it has under those,
and Google Books only when Open Library has nothing, and prints one
record per candidate in the order the register gave them. From Rust,
`bibfetch::search(&agent, title, author, limit, attempts)`.
Each candidate is one edition. The index answers with works and, asked
for `editions.*`, names the edition of each that best matches; the ISBN,
publisher, language, year and cover come from that edition, the title,
authors, subjects and length from the work. The work's own ISBN list
mixes editions: the first War and Peace work lists four, while the
edition matched is a 1917 printing with none.
`id` is the first ISBN whose check digit agrees, in thirteen digits,
thirteen-digit entries taken first. A candidate with none is named by
the register's key: `olid` (OL…M for the edition, OL…W for the work when
no edition was named) or `googlebooks` (a volume id). Google's `OTHER`
shelfmarks are never read as ISBNs.
`cover_id` keeps the meaning it has in a lookup record: the edition's
own first cover, taken from `editions.cover_i` and only when the record
names that edition. The work's `cover_i` and `cover_edition_key` are not
asked for. A work with no edition named, and every Google Books
candidate, has none.
Open Library having nothing stands when Google Books is down; both down
is exit 69. Exit 0 a candidate, 1 none, 65 an empty title.
The retrying GET that lookups use is now `ask::get`, shared by both, and
no longer sleeps after its last attempt.
The cover is the first image reached by EPUB 3's cover-image property,
EPUB 2's <meta name="cover"> (by id, or by address where a converter wrote
one), the guide's cover page and the first <img> or SVG <image> on it, and
last any manifest image whose id or address says cover. A route that names
something unusable leaves a note and the next is asked.
Addresses are resolved against the document they are written in, with the
fragment off and percent-escapes undone; one that climbs out of the archive
or carries a scheme is not followed. Every member is read through a bound,
checked against the size the archive declares and enforced on the read
itself, so an understated entry stops at the cap too.
The media type is read off the bytes, now WebP as well, and a manifest that
disagrees is a note; bytes that are not an image are no cover. MAX_COVER
(16 MiB) is the cap for both formats, so a Kindle cover record larger than
that is left out as well.
Four formats a library of ebooks holds and this had no name for, so each
came back as text, as HTML or as unknown.
RTF is its opening group, after any byte-order mark and blank space.
Microsoft Reader is its signature at the front. A FictionBook is XML
whose root element is FictionBook, found by stepping over the
declaration, comments and doctype in the first four kilobytes. That walk
is over bytes rather than text, because a great many FictionBooks are
windows-1251 and would otherwise be called binary; other XML is still
HTML, as before.
A zip whose only real member is one FictionBook is zip-wrapped-fb2, the
same way a zipped PDF is zip-wrapped-pdf: handed to a converter as
FictionBook it is still a zip. A zip whose every real member is a JPEG,
PNG, GIF or WebP is cbz, once directories, ComicInfo.xml and desktop
clutter are passed over. One PDF or one readme among the pages and it is
not a comic. Only the names in the archive's directory are read, as for
every other zip.
The kind vocabulary grows by five: lit, fb2, rtf, cbz and
zip-wrapped-fb2.
When a MOBI or AZW3 file names no cover (no EXTH 201 or 202), the cover is
the record at the first image index if it is a JPEG, PNG or GIF, and
nothing otherwise, as calibre does (metadata/mobi.py:524-532). The walk to
the next image is gone: a later record may be an inline illustration, and
kindlegen puts a font or RESC record first.
EXTH 121 is no longer read, which also stops a stale KF8 boundary below
first_image from hiding the cover.
Review id 3 on catena#2.
P0: Record.cover_id came from the search index's cover_i, the cover
chosen for the whole work, and the index is asked first, so cover()
fell back to another edition's image (Penguin War and Peace,
9780140447934, got Standard Ebooks' Maude cover 12621906). cover_i is
no longer asked for or read; cover_id is the first positive id in the
edition record's covers, or None. An ISBN request that fails, as
opposed to a 404, now ends the fetch with that failure instead of
going on to the cover id.
Redirects go only over HTTPS on the standard port to
covers.openlibrary.org, archive.org or a host under archive.org, and
never to an address by number; the covers store the fetch began at is
still allowed.
The 1 KB size floor is gone. With default=false a missing cover is a
404, and the stand-in without it is a 43-byte GIF the media type check
refuses, so the floor could only drop real Small covers.
Covers.deadline (DEADLINE, two minutes) bounds a whole fetch: every
address, attempt, redirect, body and wait. It starts once the fetch
holds the process lock. The timeout test no longer times a span that
includes the lock wait.
EXTH 113 is an ASIN only when it is ten letters and digits; a UUID goes
under uuid, as the EPUB reader files it, and anything else under other.
With no EXTH 201 or 202, the first record from the first image index that
is a JPEG, PNG or GIF is the cover, stopping at the KF8 boundary (EXTH 121).
`bibfetch::cover(&agent, isbn, cover_id, size)` asks Open Library's covers
store by the ISBN's thirteen-digit form, then its ten-digit form, then by a
cover id a lookup record named, and returns the image or `Ok(None)`. The
answer is checked before it is trusted: JPEG or PNG by content type and by
magic bytes, between 1 KB and 10 MB, at most three redirects, none of them
off HTTPS. One request at a time per process; a 429 or 503 is waited out
for its Retry-After, up to a minute.
`idnorm::isbn` converts between the two forms, and only for an ISBN whose
check digit agrees. Lookup records carry `cover_id` from the edition record
and the search index. The CLI gains `bibfetch cover <isbn> -o <file>`.
A Kindle book's title, authors, publisher, description, ISBN, ASIN,
subjects, date and language come from the EXTH block, with the full name,
the Palm database name and the header's locale as fallbacks. The cover
image (EXTH 201, or the thumbnail) is returned as Metadata::cover with its
media type; it stays out of the JSON record. Every offset is bounds-checked:
a damaged header gives an empty record with a note, a broken EXTH block
keeps what came before the break, and an encrypted book is still read and
noted as drm.
Metadata gains description, which the EPUB reader now fills from
dc:description as well.
Fixtures are a story and cover written for the tests (CC0), converted with
calibre to MOBI 6 and AZW3, plus no-EXTH, DRM-flagged and truncated copies.
The commit that reordered the chain did not carry the version bump. The
bump was chained after check.sh with &&, the commit after a semicolon, so
a failing check skipped the bump and let the commit through. The tag then
pointed at a tree still calling itself 0.3.0, which the release workflow
asserts against and would have refused.
check.sh was failing because a test names which register is asked first
about an ISBN, which is exactly what changed. It now names the index, for
the reason the reorder happened.
The chain asked OpenLibrary's edition record first and its search index
second. The edition record is the slow one: measured against this
library it takes five to twenty-eight seconds, and it names its authors
by key, so a second request was needed anyway. The index, asked for its
fields by name, returns the title, the authors, the publisher, the
language, a page count and the subjects together, in about a quarter of
a second.
Reordering them turns three requests per book into one, and a pass over
five thousand books from fifty-three hours into well under one. The
edition record is still there for a subtitle or a publisher the index
did not give.
The chain asked three registers and returned whichever answered first
with a title and an author, discarding the rest. The registers are
complementary rather than redundant: OpenLibrary's edition record
carries the publisher, the pagination and the language and names its
authors by key, while its search index carries the authors and not the
publisher. One book, two half-answers, and the whole of it only if they
are put together.
They are merged now, first answer winning each field, so the order of
the chain is the order of trust. The record says which registers spoke.
It also carries what was being thrown away: subjects, language and a
page count, from whichever register has them. The index will not return
those unless asked for by name, which is how a language it held went
unread. Subjects are capped at twelve — the index lists every heading
every edition of a work has ever been given, which on a well-known book
is hundreds.
An edition record keeps its subtitle in a separate field, so a title
came back without it.
Read out of the review queue a real run produced, where each of these was
sitting as a proposed title waiting for a person to say no:
1. General Remarks a numbered heading
2 James W. McAllister a numbered heading
5222 What Is This a folio that ran into the line
827 North Fourth Street an address
Page xxxvi / Chapter I a marker
title a form label, printed literally
Sidebottom, Harry. Ancient a bibliography entry
To Mackenzie, a dedication
The numbers are the delicate part. A leading number that could be a year
is left alone, because 1984 Revisited is a book and 5222 What Is This is
a folio, and nothing else distinguishes them. A dedication has to end in
a comma, so To Kill a Mockingbird survives.
The copyright notices in the same pile are already fixed: those
proposals were written before 0.2.2 and are stale rather than wrong.
title_case counted small words from the start of the whole string, so
every subtitle that opened with one was set in lower case: 'Aesthetic in
Kant: a Critique', 'Born Fighting: the Scots-Irish'. A colon starts a
title of its own and the first word of one is capitalised however small
it is.
Found by book-pipeline, which had been proposing a title page's capitals
verbatim into a catalogue and needed this to stop.
'(c) Alfredo Ferrarin 2004' was read as a title. The notice pattern
wanted the year to follow the symbol directly, and a copyright line puts
the holder's name in between, which is the ordinary way to set one.
The Python surface gained enough this week that three different builds
were installed on two machines all calling themselves 0.1.0: moria had
read_colophon through merge_colophon, mirkwood had a partial from the
middle of the afternoon, and PyPI had neither. A version that does not
move is a version nobody can depend on, which is how book-pipeline ended
up running a wheel scp'd into place by hand.
New since 0.1.0: colophon --shelf and catena.shelve, which file a book's
printed Library of Congress headings onto a closed list; catena
.read_colophon; catena.fetch_identifier, the registers in process; and
catena.colophon_prompt / merge_colophon, the question about what the
pages did not settle and the reply folded back in.
read_colophon is several milliseconds of computation over a few hundred
kilobytes, and it held the GIL for all of them. A caller reading a
library through asyncio.to_thread therefore got no overlap whatsoever:
six books in flight, one book's worth of work at a time.
Measured on this library, that was most of why a concurrent pass still
paced at four and a half hours.
colophon has composed the question about what its pages did not settle
since it was written, and merged a reply back into the record it read.
Both were reachable only from a terminal, so a pipeline wanting the same
fallback would have had to write its own prompt — a second one, drifting
from the first, asking for a shape merge does not accept.
merge fills only the unsettled fields, so a model cannot overwrite what a
book printed about itself. That is the property that makes asking one
safe, and it belongs to colophon rather than to any caller.
Read against five thousand real books, the title page is often not the
first thing in the file, and what is there instead has shape. Six kinds
of it were being read as titles:
vi CONTENTS a folioed heading
Copyright (c) 2013 by Barry Davies, BEM a notice
Translated from the French by Willard R. Trask a credit
Vol. 11. HENRY JAMES, JR. a collected edition's list
[PART ONE: a part heading
Piper Verlag GmbH, Munich<span style=... a converter's leavings
Rejecting one line only moves the misread down one, because a page that
converted badly converted badly throughout, so the whole shape goes:
markdown links, stylesheet declarations, HTML entities, epub filenames,
and the sentences a copyright page sets.
The notice pattern wants the symbol or the year, never the word alone.
The first draft rejected 'Copyright and Its Discontents', which its own
test caught.
On sixty books this drops five claimed titles, all of them wrong, and
fixes six that were reading one homily's heading for every other
homily's title.
The suite could read a cataloguing block and could ask a register, and
nothing showed the two together — which is the only way either is worth
acting on. A title page alone gives up a running head when the scan
starts late; an ISBN alone returns a confidently wrong book when the
catalogue has been carrying the wrong one for years.
The example's title comparison folds to letters and digits rather than
comparing strings. A register and a title page differ over case, over a
curly apostrophe against a straight one, and over whether the subtitle
counts as part of the title. An example that called those disagreements
would teach the wrong lesson, and did until it was fixed.
It runs on this library and finds both of the cases worth finding: a file
called Document5 by -ducati996 that both sources agree is Russell's
Problems of Philosophy, and a Pelikan whose stored ISBN belongs to a book
about sculpture.
book-pipeline reaches for colophon and bibfetch on every book it repairs.
Shelling out would mean a process per book per tool across five thousand
books, and parsing back the JSON it is about to parse anyway.
bibfetch's fetch loop lived in main.rs, so binding it meant either
lifting it or writing a second one. A second implementation of 'which
register answers about this kind of identifier, and in what order' would
drift from the first, and the copy nobody runs from a terminal would
drift silently. It is now bibfetch::ask, and the command line calls it
too — verified by running the CLI before and after and diffing.
fetch_identifier releases the GIL around the request. It is the only
function here that opens a socket, and holding the interpreter for the
length of an HTTP call would stop every other thread in it.
bibfetch is optional and only the python feature turns it on: its HTTP
client does not build for wasm32, and an unconditional dependency would
have taken the JavaScript package down with it.
A cataloguing block states subjects in Library of Congress form, which
is precise, deep and no use for browsing. Real blocks from this library
read 'Political science – Early works to 1800' and 'English
literature—Middle English, 1100–1500—History and criticism'. Nobody has
a shelf called that.
--shelf takes the closed list the library is actually browsed by and
files the printed headings onto it, longest term first so a period of
thought does not collapse into its parent. Matching is on whole words
after folding punctuation away, because a cataloguing block separates
with em dashes, en dashes, full stops and plain hyphens, often three in
one heading, and OCR turns any of those into any other.
What matches nothing comes back under unshelved rather than being
dropped: a subject silently discarded is indistinguishable from a book
that printed none, and the difference is the whole value of reading the
block.
An ISBN is not a DOI. The body that assigns a DOI is the only one that has
it, so one register is the whole answer. A book's identifier is assigned by
its publisher and recorded by whoever gets round to it, and the record that
comes back is as often as not missing the thing the caller wanted: asked
about 9780199205592, OpenLibrary's edition record names *The First World
War* and the press that printed it, and nobody at all as its author, because
an edition record names its authors by key.
So an ISBN now has a chain rather than a register. The edition record first,
then OpenLibrary's own index — which answers with `author_name` filled in
and costs nothing — and Google Books last, because it answers anonymous
callers only until a daily quota runs out, after which it replies 429 to
everyone behind the same address, as it is doing here today.
The first complete answer wins, an incomplete one is kept in case nothing
better comes, and a register being down is a failure only when every
register is. `source` on the record says which one answered.
colophon comes with it: the imprint reader was taking `Oxford is a
registered trade mark of Oxford University Press` for a publisher and
`Oxford N`, off `Oxford New York`, for a place. A legal notice mentioning a
press is not the press, and a city is the city and not the first letter of
the next one.
A catalogue's worst records are the ones whose file says nothing true about
itself: a PDF titled `Untitled` by an author called `Oxford`, which is a
publisher, from a filename that was never anything but a filename. What
those books do have is pages. The title page carries the title as the
publisher set it; the copyright page carries the imprint, the year, the
ISBN, and in most English books since the 1970s the cataloguing block, which
is a catalogue record a librarian already made, subject headings and all.
So this reads the pages. Every field names the page it came from, and a book
that printed nothing comes back empty with its fields in `unsettled` rather
than filled with something plausible — an extraction that starts at chapter
one has no front matter in it, and saying so is the answer.
Two pages look like a title page and are not, and both are how a catalogue
comes to name the wrong book. A series advertisement lists other titles in
the shape a title page uses, over as many as four pages with only the first
announcing itself. A translation's cataloguing block prints the original
title in brackets above the real one. Both are found and refused.
Three things the pages give that no file does: the subject headings a
librarian assigned, an ISBN that exists only in print — which turns a book
with no identifier into one `bibfetch` can look up — and a title in the case
the publisher meant, because a half title prints unbroken what the title
page breaks across two lines and sets in capitals.
`--prompt` and `--merge` are citefind's: a question about exactly the fields
the pages left unsettled, composed here and answered wherever the caller
keeps a model, with the advertising taken out of what the model is shown.
A field a model answered is marked `model`, so a record always says which of
it was read and which was inferred.
Measured against 600 books of the home corpus: the cataloguing block
answered for 21% of them and agreed with the catalogue 89% of the time, most
of the residue being the catalogue's error rather than this one's. Title
pages answered for another 41%. The remaining 38% print nothing readable,
and are what the prompt is for.
The registry answers POST /-/stage/package with a 404 for a name it has not
seen, so the first release of a new name cannot be staged at all — no token
scope changes that. The step now says so and lets the release stand rather
than failing after PyPI has already published.
The first version goes out by hand with 2FA; every one after it stages from
here.
wasm-pack downloads binaryen to get wasm-opt, and that download fails on this
runner while the same URL fetched with curl succeeds. wasm-pack uses a
wasm-opt already on PATH when it finds one, so the archive is fetched here
and put there.
Turning wasm-opt off instead would have been the smaller change and the
wrong one: the metadata in crates/bindings/Cargo.toml names the WebAssembly
features to keep the optimisation rather than skip it, and a release that
silently shipped an unoptimised module would be the first thing anyone
downloaded.
Publishing by hand needs a laptop with two credentials on it, and gets the
gate only if whoever ran it remembered. This runs the same `check.sh` every
push runs, then builds and uploads — and refuses at the first step if the tag
and the workspace version disagree, because a release tagged v0.3.0 that
publishes 0.2.0 cannot be undone: PyPI will not reuse a version even after it
is yanked.
Wheels are cross-compiled with zig for x86_64 and aarch64 against an older
glibc than the build image ships, so they are not silently pinned to
bookworm's, and an sdist covers everything else.
PyPI is uploaded directly. npm is only staged: npm is withdrawing direct
publish from tokens that bypass 2FA, and its trusted publishing is OIDC
against GitHub and GitLab, which this forge is neither. So the runner does
the part a token may do, and a maintainer approves the tarball with 2FA.
A missing NPM_TOKEN is a warning rather than a failure, so a release can go
to PyPI while the npm credential is still being sorted out.
Both registries already have a `catena`: an HPE blockchain client on PyPI,
and a JavaScript OOP wrapper at 2.2.0 on npm. Neither is going anywhere, so
the distributions take a name that is free on both.
The Python import is unaffected — a distribution name and a module name are
independent there, so `pip install catena-corpus` still gives `import
catena`, and nothing in the examples or the bindings changes. On npm the
package name is the import specifier, so that one does change: `import
{ chunkMarkdown } from "catena-corpus"`.
The crates keep their own names. `catena` is free on crates.io.
Eight of them, each doing something a pipeline actually has to do, and each
starting from something anyone can fetch — a public domain book from Project
Gutenberg, a NASA broadcast, or a sample carried in the file.
Shell, because the tools are filters first: a book to passages and what its
notes cite; a recording to chapter marks, citable passages and the one
ffmpeg call that makes it play on a phone; passages into pgvector and a
search that asks twice and reconciles.
Rust, for the case where the caller already holds the text and a subprocess
would be serialising for nothing. Python, for the shape a pipeline stage
takes once the reading is a library call. TypeScript, for the app side —
chapter marks and deep links, and fusing two rankings in process.
Three of them end with what re-ranking means, because it is two different
things under one name: fusing ranked lists, which is what `rrf` does and
needs no model, and a cross-encoder scoring query and passage together,
which is accurate, slow, and runs over what retrieval already found.
Two things between a converted EPUB and its chapter titles.
The heading level taken as the chapter map was the shallowest present. Pandoc
writes the book's title as the only `#` and every real chapter as a `##`, so
the shallowest level had one heading in it and the whole book came back as a
single chapter. The dividing level is now the shallowest one used more than
once — a level used once is a title.
And an empty bracketed span survived the heading cleaner, because the pattern
wanted at least one character between the brackets. Pandoc writes an anchor
as exactly that: `## []{#link2H_INTR} Introduction`.
Gibbon's first volume, converted from the Project Gutenberg EPUB: 2 distinct
chapter titles before, 61 after, and none of them carrying an element id.
The chapter a passage carries was resolved at the offset of the page it came
from, not the offset of the passage. A document with no page markers is a
single page beginning at zero, so every chunk in such a book answered with
whichever heading came first in the file — and took the role classification
with it, since that reads the chapter title.
Measured over a corpus of 5,386 books: all 1,232 unpaginated books with
twenty or more chunks carried exactly one chapter title from cover to cover,
as did 73% of the paginated ones. A bibliography chapter was therefore
invisible to anything selecting on the chapter title, which is the first of
the three ways the citation stage finds one.
The packer now hands back each passage with the offset it began at, and the
page's own offset is the true start of its text rather than the position of
the marker before it.
The harness compared these tools against the code they replaced — Python
imported from book-pipeline, podcast-pipeline and knowledge-mcp, and
JavaScript copied out of the podcast media server and indexer, each pinned to
a commit. All of it has since been deleted, so the live half cannot run and
the golden half pins answers against a source nobody can consult.
What was worth keeping was never the comparison. `KNOWN.md` argued, at
length, why a dozen of these tools deliberately do something a reader would
not expect: why pdfgutter measures the crop box throughout, why avplan finds
the encoder rather than reading one out of the environment, why mdscrub will
not manufacture a page boundary. Those arguments are about this code and
stand on their own, so they are now `docs/decisions.md`, rewritten to say
what each tool does and why rather than what it does differently.
Binding coverage is unaffected: the bindings carry 24 tests of their own, a
node smoke test against the built package, and a typecheck.
An entry opening was a surname at the start of a line, and a converted
bibliography is often a bulleted list — so the bullet sat exactly where the
surname was looked for and not one entry was seen:
• Skorupski, J. 1995. 'Agent-Neutrality, Consequentialism,
Utilitarianism', Utilitas, 7: 49-54.
The whole page went unflagged, no chunk of it was ever selected, and books
whose only bibliography arrived this way came back with no citations at all.
A bullet may now stand where the surname would.
Measured against the corpus: of 300 sampled body chunks that open lines on a
bullet and an inverted name, 0 were flagged before and 223 after. Across
3,000 randomly sampled unflagged body chunks, the change fires on none — it
buys the bulleted list and nothing else.
The bar a guessed title has to clear was waived when the entry also gave up
a container, on the reasoning that a container vouches for the rest of the
reading. Auditing the corpus rather than the reference chapters showed that
is wrong, and wrong on the commonest note there is:
As Lewis White Beck has rightly put it. See his Early German
Philosophy (Cambridge, Mass., 1969), p. 439.
The remark opens, the citation follows, and the second sentence becomes the
container — so the first was stored as the work and the book named in it was
lost. Neither a siglum nor a remark is the name of a work now, whatever else
came off the entry.
Measured where these notes actually live, which is flagged body chunks
rather than bibliography chapters: on the corpus the class ran at 1.43% of
rows naming a work. The cost is 46 rows in 34,359 on the reference holdout,
where the share naming an author rises from 88.1% to 88.2%.
The stop after `P.` is an initial rather than a sentence stop, so `Hoyt,
Edwin P. The Supersalesmen. New York: World Publishing, 1962.` never got cut
there. The name came back as `Hoyt, Edwin P. The Supersalesmen` and the next
segment along — the imprint — became the work. Both fields wrong from one
missing cut.
An inverted name whose forenames end in an initial, with two or more words
after it, now ends at that initial. Two words, because one is a forename:
`Price, M. Philips` and `Drexler, K. Eric` are whole names and must not be
cut.
And the segment the reader settles on for a title may not be an imprint —
which is what the function has said it did since it was written, in a comment
above a predicate that never checked. Asked narrowly enough that a title with
a colon in it survives.
On 199 reference chapters never used for tuning, imprints stored as the work
fall from 376 to 55, and the share of rows naming an author rises from 86.1%
to 88.1%.
Three refusals named on the page: a note's own remark, a siglum, and the
navigation strip a PDF reader leaves behind. Says where the line is drawn —
at the guess rather than at the title, so an entry that marked its own title
is taken at its word.
A title read off markup was vouched for by the entry. A title guessed at by
cutting the entry at its sentence stops was not, and where nothing else came
off the entry — no author, no identifier, no container — that guess was the
whole of what got stored. Two things kept arriving that way.
A note that remarks on the argument rather than citing anything: "The
Commentator seems to have based himself principally on the Bible", "Dewey
remarks that we experience most things as temporal". A page of endnotes is
mostly these. They are noun phrases nowhere: what separates them from
"Quaestiones Parisienses" is a finite verb, so that is what is asked.
A siglum and a page — `H, 272`, `FR,120-122`, `GNO` — whose key is a page of
its own the entry does not carry. An atlas of Middle-earth is a thousand of
them, and the heading each sits under is not its title, so a siglum is
refused whatever else came off the entry.
Also refuses the navigation strip a PDF reader leaves in the text, `572 next
page > < previous page`, and the local path it points at.
On 400 flagged body chunks the junk rate falls from 3.5% to 0.5% of stored
rows; on the 202-file reference sample the share naming a work holds at
97.2% across 42 fewer rows, all of them sigla, scripture or prose.
Three more shapes in the table — a reading list with nothing marking it, an
editor in the author's place, a reprint's two years — and three more refusals
named: a keyed short cite, a translator's signed note, and a page of
references whose lines happen to begin on numbers.
Also says what the cut does with a blank line now: it is often after the
running head rather than between the entries under it, so every block it
produces is looked at again.
Kymlicka, Will, Politics in the Vernacular: Nationalism,
Multiculturalism and Citizenship, Oxford, 2001
The imprint is found by counting commas, and the last comma-part carrying a
colon is taken to be it — which is right when the colon is `Place:
Publisher` and wrong when it is the title's own. Here it is the only colon in
the entry, and looking for it before the names have been read meant giving up
rather than falling through to the year at the end. The search starts after
the names now, so a colon that is part of the title is not mistaken for one.
Juvenal, Saturae XIV, ed. J. D. Duff (Cambridge, 1962), p. 9.
A title that ran on into an editor's credit was refused outright. Trimming
the credit off is the repair that was wanted: the credit belongs to the
entry, not to the work's name, and the work is `Saturae XIV`.
Mate selection: Buss, 1989; Symons, 1979. Jealousy: Buss, 2000.
A notes section written this way keys the author's own words to works given
in full in the bibliography. `Mate selection` is not a name and `Buss, 1989`
is not a work, so the note points at something it does not name — the same
thing `ibid.` does, and it is flagged the same way. It was being stored as a
reference whose author was the topic, and the entry below it inherited that
as a repeated author.
The pattern is deliberately narrow: the part after the colon must be nothing
but surnames and years. `Miller, David, On Nationality, Oxford: Clarendon
Press, 1995` has a colon and a year too, and is a book.
And a run the style itself marked as a title is a title whatever it opens on.
The rule against a title beginning with a number is about a page reference
picked up by mistake; inside italics there is no mistake to make, and
`*1 & 2 Thessalonians*` is two books of the Bible.
funkpower