mirror of
https://git.pumpin.online/funkpower/catena.git
synced 2026-11-13 23:55:40 +00:00
funkpower
5c4f770d76
Four formats a library of ebooks holds and this had no name for, so each came back as text, as HTML or as unknown. RTF is its opening group, after any byte-order mark and blank space. Microsoft Reader is its signature at the front. A FictionBook is XML whose root element is FictionBook, found by stepping over the declaration, comments and doctype in the first four kilobytes. That walk is over bytes rather than text, because a great many FictionBooks are windows-1251 and would otherwise be called binary; other XML is still HTML, as before. A zip whose only real member is one FictionBook is zip-wrapped-fb2, the same way a zipped PDF is zip-wrapped-pdf: handed to a converter as FictionBook it is still a zip. A zip whose every real member is a JPEG, PNG, GIF or WebP is cbz, once directories, ComicInfo.xml and desktop clutter are passed over. One PDF or one readme among the pages and it is not a comic. Only the names in the archive's directory are read, as for every other zip. The kind vocabulary grows by five: lit, fb2, rtf, cbz and zip-wrapped-fb2.
140 lines
5.1 KiB
Markdown
140 lines
5.1 KiB
Markdown
# The records
|
|
|
|
Five shapes travel between these tools. Each carries a version field `v`, so a
|
|
reader can refuse a stream it does not understand rather than parsing it into
|
|
whatever happens to line up.
|
|
|
|
Every list-shaped stream is [NDJSON](contract.md#anything-list-shaped-is-ndjson): one
|
|
record per line, flushed as it is produced.
|
|
|
|
Part of [catena](../README.md).
|
|
|
|
## `doc` — what a file is
|
|
|
|
```json
|
|
{"v":1,"path":"scan.pdf","kind":"pdf","declared_ext":"pdf","bytes":9261184,
|
|
"pages":412,"language":"eng","text_layer":"garbled","needs_ocr":true,"notes":[]}
|
|
```
|
|
|
|
| Field | Meaning |
|
|
|---|---|
|
|
| `path` | Where the file was read from, as given |
|
|
| `kind` | `pdf`, `epub`, `djvu`, `azw3`, `mobi`, `lit`, `fb2`, `rtf`, `txt`, `html`, `ole2`, `cbz`, `zip-wrapped-pdf`, `zip-wrapped-fb2`, `unknown` |
|
|
| `declared_ext` | The extension the name claimed, lowercased, without the dot |
|
|
| `bytes` | Size on disk |
|
|
| `pages` | Page count, for paginated formats |
|
|
| `language` | Language code, when one was detected |
|
|
| `text_layer` | `healthy`, `garbled`, `sparse`, `reflected-font`, `absent` |
|
|
| `needs_ocr` | Whether the file has to be OCRed to be searchable |
|
|
| `notes` | Observations, in the order they were made |
|
|
|
|
A tool fills what it can and leaves the rest null: a format sniffer fills
|
|
`kind` and leaves `text_layer` empty, a triage tool does the reverse.
|
|
|
|
Written by [`booksniff`](../crates/booksniff/README.md) and [`pdftriage`](../crates/pdftriage/README.md).
|
|
|
|
## `chunk` — a retrievable passage
|
|
|
|
```json
|
|
{"v":1,"idx":7,"text":"…","tokens":512,"page":37,"chapter":"III. The Second Estate",
|
|
"role":"body","start_time":null,"end_time":null,"flags":[]}
|
|
```
|
|
|
|
| Field | Meaning |
|
|
|---|---|
|
|
| `idx` | Position in the stream this chunk was cut from, from zero |
|
|
| `text` | The passage |
|
|
| `tokens` | Token count of `text` |
|
|
| `page` | Page the passage starts on, for paginated sources |
|
|
| `chapter` | Nearest enclosing heading, as written |
|
|
| `role` | `body`, `bibliography`, `index`, `toc`, `frontmatter` |
|
|
| `start_time`, `end_time` | Seconds from the start, for recordings |
|
|
| `flags` | Set tags, sorted and deduplicated |
|
|
|
|
`page` and `chapter` for documents, `start_time` and `end_time` for
|
|
recordings. One shape either way, which is what lets a single index serve a
|
|
book and a lecture and cite both.
|
|
|
|
Written by [`chunkmd`](../crates/chunkmd/README.md) and [`cuepack`](../crates/cuepack/README.md).
|
|
|
|
## `hit` — one scored candidate
|
|
|
|
```json
|
|
{"v":1,"id":"c81a","channel":"vec","rank":1,"score":0.83,"payload":{}}
|
|
```
|
|
|
|
| Field | Meaning |
|
|
|---|---|
|
|
| `id` | Identifier of the thing retrieved, opaque to fusion |
|
|
| `channel` | `vec`, `fts` or `meta` |
|
|
| `rank` | Position within that channel's own ranking, from one |
|
|
| `score` | The channel's raw score, on whatever scale it uses |
|
|
| `payload` | Anything the producer wants carried through untouched |
|
|
|
|
`rank` is what fusion consumes. `score` is carried along because it is useful
|
|
to a person reading a trace and useless for comparing across channels.
|
|
|
|
Read and written by [`rrf`](../crates/rrf/README.md).
|
|
|
|
## `cue` — one line of a transcript
|
|
|
|
```json
|
|
{"v":1,"start":12.5,"end":15.0,"text":"the mill stood at the northern end"}
|
|
```
|
|
|
|
Smaller than a chunk on purpose: a cue is what a subtitle file holds, a few
|
|
seconds and often half a sentence.
|
|
|
|
Written by [`cuenorm`](../crates/cuenorm/README.md), read by [`cuepack`](../crates/cuepack/README.md) and
|
|
[`cueseg`](../crates/cueseg/README.md).
|
|
|
|
## `segment` — one topic chapter
|
|
|
|
```json
|
|
{"v":1,"start":0.0,"end":1190.7,"title":"The Mill Stood At The Northern End",
|
|
"excerpt":"the mill stood at the northern end of the valley where…"}
|
|
```
|
|
|
|
The marks a scrubber needs, and the spans a summariser should be given one at
|
|
a time.
|
|
|
|
Written by [`cueseg`](../crates/cueseg/README.md).
|
|
|
|
## Records that belong to one tool
|
|
|
|
Not everything is shared. A tool whose answer nothing downstream consumes
|
|
has a shape of its own, described where that tool is:
|
|
|
|
- [`bibclean`](../crates/bibclean/README.md) — catalogue records, author
|
|
suggestions, slug proposals and vocabulary merge proposals.
|
|
- [`avplan`](../crates/avplan/README.md) — what has to happen to a recording,
|
|
and the call that would do it.
|
|
- [`pdfgutter`](../crates/pdfgutter/README.md) — whether a scan is two pages
|
|
to a sheet, and what each sampled page measured.
|
|
- [`citefind`](../crates/citefind/README.md) — one reference out of a
|
|
bibliography, and the fields it could not read off the entry.
|
|
- [`bookmeta`](../crates/bookmeta/README.md) — what a file says about itself.
|
|
- [`tocfind`](../crates/tocfind/README.md) — one entry of a table of contents.
|
|
- [`seriesof`](../crates/seriesof/README.md) — what a title says about its
|
|
series.
|
|
- [`idnorm`](../crates/idnorm/README.md) — an identifier as it arrived and as
|
|
it should be stored.
|
|
|
|
## Getting the schemas
|
|
|
|
The JSON Schema for the first three is printed by the record library itself:
|
|
|
|
```
|
|
$ cargo run -p corpus-core -- schema chunk
|
|
{
|
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
"title": "Chunk",
|
|
…
|
|
}
|
|
```
|
|
|
|
See [`corpus-core`](../crates/corpus-core/README.md).
|
|
|
|
See also [the recipes](recipes.md), for what these rules buy when the
|
|
tools are put in a row.
|