# Vector storage and data flow

This note describes where article content and vector representations are
stored, which artifacts are authoritative, and which vectors are deployed to
Cloudflare.

## Source article corpus

The source articles are JSON files under:

```text
bmj-content-pipeline-integration/exported-content/
```

Articles are grouped into directories by journal identifier. The export also
contains `journals.json`, which supplies journal names and identifiers used
when building the Cloudflare journal index.

These article JSON files contain source metadata and text. They do not contain
the vector matrix used by the picker.

## Local vector-service storage

The local vector service stores its queryable vectors in:

```text
vector-service/data/embeddings.sqlite3
```

The `document_vectors` table currently contains:

| Model | Profile | Vectors |
|---|---|---:|
| `sentence-transformers/allenai-specter` | `title_abstract` | 32,814 |
| `sentence-transformers/all-MiniLM-L6-v2` | `title` | 636 |
| `sentence-transformers/all-MiniLM-L6-v2` | `title_abstract` | 623 |
| `sentence-transformers/all-MiniLM-L6-v2` | `full_metadata` | 623 |

The full SPECTER snapshot used to populate or reproduce that local store is:

```text
vector-service/data/vectors-specter/
├── title_abstract.sentence-transformers-allenai-specter.f32
└── title_abstract.sentence-transformers-allenai-specter.json
```

SPECTER is the full-corpus embedding used by the local vector service. It is
not compatible with the BGE embedding space used by Workers AI, so SPECTER and
BGE vectors must never be compared directly.

## Cloudflare full-corpus vectors

The full BGE article snapshot for the Cloudflare picker is stored in:

```text
cf_journal_picker_deploy/data/vectors-bge/
├── title_abstract.cf-baai-bge-small-en-v1-5.f32
└── title_abstract.cf-baai-bge-small-en-v1-5.json
```

This snapshot contains 32,814 normalized vectors with 384 dimensions, produced
with the same BGE checkpoint exposed by Workers AI as:

```text
@cf/baai/bge-small-en-v1.5
```

The binary matrix is approximately 48 MiB. It is an offline build and
evaluation artifact and is not shipped as a Cloudflare static asset.

## Binary matrix and sidecar format

Every snapshot consists of a matching `.f32` and `.json` pair:

- The `.f32` file contains a little-endian, row-major array of float32 values.
- The `.json` sidecar contains the model ID, dimensions, vector count,
  normalization flag, and the ordered `ids` array.

For a snapshot with `D` dimensions:

```text
vector for ids[N] = floats[N * D : (N + 1) * D]
```

The sidecar order is therefore part of the data contract. Reordering the IDs
without rebuilding the matrix would associate vectors with the wrong
articles.

## Cloudflare deployed journal index

The Cloudflare picker does not compare a submitted manuscript with all 32,814
articles during a request. The full BGE matrix is aggregated offline into one
normalized centroid per journal:

```text
cf_journal_picker_deploy/workers/picker-api/assets/journals/
├── title_abstract.cf-baai-bge-small-en-v1-5.centroids.f32
└── title_abstract.cf-baai-bge-small-en-v1-5.centroids.json
```

The deployed index contains 74 journal centroids and is approximately 111 KiB.
The sidecar includes each journal's ID, slug, display name, ISSN-L, and source
article count.

At request time:

1. Workers AI embeds the submitted title or title and abstract.
2. `picker-api` loads the 74-row centroid matrix.
3. It computes cosine similarity against the journal centroids.
4. It returns ranked, named journals.

This keeps request cost bounded by 74 comparisons and avoids loading or
reprocessing the full article corpus in a live request.

The centroid index is built with:

```sh
python3 cf_journal_picker_deploy/scripts/build_journal_centroids.py
```

The build validates that the ordered IDs in the BGE sidecar exactly match the
articles loaded from `bmj-content-pipeline-integration/exported-content/`
before assigning vectors to journals.

## Diagnostic and legacy snapshots

The following directories are retained for diagnostics and comparison:

```text
cf_journal_picker_deploy/data/vectors/
```

Contains the original small MiniLM snapshots.

```text
cf_journal_picker_deploy/data/vectors-bge-640doc-backup/
```

Contains the preserved BGE version of the approximately 640-article diagnostic
corpus.

```text
cf_journal_picker_deploy/workers/picker-api/assets/vectors/
```

Contains deployable copies of those small diagnostic spaces. They support the
lower-level `/api/v1/spaces` and `/api/v1/similar` endpoints, but they are not
the product journal-recommendation index.

## Rebuild flow

The intended full-corpus rebuild is:

```text
Article JSON export
    ↓
load title + abstract and preserve article IDs
    ↓
embed articles into the BGE space
    ↓
write full .f32 matrix and ordered .json sidecar
    ↓
average article vectors by journal
    ↓
normalize and write 74 journal centroids
    ↓
deploy picker-api
```

Relevant scripts:

| Script | Responsibility |
|---|---|
| `scripts/corpus_loader.py` | Loads usable title/abstract articles from the export |
| `scripts/reembed_bge_from_export.py` | Builds the full 32,814-article BGE snapshot |
| `scripts/build_journal_centroids.py` | Builds the 74-journal deployable index |
| `scripts/build_worker_assets.py` | Builds the small diagnostic Worker spaces while enforcing the static-asset size limit |
| `scripts/validate_hackathon_assets.py` | Validates counts, dimensions, normalization, asset sizes, and configuration |

## Operational rules

1. Treat article JSON as source content, not as the vector store.
2. Treat each `.f32` and `.json` pair as an inseparable snapshot.
3. Never compare vectors produced by different models.
4. Never compare vectors created from different input profiles without an
   explicit, tested design for doing so.
5. Do not deploy the 48 MiB full article matrix as one Cloudflare static asset.
6. Rebuild centroids after changing the full BGE snapshot.
7. Run `./scripts/hackathon_preflight.sh` before deploying the Cloudflare
   picker.