Craftpine

Preprint, not peer reviewed · Version 1 · October 6, 2026

A few vectors for a mixed folder: a pre-registered measurement of pooled directory embeddings

Ali Basheer, Craftpine Inc., Toronto. ali@basheerali.com

Author
Ali Basheer, Craftpine Inc. · ORCID 0009-0006-2396-9932
Published
on Zenodo, version 1
DOI
10.5281/zenodo.23178122
Licence
CC BY 4.0; code under MIT

DOIZenodo recordCite

Abstract

A folder's metadata is read before its files and holds a few kilobytes. A vector stored there can route a query without opening the folder, and the cheapest is the mean of its files' embeddings. We ask when the mean is enough and what to store when it is not. Under our main vision-language embedder, the mean does at least as well as a few representatives on GitHub directories that do not mix media, about 95 percent of those eligible. Representatives for mixed folders only are level with the mean over all directories (both post hoc). Where a folder mixes images and texts, the mean loses its minority modality. On Zenodo record folders a held-out minority file finds its folder 0.16 to 0.25 recall@5 less often with the mean than with a few per-modality representatives under two vision-language embedders, and 0.35 to 0.50 less often under two CLIP-family dual towers. There, four representatives per label are not inferior to searching every file, at a 0.02 margin, over all queries and in every cell under all four encoders (post hoc). On queries that a language model wrote as a simulated searcher, the loss under the main embedder is 0.14 (interval 0.02 to 0.28) for pictures in text-heavy folders and 0.42 (0.28 to 0.56) for documents in image-heavy ones. At three per label every folder's representatives fit in 8 KiB at a further cost of at most 0.012 in any cell. In 2 KiB a per-kind sample of the folder's files keeps the minority files the mean loses, and on ext4 a summary of 2,048 or 4,000 bytes costs 1.2 blocks per cold directory read. With every creator weighted equally the main embedder's file-query loss is 0.13 and 0.14, and 0.06 and 0.12 on queries whose names do not give the folder away. Every hypothesis was fixed in a brief before its test. Five briefs are timestamped externally, the rest dated by the working repository's history. Code, manifests and briefs are released.

Keywordsdirectory embeddings · pooled embeddings · modality gap · mixed-modality retrieval · vision-language embeddings · collection representation · semantic file systems · information retrieval · pre-registration

The question

A file system reaches a folder before it reaches the folder’s files. The folder’s own metadata can be read without opening anything inside it, but it holds little. On ext4 a file’s extended attributes, the named values attached to a file or a directory, normally have to fit in its inode and one block of at most 4,096 bytes.

The cheapest vector to keep there is the mean of the folder’s file embeddings. It is the dense counterpart of the single bag of words by which early federated search summarised a collection, and one router for retrieval-augmented generation already takes each source’s centroid as an input.

Folders mix media, and in contrastive models image and text embeddings occupy offset regions of one space. The study asks when the mean is enough, what it loses in a mixed folder, and what a folder’s few kilobytes can keep instead.

Four panels: a folder of six images and three documents; one vector per file, with images and texts in two regions separated by a gap; the pooled mean between the regions, far from a text query; and a few vectors per modality, one of them next to the query.
The pooled representation (a) keeps the mean of the folder’s files, which sits between the images and the texts and far from a text query. The per-modality representation (c) keeps a few k-means vectors for each modality, and one of them sits next to the query. A folder’s score for a query is the best cosine between the query and the folder’s vectors. A schematic, not data. Source: Figure 1 of the paper.

Data and models

The Zenodo folders are public records of 3 to 60 files that hold both an image type and a document type, under CC0 or CC-BY, from 2017 to 2026. Their images are mostly scientific figures.

The GitHub folders are directories of public repositories with at least five stars and a permissive licence. Their tests take the repository’s owner as the independent unit. One of the study’s 27 sessions, session 23, also walks 261 whole repository trees.

DrawSourceFoldersFilesRole
S1Zenodo4004,773Pilot
S3Zenodo4004,891Pre-registered retest
S5Zenodo2,80827,545Main set
S11Zenodo2,59725,990Confirmation, four encoders
S19GitHub1,60012,656First draw
S24GitHub2,60021,854Second draw
Six draws of single folders, four of Zenodo records and two of GitHub directories. Source: Table 1 of the paper, which also gives the queries and image-fraction strata of each.

Four encoders from three vendors read the same files. Two are vision-language embedders, jina-embeddings-v4 (E1, the main one, 2,048 dimensions) and gme-Qwen2-VL-2B-Instruct (E3, 1,536). The other two are CLIP-family dual towers, jina-clip-v2 (E2, 1,024) and Nomic Embed v1.5 (E4, 768).

Images are read as images. Text, tables, text PDFs and office documents give their first 2,000 tokens of extracted text.

DimensionsRecall@5 of c, S11, all queriesVision-language embeddersE1jina-embeddings-v4the main encoderE1: 2,048 dimensions2,048E1: recall@5 0.7960.796E3gme-Qwen2-VL-2B-InstructE3: 1,536 dimensions1,536E3: recall@5 0.8110.811CLIP-family dual towersE2jina-clip-v2E2: 1,024 dimensions1,024E2: recall@5 0.6830.683E4Nomic Embed v1.5E4: 768 dimensions768E4: recall@5 0.6930.693DimensionsRecall@5 of c, S11Vision-language embeddersE1jina-embeddings-v4, the main one2,0480.796E3gme-Qwen2-VL-2B-Instruct1,5360.811CLIP-family dual towersE2jina-clip-v21,0240.683E4Nomic Embed v1.57680.693
The two dual towers have fewer dimensions and are the weaker encoders on these files, by recall@5 of c (section 04 defines both) over all evaluation queries of the confirmation set S11. E1 and E2 come from one vendor, E3 from Alibaba and E4 from Nomic AI. Source: Section 3 and Table 3 of the paper.

Experimental design

The task is directory retrieval. A held-out file is a simulated known-item query, and its own folder is the one relevant item. Recall@5 is the fraction of queries whose own folder ranks in the top five of all folders in the set. It is reported with 95 percent paired bootstrap intervals over directories, and the GitHub tests resample owners.

Each folder is represented from its remaining files, with the query file left out, and scored by the maximum cosine over its vectors. The pooled centroid (a) is the mean of the file vectors. Per-label centroids (b) keep one mean per modality label. Per-modality representatives (c) run k-means within each label, up to three vectors per label. The flat walk (d) keeps every file vector and is the reference.

aPooledcentroid1 vector, the meanof every filebPer-labelcentroids1 mean per label,up to 6cPer-modalityrepresentativesup to 3 per label,5.20 per folder on S11dFlat walk,the referenceevery file,9.87 per folder on S11aPooledcentroid1 vector, the meanof every filebPer-labelcentroids1 mean per label,up to 6cPer-modalityrepresentativesup to 3 per label,5.20 per folder on S11dFlat walk,the referenceevery file,9.87 per folder on S11
What one folder of six images and three documents keeps under each representation, with the average count per folder on S11 for c and d. A schematic, with counts from Table 3 of the paper.

Queries are read in cells by modality and by the image fraction of their folder, so that images in text-heavy folders and texts in image-heavy ones are measured apart. Later sessions add record titles, commit subjects and queries written by a simulated searcher.

Text-heavy folderimage fraction below 0.5Image-heavy folderimage fraction 0.5 or moreImage queryan image fileText-like querytext, table, PDF text, officeP1minority cell1,959queries on S11M1majority cell5,571queries on S11M2majority cell7,272queries on S11P2minority cell2,146queries on S11Text-heavyimage fraction below 0.5Image-heavy0.5 or moreImage queryText-like queryP11,959minority cellM15,571majority cellM27,272majority cellP22,146minority cell
The two minority cells hold image queries in text-heavy folders (P1) and text-like queries in image-heavy folders (P2). Counts are the S11 evaluation queries, the same under every encoder. Source: Table 3 of the paper.

Findings

The paper asks seven questions. Its answers follow, with differences in recall@5. The paper’s appendix lists every pre-registered hypothesis with its verdict.

  1. When is the mean enough?

    On GitHub directories that do not mix media, about 95 percent of those eligible, the mean does at least as well as a few representatives under E1 (post hoc).

  2. What does the mean lose in a mixed folder?

    Its minority modality. On Zenodo folders the loss is 0.16 to 0.25 under the two vision-language embedders and 0.35 to 0.50 under the two dual towers. On GitHub directories it is about 0.2 under E1, for files with a sibling of their kind.

    P1 image queries, text-heavy foldersP2 text queries, image-heavy foldersM1, M2 majority cells0.16 to 0.25Vision-language embedders0.35 to 0.50CLIP-family dual towers0.00.10.20.30.40.50.6S3 retest, E1S5 main set, E1S11, E1 jina-v4S11, E3 GMES11, E2 jina-clip-v2S11, E4 Nomicrecall@5 lost by the pooled vector (c minus a), with 95 percent intervalsP1 image queries, text-heavy foldersP2 text queries, image-heavy foldersM1, M2 majority cells0.00.10.20.30.40.50.6Vision-language embedders0.16 to 0.25CLIP-family dual towers0.35 to 0.50S3 retest, E1S5 main set, E1S11, E1 jina-v4S11, E3 GMES11, E2 jina-clip-v2S11, E4 Nomicrecall@5 lost by the pooled vector (c minus a)
    The pooled vector loses most in the minority cells, P1 and P2, and less in the majority cells. Points are recall@5 of c minus a with paired 95 percent intervals, and the shaded bands span the ranges quoted above. Source: Table 2 and Figure 3 of the paper.
  3. Does the loss reach a simulated searcher’s queries?

    Under E1 the loss is 0.14 for pictures in text-heavy folders and 0.42 for documents in image-heavy ones, on queries a language model wrote as a simulated searcher.

  4. Why?

    A mixed folder is two clusters, its images and its texts, and the mean sits between them. Per-modality mean-centering recovers 8 to 62 percent of the loss, depending on the encoder and the cell.

    image with imagetext with textimage with textBefore centering0.00.20.40.60.81.0gap 0.381gap 0.369gap 0.787gap 1.095mean cosine of two files in the same folderAfter per-modality centering0.00.20.40.60.81.0gap 0.080gap 0.080gap 0.092gap 0.106mean cosine of two files in the same folderE1 jina-v4E3 GMEE2 jina-clip-v2E4 Nomicimage with imagetext with textimage with textBefore centeringgap0.00.20.40.60.81.0E1 jina-v40.381E3 GME0.369E2 jina-clip-v20.787E4 Nomic1.095After per-modality centeringgap0.00.20.40.60.81.0E1 jina-v40.080E3 GME0.080E2 jina-clip-v20.092E4 Nomic0.106mean cosine of two files in the same folder
    Under every encoder an image and a text from the same folder are less alike than two images or two texts, even after centering has closed most of the gap between the modality means (right-hand numbers). Mean cosine over same-folder file pairs on S11. Source: Table 4 and Figure 5 of the paper.
  5. What fixes it, and at what budget?

    A few k-means representatives per modality label. None of the single vectors tried recovers the loss without giving up the majority. Four per label are not inferior to searching every file, at a 0.02 margin, over all queries and in every cell under all four encoders (post hoc). At three per label every folder’s representatives fit in 8 KiB, at a further cost of at most 0.012 in any cell. On ext4 a summary of 2,048 or 4,000 bytes costs 1.2 blocks per cold directory read.

    recall@5, E10.30.40.50.60.70.80.9All queries 0.804P1 0.685P2 0.551a1.00tb2c2.00c5.20d9.87vectors per folder on S11 (log scale)recall@5, E10.30.40.50.60.70.80.9All 0.804P1 0.685P2 0.551a1.00tb2c2.00c5.20d9.87vectors per folder on S11 (log scale)
    Most of what searching every file (d) gains over the mean (a) is already there at c, the per-modality representatives. Recall@5 under E1 on S11, with tb2c for two label-blind centered clusters. Source: Table 3 and Figure 7 of the paper.
    1 KiB2 KiB4 KiB8 KiB16 KiB32 KiB64 KiB8 KiBc at one bit per dimension1,331a, one vector in float328,192c at one byte per dimension10,648c in float3242,594bytes per folder, average under E1 (log scale)c at one bit per dimension1,331a, one vector in float328,192c at one byte per dimension10,648c in float3242,5941248163264KiB per folder, average under E1 (log scale)
    Stored at one bit per dimension, the representatives of c take less room on average than the single float32 vector of a. Every S11 folder’s c fits in 8 KiB at one or four bits per dimension, depending on the encoder. Source: Section 12 of the paper.
  6. Does a folder summary guide a walk down a tree?

    Not yet at 2 KiB. On 261 repository trees the guided walk finds the target 0.069 less often than reading every directory’s own summary, though it reads 0.39 as many.

  7. Does a written folder abstract do better?

    Not for the folder’s images.

Limitations

The Zenodo folders are researchers’ deposits, and the GitHub folders come from public repositories. Personal working directories, mailboxes and agent memories were not measured. Relevance is structural. A file’s one relevant folder is its own, so the test measures whether a folder vector represents its files. It does not measure topical relevance across folders.

No person wrote queries. The searcher queries cover 100 folders of one set and were written by a model, whose queries may be more specific than a person’s. The two vision-language embedders share a Qwen lineage. The two dual towers are the weakest encoders here, so part of their larger loss is the encoders.

Each Zenodo set ranks 400 to 2,808 folders, and summaries were written to one file system, ext4. Larger folders, deeper trees and other file systems were not measured.

MeasuredNot measuredZenodo deposits of 3 to 60 filesPersonal working directoriesDirectories of public GitHub repositoriesMailboxes and agent memoriesStructural relevance: a file’s own folderTopical relevance across foldersSearcher queries written by a model, 100 foldersQueries written by people400 to 2,808 folders ranked per Zenodo setLarger folders and deeper treesSummaries written to one file system, ext4Other file systemsMeasuredZenodo deposits of 3 to 60 filesDirectories of public GitHub repositoriesStructural relevance: a file’s own folderSearcher queries written by a model, 100 folders400 to 2,808 folders ranked per Zenodo setSummaries written to one file system, ext4Not measuredPersonal working directoriesMailboxes and agent memoriesTopical relevance across foldersQueries written by peopleLarger folders and deeper treesOther file systems
What the study measured, beside what it left out. Source: the limitations of the paper.

Pre-registration, code and data

The study ran as 27 sessions, each with a brief written before it. Twenty-two tested a hypothesis whose kill criterion was fixed before the numbers that decide it were computed.

The briefs of sessions 19, 21, 23, 24 and 27 are timestamped with OpenTimestamps, each anchored in a Bitcoin block, and the repository holds the proofs. The others are dated by the working repository’s history. Analyses added after review are marked post hoc, and the hypotheses that died or stayed inconclusive are reported beside the ones that survived.

after reviewSession 1: corpus, no hypothesis1Session 2: tested a hypothesis2Session 3: tested a hypothesis3Session 4: cost, no hypothesis4Session 5: tested a hypothesis5Session 6: tested a hypothesis6Session 7: tested a hypothesis7Session 8: tested a hypothesis8Session 9: tested a hypothesis9Session 10: tested a hypothesis10Session 11: tested a hypothesis11Session 12: tested a hypothesis12Session 13: re-run, no hypothesis13Session 14: tested a hypothesis14Session 15: tested a hypothesis15Session 16: tested a hypothesis16Session 17: tested a hypothesis17Session 18: tested a hypothesis18Session 19: tested a hypothesis19Session 20: tested a hypothesis20Session 21: tested a hypothesis21Session 22: file system, no hypothesis22Session 23: tested a hypothesis23Session 24: tested a hypothesis24Session 25: estimates, no hypothesis25Session 26: tested a hypothesis26Session 27: tested a hypothesis27corpuscostre-runfile systemestimatestested a hypothesis (22)no hypothesis (5)brief timestamped in a Bitcoin block (5)after reviewSession 1: corpus, no hypothesis1Session 2: tested a hypothesis2Session 3: tested a hypothesis3Session 4: cost, no hypothesis4Session 5: tested a hypothesis5Session 6: tested a hypothesis6Session 7: tested a hypothesis7Session 8: tested a hypothesis8Session 9: tested a hypothesis9Session 10: tested a hypothesis10Session 11: tested a hypothesis11Session 12: tested a hypothesis12Session 13: re-run, no hypothesis13Session 14: tested a hypothesis14Session 15: tested a hypothesis15Session 16: tested a hypothesis16Session 17: tested a hypothesis17Session 18: tested a hypothesis18Session 19: tested a hypothesis19Session 20: tested a hypothesis20Session 21: tested a hypothesis21Session 22: file system, no hypothesis22Session 23: tested a hypothesis23Session 24: tested a hypothesis24Session 25: estimates, no hypothesis25Session 26: tested a hypothesis26Session 27: tested a hypothesis27corpuscostre-runfile systemestimatestested a hypothesis (22)no hypothesis (5)brief timestamped in a Bitcoin block (5)
The 27 sessions in order. Filled squares tested a pre-registered hypothesis, and a dot marks a brief with an OpenTimestamps proof. Source: Section 3 and Figure 2 of the paper.

The repository holds the embedding and evaluation scripts, a second evaluation written from a specification, and the script and driver of every session. The outputs of sessions 25 to 27 rebuild from per-query files without a GPU.

The corpora are not redistributed. The manifests give each file’s path, hash, licence and source URL, and two fetch scripts rebuild the corpora from them. Ground truth, selections and per-query ranks are included.

Every brief is in BRIEF.md, and every session’s verdicts, readings and deviations are in results/. The script scripts/verify_prereg.py checks each timestamped commit from its git objects against the briefs.

Integrity

To check a copy of the PDF, compare its SHA-256 hash with this one.

SHA-256 of the PDF

8bef21aed187c640b44ebeb35f17b89704fc0df508e0e21186e8dbb82a9d5068

OpenTimestamps proof of the PDF

Cite as

APABasheer, A. (2026). A few vectors for a mixed folder: a pre-registered measurement of pooled directory embeddings (Version 1) [Preprint]. Zenodo. https://doi.org/10.5281/zenodo.23178122

IEEEA. Basheer, "A few vectors for a mixed folder: a pre-registered measurement of pooled directory embeddings," Zenodo, preprint, Oct. 6, 2026, doi: 10.5281/zenodo.23178122.

ChicagoBasheer, Ali. 2026. "A Few Vectors for a Mixed Folder: A Pre-Registered Measurement of Pooled Directory Embeddings." Version 1. Preprint, Zenodo, October 6. https://doi.org/10.5281/zenodo.23178122.

@misc{basheer2026fewvectors,  author       = {Basheer, Ali},  title        = {A few vectors for a mixed folder: a pre-registered measurement of pooled directory embeddings},  year         = {2026},  month        = oct,  date         = {2026-10-06},  publisher    = {Zenodo},  version      = {1},  doi          = {10.5281/zenodo.23178122},  url          = {https://doi.org/10.5281/zenodo.23178122},  note         = {Preprint, not peer reviewed. Code and data: https://github.com/ali-basheer/dirvec-study}}
BibTeX. Download as BibTeX, RIS, CSL JSON. The concept DOI 10.5281/zenodo.23178121 stands for all versions.

Related research

Work the paper builds on, grouped as the paper discusses it. The full bibliography has 69 entries.

The modality gap

Representing a collection

  • Callan, Lu and Croft (1995). Searching distributed collections with inference networks. SIGIR. doi:10.1145/215206.215328
  • Si and Callan (2003). Relevant document distribution estimation method for resource selection. SIGIR. doi:10.1145/860435.860490
  • Shokouhi and Si (2011). Federated Search. Foundations and Trends in Information Retrieval. doi:10.1561/1500000010

Folders and sources as units of retrieval

Several vectors per item

The four encoders

Corrections

Send corrections to ali@basheerali.com.