Commands

odoc-index

Index selected OCaml documentation packages and understand the fixed package filter.

odoc-index is a small command-line tool that converts the HTML pages generated by the odoc tool-chain into a search-optimised corpus made of:

dense vector embeddings – suitable for semantic / nearest-neighbour search;
a BM-25 lexical index – for traditional full-text queries;
the original Markdown snippets – so results can be displayed with context.

A companion executable, odoc-search, consumes the artefacts produced by odoc-index to answer user queries. The two tools mirror the “index → search” workflow provided by utilities such as apropos or man -k, but with modern machine-learning goodies under the hood.

When should I run it?

• After regenerating your documentation (dune build @doc, odig odoc, …).
• After upgrading or pinning new opam packages.
• Whenever you want the search index to reflect the latest docs on disk.

The current binary is not a general incremental package updater. Its hard-coded Update filter writes snippet/vector/BM25 indexes only for ochat, textmate-language, irmin, irmin-git, and irmin-watcher. It crawls other non-excluded packages for the coarse package catalog, which does not imply their snippet indexes exist. Use the library with an explicit filter, or change the wrapper, when you need other packages; no CLI package-filter flag is exposed.

CLI reference

odoc-index --root <html-doc-root> [--out <output-dir>]

Options:
  --root <path>   root directory containing package sub-folders generated by
                  odoc / odig.  **Required.**
  --out  <dir>    destination directory for the index (default `.odoc_index`).

Both paths can be absolute or relative; they are resolved with respect to the current working directory at runtime.

The tool exits with status 1 if --root is omitted or does not exist.

End-to-end example

$ dune build @doc                                 # 1. generate html docs
$ odoc-index --root _build/default/_doc/_html      # 2. index them
Indexing completed in .odoc_index

# Illustrative layout when these selected packages are present:
$ tree -L 2 .odoc_index | head -n 15
├── ochat
│   ├── bm25.binio
│   ├── vectors.binio
│   ├── 00003d54-….md
│   ├── 00007af9-….md
│   └── …
├── textmate-language
│   ├── bm25.binio
│   ├── vectors.binio
│   ├── 000012ab-….md
│   └── …
└── package_index.binio

You can now run odoc-search (or any other tool that understands the on-disk format) to query the index.

How does it work internally?

odoc-index is a thin wrapper around the Odoc_indexer.index_packages function. The indexer performs the heavy lifting:

Crawling – traverses every HTML file under --root.
Slicing – converts HTML to Markdown and chops it into 64–320 token windows so each chunk fits in the OpenAI embedding limit.
Embedding – batches chunks (≤ 300 at a time) and calls the OpenAI Embeddings API, enforcing provider rate limits.
Persistence – writes the vectors, BM-25 structures and raw Markdown to --out/<pkg>/.
Package index – stores a one-line blurb per package in package_index.binio so the search tool can provide quick previews.

All IO-bound operations (HTTP calls, filesystem writes) run concurrently using Eio. CPU-heavy slicing tasks are off-loaded to a pool of domains for maximal throughput.

Configuration & customisation

The current implementation hard-codes a small exclude list for packages that either duplicate the OCaml stdlib or contain documentation noise:

Exclude [ "ocaml"; "ocaml_intrinsics_kernel"; "ocaml-compiler-libs"
        ; "ocamlgraph"; "tls" ]

Feel free to adjust the list or replace it with:

Include [ "my_pkg1"; "my_pkg2" ]

to whitelist a subset. See the package_filter type in Odoc_indexer for all available options.

Those are source/library changes, not accepted command-line arguments. The binary actually wraps the exclusion set in Update (..., ["ochat"; "textmate-language"; "irmin"; "irmin-git"; "irmin-watcher"]). Selected packages are re-embedded; the filter does not compare old and new source hashes.

Limitations & gotchas

Embedding mode – missing/empty OPENAI_API_KEY or a present OPENAI_EMBEDDINGS_STUB selects test vectors; see embedding configuration.
Rate limits – despite local throttling the OpenAI API may still return 429 / 502 errors during traffic spikes. The indexer retries up to three times after the first attempt. Failed snippet batches are caught and omitted, so completion can leave a partial index; inspect diagnostics and results.
Large documentation trees – a full odoc render of the OCaml ecosystem can weigh several GB. Make sure you have enough disk space in --out.
Schema stability – the on-disk format is alpha; breaking changes may occur between minor versions.

API summary (for developers)

val main : Eio.Stdenv.base -> unit
(* Entry-point used by [Eio_main.run]. *)

val root_dir : string ref
val out_dir  : string ref
(* Mutable references updated through [Arg.parse]. *)

For more granular control (e.g. alternative slicing strategies, different embedding models) call Odoc_indexer.index_packages directly.