LTR Reranker — Juan Martos

Project 7/10 of the beacon-search-engine ecosystem — Ranking and Relevance category. Repository: github.com/juanmmm21/learning-to-rank-reranker

The piece that separates a toy search engine from a production one: instead of combining lexical relevance and link authority through a hand-tuned formula, this project trains a gradient-boosted learning-to-rank model (LightGBM, lambdarank objective) that learns how to weigh and combine BM25, PageRank, and index-derived matching features from labeled examples — the same family of approach real-world search rankers (web, e-commerce, marketplaces) actually use once a fixed formula stops scaling to the number of signals worth considering.

What problem it solves

bm25-ranking-engine scores lexical relevance to a query. pagerank-link-analysis scores structural authority independent of any query. Both are useful, both live on different, unbounded scales, and neither alone produces the best final ordering — the honest answer to "how much should authority matter relative to an exact term match in the title?" is not a constant any human should be confident hand-picking, and it very likely isn't even a fixed linear weight: the right trade-off plausibly depends on how a document matches (an early, dense match with high authority is not just BM25 + PageRank added together). This project's job is to stop treating that trade-off as a constant: it engineers a small set of explicitly normalized features per candidate document, generates a synthetic relevance-labeled training set (documented in full below), and lets a gradient-boosted tree ensemble learn the interactions between signals — including non-linear ones a fixed formula could never express — while still reporting standard, independently implemented ranking metrics (NDCG, MAP) on data the model never trained on.

Role in `beacon-search-engine`

inverted-index-builder ── bm25-ranking-engine
        │                          │ (doc_id, bm25_score) — not serialized,
        │ documents.jsonl,         │  caller writes this project's own
        │ postings.jsonl           │  candidates.jsonl contract
        ▼                          ▼
┌─────────────────────────────────────────┐
│         learning-to-rank-reranker         │   (this project)
│  BM25 + PageRank + match features → LTR   │
└──────────────────┬─────────────────────────┘
                    │ (doc_id, final_score), reordered
                    ▼
     beacon-search-console / distributed-index-sharding
                    ▲
                    │ doc_id → pagerank_score
           pagerank-link-analysis

learning-to-rank-reranker sits directly downstream of both ranking signals: it reads the uncompressed index from inverted-index-builder directly (for document length and term-position features), the on-disk output of pagerank-link-analysis, and a candidate list a caller has produced with bm25-ranking-engine. It never imports any of those projects' Python packages — integration is always through the formats each of them documents on disk, per beacon-search-engine's polyrepo contract. Its own output — a reordered (doc_id, final_score) list — is what beacon-search-console presents to a user, and what distributed-index-sharding applies after merging results from multiple shards.

Goal and skills demonstrated

Feature engineering over heterogeneous, differently-scaled signals: BM25 is unbounded and corpus/query-dependent, PageRank is a bounded probability, match position and document length are raw index quantities — every one of them is made explicitly comparable before being handed to a model (../CLAUDE.md, section 2.C), never combined on their raw scales.
Train/serve consistency treated as a first-class correctness concern, not an afterthought: the exact same normalization function (normalization.min_max_normalize_within_group) is used to build the synthetic training set and to score real candidates at inference time — a single shared implementation, not two independently-written copies that could silently drift apart (the classic, hard-to-detect "train/serve skew" bug class in production ML systems).
A clearly documented synthetic labeling heuristic, not a black box: v1 has no human-labeled relevance judgments available (../AGENTS.md), so synthetic_dataset.py generates one from distributions that mimic each real signal's actual shape (BM25's right skew, PageRank's power-law concentration) plus injected noise and per-query quantile discretization — every choice justified in that module's docstring, specifically so that no single feature alone can perfectly explain the resulting labels, giving a learned model real room to find interactions a fixed formula wouldn't.
Correct query-grouped evaluation, not row-wise metrics: NDCG and MAP are reimplemented from scratch (metrics.py) and computed per query group, and the train/validation split is done by whole query groups rather than individual rows — mixing rows from the same query between the two sets would leak information and produce misleadingly optimistic metrics, the standard information-leakage failure mode specific to learning-to-rank.
A third-party dependency used exactly where it is justified, and nowhere else: beacon-search-engine implements every core algorithm from scratch, but this project's explicit purpose is demonstrating the feature/pipeline engineering around a learned ranker, not re-deriving gradient boosting — so lightgbm is used for the model itself, while feature extraction, dataset generation, normalization, and evaluation metrics remain hand-written.

How it works

Reading the index (index_reader.py): a minimal, independent reader for the uncompressed inverted-index-builder format (documents.jsonl + postings.jsonl + stats.json) — deliberately not the compressed index-compression-codec format, since decoding that binary layout is already demonstrated in bm25-ranking-engine; duplicating it here would add surface area without demonstrating anything new (see the module's docstring for the full reasoning).
Reading PageRank scores (pagerank_reader.py): an independent reader of pagerank-link-analysis's output directory (manifest.json + pagerank_scores.jsonl), validating every score falls in [0, 1].
Reading candidates (candidates_io.py): bm25-ranking-engine does not serialize its ranked list to disk at all — its own README documents that its output is purely in-memory. This project therefore defines its own minimal JSONL contract for (doc_id, bm25_score) candidates (see Data formats below), which a caller (in production, beacon-search-console, which does import bm25-ranking-engine as a package dependency) writes before invoking this one.
Extracting features (feature_extraction.py): for each candidate, BM25 is min-max normalized against the other candidates of the same query; PageRank is used as-is (already bounded); document length and the earliest matched-term position are turned into bounded ratios relative to the corpus average length and the document's own length, respectively; a candidate with no matching query term gets explicit sentinel values instead of NaN. normalization.py documents each transform.
Generating training data (synthetic_dataset.py): produces a query-grouped synthetic dataset with graded relevance labels 0..4, sampling each feature from a distribution shaped like its real-world counterpart and deriving labels from a weighted combination plus noise, discretized per query by quantile — the exact recipe is documented in this module's docstring, in full, as ../AGENTS.md requires for v1.
Training and evaluating (model.py, metrics.py, pipeline.py): LightGBMReranker wraps a lambdarank booster forced into single-threaded deterministic mode for reproducible output ordering; pipeline.train splits the synthetic dataset by whole query groups (never by row) into training and validation, fits the model, and evaluates NDCG@10/MAP@10 — both reimplemented from scratch in metrics.py — on the held-out queries.
Serving (pipeline.rerank, __main__.py): loads a trained model, extracts features for a real candidate list against a real index and PageRank output, predicts a score per candidate, and returns the final order — ties broken by ascending doc_id, never by input order (../CLAUDE.md, section 2.B).

Architecture

src/learning_to_rank_reranker/
├── models.py                # FEATURE_NAMES, RankingFeatures, Candidate, TrainingDataset, ...
├── protocols.py               # Reranker / TrainableReranker interfaces
├── normalization.py            # min-max-within-group, bounded ratio — shared train/inference
├── index_reader.py              # InvertedIndexReader (inverted-index-builder, uncompressed)
├── pagerank_reader.py            # independent reader of pagerank-link-analysis's output
├── candidates_io.py               # this project's own (doc_id, bm25_score) JSONL contract
├── feature_extraction.py           # candidates + index + pagerank -> RankingFeatures
├── synthetic_dataset.py             # v1 synthetic/heuristic training data generator
├── metrics.py                        # NDCG@k, Average Precision@k, from scratch
├── model.py                           # LightGBMReranker: fit, predict, save, load
├── tokenizer.py                        # minimal NFC + casefold tokenizer, CLI convenience only
├── pipeline.py                          # RerankerPipeline-equivalent: train() and rerank()
└── __main__.py                           # CLI (`train`, `rerank`)

Requirements and installation

Python >=3.11
Runtime dependencies: numpy (feature arrays, metrics) and lightgbm (the one third-party dependency justified for the model itself — see Goal and skills demonstrated).
LightGBM's compiled backend needs the OpenMP runtime on macOS: brew install libomp if import lightgbm fails with a libomp.dylib load error (not required on Linux, where it ships with the wheel).

git clone https://github.com/juanmmm21/learning-to-rank-reranker.git
cd learning-to-rank-reranker
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

Usage (CLI)

# Train a model on the v1 synthetic dataset and serialize it
learning-to-rank-reranker train ./model --num-queries 200 --candidates-per-query 20 --seed 42

# Rerank a candidate list against a real index + PageRank output + query
learning-to-rank-reranker rerank ./index ./pagerank-output ./candidates.jsonl ./model \
  "search engine ranking" --top-k 10

Try it end-to-end against the tiny fixture checked into samples/tiny_example/ (a 5-document index, hand-picked PageRank scores, and 4 BM25 candidates for the query "search engine ranking"):

learning-to-rank-reranker train /tmp/ltr-example-model --num-queries 60 --candidates-per-query 12
learning-to-rank-reranker rerank \
  samples/tiny_example/index samples/tiny_example/pagerank samples/tiny_example/candidates.jsonl \
  /tmp/ltr-example-model "search engine ranking"

It can also be invoked as a module: python -m learning_to_rank_reranker train ....

Data formats

Input — index: the uncompressed manifest.json + documents.jsonl + postings.jsonl + stats.json produced by inverted-index-builder (see that project's README for the full schema). A compressed index-compression-codec index is rejected explicitly with a clear error — see How it works, step 1, for why.

Input — PageRank output: the manifest.json + pagerank_scores.jsonl directory produced by pagerank-link-analysis.

Input — candidates, this project's own contract (candidates_io.py), since bm25-ranking-engine produces no on-disk format of its own: one JSON object per line —

{"doc_id": 4, "bm25_score": 8.5}

Output — trained model: train/LightGBMReranker.save write a directory with two files —

manifest.json — format version, feature order, and training hyperparameters, so a serialized model is self-describing:

{
  "format_version": 1,
  "model_file": "model.txt",
  "feature_names": [
    "bm25_normalized", "pagerank_score", "matched_term_ratio",
    "min_term_position_normalized", "term_span_normalized",
    "document_length_normalized"
  ],
  "params": {"objective": "lambdarank", "num_leaves": 15, "seed": 42, "...": "..."}
}

model.txt — LightGBM's native text model format: human-readable, reconstructible exactly via lightgbm.Booster(model_file=...). LightGBMReranker.load refuses to load a model whose recorded feature_names don't match this codebase's current FEATURE_NAMES order.

Output — reranked results: an in-memory list of RankedDocument(doc_id, final_score), printed by the CLI as rank. doc_id=... final_score=... — this project produces no on-disk format of its own for its final output, matching bm25-ranking-engine's own convention for its ranked list.

Programmatic usage

from pathlib import Path

from learning_to_rank_reranker.candidates_io import read_candidates
from learning_to_rank_reranker.model import LightGBMReranker, LightGBMRankerParams
from learning_to_rank_reranker.pipeline import rerank, train

# Training
model = LightGBMReranker(LightGBMRankerParams(seed=42))
report = train(model, num_queries=200, candidates_per_query=20, seed=42)
print(report.validation.ndcg_at_10, report.validation.map_at_10)
model.save(Path("./model"))

# Inference
loaded_model = LightGBMReranker.load(Path("./model"))
candidates = read_candidates(Path("./candidates.jsonl"))
results = rerank(
    loaded_model,
    candidates,
    ["search", "engine", "ranking"],
    Path("./index"),
    Path("./pagerank-output"),
    top_k=10,
)
for ranked in results:
    print(ranked.doc_id, ranked.final_score)

Development

pytest
ruff check .
ruff format --check .
mypy --strict src/

Tests cover the domain models (feature range validation, dataset shape validation), the normalization utilities (degenerate and boundary cases), the index/PageRank/candidates readers (round-trips and malformed-input errors, against small on-disk fixtures), feature extraction (hand-verified expected values, including the no-match sentinel path), the synthetic dataset generator (determinism under a fixed seed, per-query grade spread, parameter validation), the ranking metrics (values checked against an independently hand-computed DCG/AP for a known ranking), the LightGBM model wrapper (fit/predict/save/load round-trip, determinism under a fixed seed, rejecting a model saved with a different feature order), the end-to-end pipeline (query-grouped train/validation split, deterministic tie-break in rerank, using in-memory Reranker/TrainableReranker test doubles rather than a real LightGBM model where the intent is testing this project's own orchestration logic), and the CLI.

Troubleshooting

OSError: ... Library not loaded: @rpath/libomp.dylib on macOS: LightGBM's compiled backend needs the OpenMP runtime, which isn't bundled on macOS — brew install libomp and retry.
ValueError: el modelo en <path> se entrenó con features (...) pero esta versión del código espera (...): the model directory was produced by an older or incompatible version of this project's feature set — FEATURE_NAMES in models.py changed after that model was trained. Retrain with the current code rather than trying to reuse it.
RuntimeError: el modelo no está entrenado from predict() or save(): call fit() (or load a serialized model with LightGBMReranker.load) before either.
rerank returns a document with pagerank_score unexpectedly at 0.0: that doc_id has no entry in the PageRank output directory — either it wasn't part of the crawled link graph, or the PageRank and index runs are out of sync. This is treated as "no measurable link authority", not an error (see feature_extraction.py's docstring).
ValueError: ... es un índice comprimido de index-compression-codec: this project only reads the uncompressed inverted-index-builder format — see How it works, step 1, for why the compressed format isn't supported here.
Training metrics (NDCG@10/MAP@10) look unrealistically high: this is expected on v1's synthetic dataset — the labels are generated from a known (if noisy) function of the same features the model trains on, so a gradient-boosted model recovers that relationship closely. The point of the synthetic data isn't a realistic absolute metric value, it's a reproducible pipeline that can be pointed at real relevance judgments later without changing any of the surrounding code.

License

MIT — see LICENSE.

Layer 3 of Beacon — ranking & relevance.