[zhangstar333]

Add a versioned protobuf contract and Rust/JNI/Java APIs to collect global BM25 statistics across all committed FTS segments, preserving prepared query tokens and positions.

V1 supports one corpus with Match/Phrase leaves. Executor-side payload consumption remains follow-up work.

Part of #8937. Incorporates the producer implementation from #8822.
and supports the downstream Doris use case described in apache/doris#67435.

[zhangstar333]

@Xuanwo hi, could you please have a look at this pr when you are free, very Thanks!

[sbrunk]

Suggestion to make V1 easier to extend: use a multi-corpus layout, but restrict V1 to one corpus, document counts, Match/Phrase leaves.

 message FtsGlobalStatisticsProto {
   FtsGlobalStatisticsSchemaVersion schema_version = 1;
   uint64 dataset_version = 2;
-  string index_name = 3;
-  string column = 4;
-  repeated bytes segment_uuids = 5;
-  optional uint64 total_tokens = 6;
-  optional uint64 num_docs = 7;
-  repeated FtsTermStatisticsProto terms = 8;
-  FtsDocumentGranularity document_granularity = 9;
-  repeated FtsPreparedQueryLeafProto query_leaves = 10;
+  // Unique prepared terms referenced by any leaf, sorted by UTF-8 bytes.
+  repeated string terms = 3;
+  // V1: exactly one entry.
+  repeated FtsCorpusStatisticsProto corpora = 4;
+  repeated FtsPreparedQueryLeafProto query_leaves = 5;
 }

-message FtsTermStatisticsProto {
-  string term = 1;
-  optional uint64 document_frequency = 2;
-}
+// Statistics of one FTS index on one column, over its committed segments.
+message FtsCorpusStatisticsProto {
+  string index_name = 1;
+  string column = 2;
+  FtsDocumentGranularity document_granularity = 3;
+  // V1: only DOCUMENT.
+  FtsCorpusCountUnit count_unit = 4;
+  repeated bytes segment_uuids = 5;
+  optional uint64 num_docs = 6;
+  optional uint64 total_tokens = 7;
+  // Sorted by term_index.
+  repeated FtsCorpusTermStatisticsProto terms = 8;
+}
+
+message FtsCorpusTermStatisticsProto {
+  optional uint32 term_index = 1;           // into FtsGlobalStatisticsProto.terms
+  optional uint64 document_frequency = 2;
+}

 message FtsPreparedQueryLeafProto {
   optional uint32 leaf_ordinal = 1;
-  repeated FtsPreparedTokenProto tokens = 2;
-  optional bool has_all_query_positions = 3;
+  // V1: only MATCH or PHRASE.
+  FtsQueryLeafKind kind = 2;
+  // V1: always [0].
+  repeated uint32 corpus_indices = 3;
+  repeated FtsPreparedTokenProto tokens = 4;
+  optional bool has_all_query_positions = 5;
 }
+
+enum FtsCorpusCountUnit {
+  FTS_CORPUS_COUNT_UNIT_UNSPECIFIED = 0;
+  FTS_CORPUS_COUNT_UNIT_DOCUMENT = 1;
+}
+
+enum FtsQueryLeafKind {
+  FTS_QUERY_LEAF_KIND_UNSPECIFIED = 0;
+  FTS_QUERY_LEAF_KIND_MATCH = 1;
+  FTS_QUERY_LEAF_KIND_PHRASE = 2;
+}

FtsPreparedTokenProto is unchanged. Its term_index still points into FtsGlobalStatisticsProto.terms, which now holds only the term strings.

V1 consumers reject any payload outside these restrictions, including unknown enum values.

Why

Cross-column queries like cross-column compound (#8689) and combined_fields/BM25F (WIP in #7905), will need:

• one corpus per column
• leaves that reference one or more corpora
• for BM25F, statistics counted per row instead of per document, since legacy V1/V2 list indexes store one document per list element.

With the current flat layout, that means bumping schema_version and restructuring. Just adding fields isn't safe: V1 consumers ignore unknown fields and would score with incomplete or wrongly counted statistics, without raising an error.

With the layout above, supporting those queries later only means allowing more values: several corpora, a ROW count unit, a COMBINED_FIELDS leaf kind. There are no layout changes and no new fields. V1 consumers reject such payloads instead of misreading them, and single-column payloads stay valid across versions.
The shared term list exists because one combined_fields token is looked up in several columns, each with its own df.

Adding full cross-column support would then look like this sketch. Alternatively, we could also go with full support in V1 directly.

 message FtsGlobalStatisticsProto {
   FtsGlobalStatisticsSchemaVersion schema_version = 1;
   uint64 dataset_version = 2;
   // Unique prepared terms referenced by any leaf, sorted by UTF-8 bytes.
   repeated string terms = 3;
-  // V1: exactly one entry.
+  // Unique and sorted by (index_name, column, document_granularity, count_unit).
+  // Every corpus is referenced by at least one leaf.
   repeated FtsCorpusStatisticsProto corpora = 4;
   repeated FtsPreparedQueryLeafProto query_leaves = 5;
 }

 message FtsCorpusStatisticsProto {
   string index_name = 1;
   string column = 2;
   FtsDocumentGranularity document_granularity = 3;
-  // V1: only DOCUMENT.
   FtsCorpusCountUnit count_unit = 4;
   repeated bytes segment_uuids = 5;
   optional uint64 num_docs = 6;
   optional uint64 total_tokens = 7;
-  // Sorted by term_index.
+  // Sorted by term_index. Covers every term that a leaf bound to this
+  // corpus references.
   repeated FtsCorpusTermStatisticsProto terms = 8;
 }

 message FtsPreparedQueryLeafProto {
+  // CombinedFields emits one leaf and is not descended into.
   optional uint32 leaf_ordinal = 1;
-  // V1: only MATCH or PHRASE.
   FtsQueryLeafKind kind = 2;
-  // V1: always [0].
+  // Indices into corpora.
+  // MATCH/PHRASE: exactly one corpus, with count_unit DOCUMENT.
+  // COMBINED_FIELDS: one corpus per target column, in the query's column
+  //   order (which pairs each corpus with its weight), all with count_unit
+  //   ROW and document_granularity ROW, no duplicates.
   repeated uint32 corpus_indices = 3;
   repeated FtsPreparedTokenProto tokens = 4;
+  // Always true for COMBINED_FIELDS (no fuzziness).
   optional bool has_all_query_positions = 5;
 }

 enum FtsCorpusCountUnit {
   FTS_CORPUS_COUNT_UNIT_UNSPECIFIED = 0;
   FTS_CORPUS_COUNT_UNIT_DOCUMENT = 1;
+  // One unit per distinct row id. Required by COMBINED_FIELDS (BM25F), since
+  // legacy V1/V2 list indexes store one document per list element.
+  FTS_CORPUS_COUNT_UNIT_ROW = 2;
 }

 enum FtsQueryLeafKind {
   FTS_QUERY_LEAF_KIND_UNSPECIFIED = 0;
   FTS_QUERY_LEAF_KIND_MATCH = 1;
   FTS_QUERY_LEAF_KIND_PHRASE = 2;
+  FTS_QUERY_LEAF_KIND_COMBINED_FIELDS = 3;
 }

@BubbleCal

[zhangstar333]

Thanks @sbrunk! Updated the schema following your suggestion: shared terms, per-corpus statistics, and explicit leaf kinds and corpus references. V1 remains limited to one corpus.
I also included the Rust/JNI/Java producer APIs in this PR. Could you take another look when you have a chance?

[lance-gatekeeper]

✅ Gate recommendation: approve.

The producer preserves logical Match term groups separately from Phrase tokenizer offsets, including through fuzzy expansion. The statistics-only segment opener avoids live-row coverage I/O without changing the committed-corpus scope. Focused Rust producer and coverage regressions pass.

V1 remains limited to one DOCUMENT-counted corpus and Match/Phrase leaves. The trusted planner must attach the payload only to its source query; payload consumption, validation, and distributed ranking equivalence remain follow-up work under #8937.

[Xuanwo]

Approved the final merged head. The producer preserves logical Match term groups and Phrase tokenizer positions, computes DOCUMENT statistics over committed segments, and avoids live-row coverage I/O for statistics-only loading. The final head passed all applicable CI. Payload consumption, validation, and distributed ranking equivalence remain follow-up work under #8937.