OpenAPI: emit labels from the REST server test fixture by laskoviymishka · Pull Request #18047 · apache/iceberg

laskoviymishka · 2026-09-10T11:34:28Z

Part of the catalog-labels work (spec: #15750).

Adds an include-labels option to the REST server test fixture (RESTServerCatalogAdapter) that attaches catalog-provided labels to load-table and load-view responses, plus a test that asserts they round-trip over the wire. This gives other client implementations (PyIceberg, iceberg-go, iceberg-rust) a reference server that returns labels to test against.

Add an include-labels option to the REST server test fixture (RESTServerCatalogAdapter) that attaches catalog-provided labels to load-table and load-view responses, plus a test that asserts they round-trip over the wire. This gives other client implementations a reference server that returns labels to test against.

Inline the catalog-name / catalog-host / classification label keys at their single use sites, per review. Co-authored-by: Isaac <no-reply@databricks.com>

InetAddress.getLocalHost().getHostName() can trigger a DNS lookup and is flagged by error-prone's ReverseDnsLookup check. Read the HOSTNAME environment variable instead: it is set in container environments such as the REST fixture image, and the label is omitted when it is absent. Generated-by: Claude Code

…em table ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: The Iceberg REST catalog spec now lets a catalog return `labels` with each table load (apache/iceberg#15750): catalog-owned, per-request metadata such as ownership, classification or cost attribution, at the table level (`object-labels`) and per field (keyed by `field-id`). Labels are not part of the table metadata, have no snapshots and are never written to `metadata.json`. Iceberg 1.12.0 exposes them on the loaded table via `SupportsLabels#labels()`. Doris had no way to show them. Add a `t$labels` system table for iceberg tables in a REST catalog: SELECT * FROM rest_ctl.db.tbl$labels; scope | field_id | field_name | key | value object | NULL | NULL | owner | team-a field | 2 | email | classification | pii One row per label. `field_name` is resolved against the current table schema and is NULL for a field id that is no longer in it (a dropped column). The schema mirrors iceberg's proposed `LabelsTable`. Implementation: - Keep labels when Doris rebuilds a loaded table. iceberg's REST client attaches them to the loaded `BaseTable` as a transient field; they are not table metadata, so a `BaseTable` rebuilt from its `TableOperations` must pass them explicitly. `IcebergTableCache.TableLease#snapshotReadTable` (cached reads), `IcebergStatementScope#snapshotReadTable` (statement reads) and `IcebergScanPlanProvider#wrapTableForScan` (Kerberos FileIO wrap) used constructors that reset labels to `Labels.EMPTY`, so every read saw no labels; they now pass `baseTable.labels()`. - `labels` is not a `MetadataTableType`, so it is handled next to the SDK metadata tables rather than through `MetadataTableUtils`. `IcebergConnectorMetadata` lists and resolves it only when the catalog flavor is `rest` (no other catalog returns labels), and serves its fixed schema and column handles without loading the base table. - `IcebergScanPlanProvider` plans it as one split. A new split-package helper `org.apache.iceberg.IcebergLabelsDataTask` builds an iceberg `StaticDataTask` (package-private in the SDK) holding the label rows, projected to the requested columns in BE's required-field order; its metadata `DataFile` is synthesized, so no storage request is made. The task is serialized like every other static metadata table, so BE's `IcebergSysTableJniScanner` reads it with no BE change. - The ordered system-table projection is extracted into `projectSystemTableSchema` and shared with the existing path, and the base-table load is split out of `resolveSysTable` as `resolveSysBaseTable`. - Labels are not versioned: like `$snapshots`/`$history`, the snapshot pin of the handle is ignored, and `$labels` is exempt from the resolved-empty-snapshot fence, so a table with no commit still shows its labels. Labels reflect the table load the statement used, so they follow the iceberg table cache (`meta.cache.iceberg.table.ttl-second`); REFRESH TABLE picks up new labels. Like the other system tables, a REST catalog that requires server-side scan planning rejects `$labels`. ### Release note Support querying REST catalog labels of an iceberg table through the `tbl$labels` system table. ### Check List (For Author) - Test: Unit Test - Added IcebergLabelsDataTaskTest, the $labels cases in IcebergConnectorMetadataSysTableTest, IcebergTableCacheTest#snapshotReadTableKeepsCatalogLabels and IcebergStatementScopeTest#sharedTableKeepsCatalogLabels. - `build.sh --fe` (Maven build cache disabled) passes on top of current master. Ran every iceberg and hudi FE unit test with run-fe-ut.sh: 1990 tests, 0 failures, 5 skipped. - Regression test pending a REST fixture that returns labels (apache/iceberg#18047). - Behavior changed: No (adds a new system table for REST catalogs only) - Does this need documentation: Yes (doc PR to follow) Co-authored-by: Isaac <no-reply@databricks.com>

…m table ### What problem does this PR solve? Issue Number: apache#68916 Related PR: apache#69135 Problem Summary: t$labels was covered by unit tests only: nothing ran a REST table load that carries labels through FE planning and the BE JNI system-table scanner. No REST fixture returns labels yet (the upstream fixture option is apache/iceberg#18047), so the new suite test_iceberg_rest_labels starts a small HTTP proxy inside the regression runner, in front of the docker iceberg REST fixture. The proxy forwards every request unchanged and adds the suite's labels to load-table responses; the iceberg REST client, FE and BE run unmodified. It needs no docker change and no new image. The suite checks: - object labels, field labels and a nested field named by its full path (address.city), including a non-ASCII value; - another column order, a column subset, filters, and COUNT(*), which counts the labels (7) rather than the table's records (2); - a table with no snapshot still shows its labels; a table without labels, and the labelled table read without the proxy, show none; - a catalog with vended credentials, which has no table cache, shows the labels too; - after RENAME COLUMN a label follows the new name, after DROP COLUMN its field_name is NULL; FOR VERSION AS OF is ignored, because labels are not versioned, and names still come from the current schema; - the cached catalog keeps the loaded labels until REFRESH TABLE, while a catalog with meta.cache.iceberg.table.ttl-second = 0 sees new labels at once; - a hadoop catalog rejects t$labels with "Unknown sys table". ### Release note None ### Check List (For Author) - Test: Regression test - test_iceberg_rest_labels passes locally against the iceberg docker fixture, with FE built from this PR merged into master. - Mutation check: dropping the labels in IcebergTableCache.TableLease#snapshotReadTable, building the labels split without the requested projection, or offering t$labels on every catalog type each makes the suite fail. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…em table ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: The Iceberg REST catalog spec now lets a catalog return `labels` with each table load (apache/iceberg#15750): catalog-owned, per-request metadata such as ownership, classification or cost attribution, at the table level (`object-labels`) and per field (keyed by `field-id`). Labels are not part of the table metadata, have no snapshots and are never written to `metadata.json`. Iceberg 1.12.0 exposes them on the loaded table via `SupportsLabels#labels()`. Doris had no way to show them. Add a `t$labels` system table for iceberg tables in a REST catalog: SELECT * FROM rest_ctl.db.tbl$labels; scope | field_id | field_name | key | value object | NULL | NULL | owner | team-a field | 2 | email | classification | pii One row per label. `field_name` is resolved against the current table schema and is NULL for a field id that is no longer in it (a dropped column). The schema mirrors iceberg's proposed `LabelsTable`. Implementation: - Keep labels when Doris rebuilds a loaded table. iceberg's REST client attaches them to the loaded `BaseTable` as a transient field; they are not table metadata, so a `BaseTable` rebuilt from its `TableOperations` must pass them explicitly. `IcebergTableCache.TableLease#snapshotReadTable` (cached reads), `IcebergStatementScope#snapshotReadTable` (statement reads) and `IcebergScanPlanProvider#wrapTableForScan` (Kerberos FileIO wrap) used constructors that reset labels to `Labels.EMPTY`, so every read saw no labels; they now pass `baseTable.labels()`. - `labels` is not a `MetadataTableType`, so it is handled next to the SDK metadata tables rather than through `MetadataTableUtils`. `IcebergConnectorMetadata` lists and resolves it only when the catalog flavor is `rest` (no other catalog returns labels), and serves its fixed schema and column handles without loading the base table. - `IcebergScanPlanProvider` plans it as one split. A new split-package helper `org.apache.iceberg.IcebergLabelsDataTask` builds an iceberg `StaticDataTask` (package-private in the SDK) holding the label rows, projected to the requested columns in BE's required-field order; its metadata `DataFile` is synthesized, so no storage request is made. The task is serialized like every other static metadata table, so BE's `IcebergSysTableJniScanner` reads it with no BE change. - The ordered system-table projection is extracted into `projectSystemTableSchema` and shared with the existing path, and the base-table load is split out of `resolveSysTable` as `resolveSysBaseTable`. - Labels are not versioned: like `$snapshots`/`$history`, the snapshot pin of the handle is ignored, and `$labels` is exempt from the resolved-empty-snapshot fence, so a table with no commit still shows its labels. Labels reflect the table load the statement used, so they follow the iceberg table cache (`meta.cache.iceberg.table.ttl-second`); REFRESH TABLE picks up new labels. Like the other system tables, a REST catalog that requires server-side scan planning rejects `$labels`. ### Release note Support querying REST catalog labels of an iceberg table through the `tbl$labels` system table. ### Check List (For Author) - Test: Unit Test - Added IcebergLabelsDataTaskTest, the $labels cases in IcebergConnectorMetadataSysTableTest, IcebergTableCacheTest#snapshotReadTableKeepsCatalogLabels and IcebergStatementScopeTest#sharedTableKeepsCatalogLabels. - `build.sh --fe` (Maven build cache disabled) passes on top of current master. Ran every iceberg and hudi FE unit test with run-fe-ut.sh: 1990 tests, 0 failures, 5 skipped. - Regression test pending a REST fixture that returns labels (apache/iceberg#18047). - Behavior changed: No (adds a new system table for REST catalogs only) - Does this need documentation: Yes (doc PR to follow) Co-authored-by: Isaac <no-reply@databricks.com>

…m table ### What problem does this PR solve? Issue Number: apache#68916 Related PR: apache#69135 Problem Summary: t$labels was covered by unit tests only: nothing ran a REST table load that carries labels through FE planning and the BE JNI system-table scanner. No REST fixture returns labels yet (the upstream fixture option is apache/iceberg#18047), so the new suite test_iceberg_rest_labels starts a small HTTP proxy inside the regression runner, in front of the docker iceberg REST fixture. The proxy forwards every request unchanged and adds the suite's labels to load-table responses; the iceberg REST client, FE and BE run unmodified. It needs no docker change and no new image. The suite checks: - object labels, field labels and a nested field named by its full path (address.city), including a non-ASCII value; - another column order, a column subset, filters, and COUNT(*), which counts the labels (7) rather than the table's records (2); - a table with no snapshot still shows its labels; a table without labels, and the labelled table read without the proxy, show none; - a catalog with vended credentials, which has no table cache, shows the labels too; - after RENAME COLUMN a label follows the new name, after DROP COLUMN its field_name is NULL; FOR VERSION AS OF is ignored, because labels are not versioned, and names still come from the current schema; - the cached catalog keeps the loaded labels until REFRESH TABLE, while a catalog with meta.cache.iceberg.table.ttl-second = 0 sees new labels at once; - a hadoop catalog rejects t$labels with "Unknown sys table". ### Release note None ### Check List (For Author) - Test: Regression test - test_iceberg_rest_labels passes locally against the iceberg docker fixture, with FE built from this PR merged into master. - Mutation check: dropping the labels in IcebergTableCache.TableLease#snapshotReadTable, building the labels split without the requested projection, or offering t$labels on every catalog type each makes the suite fail. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

github-actions Bot added the OPENAPI label Sep 10, 2026

laskoviymishka force-pushed the laskoviymishka/labels-fixture branch from ff2a8b0 to f704351 Compare September 10, 2026 14:50

laskoviymishka changed the base branch from laskoviymishka/labels-serde to laskoviymishka/labels-spark September 10, 2026 14:51

laskoviymishka added this pull request to stack #18051 September 10, 2026 14:51

laskoviymishka mentioned this pull request Sep 10, 2026

Core: add labels to the load table/view read path #18045

Merged

laskoviymishka requested a review from nastra September 10, 2026 15:59

laskoviymishka force-pushed the laskoviymishka/labels-spark branch from 5c43066 to 7e89da2 Compare September 14, 2026 15:31

laskoviymishka force-pushed the laskoviymishka/labels-fixture branch from f704351 to 53bce97 Compare September 14, 2026 15:31

laskoviymishka mentioned this pull request Sep 15, 2026

Catalog Labels: client-library read support (Go / Rust / Python / C++) #18126

Open

14 tasks

laskoviymishka force-pushed the laskoviymishka/labels-spark branch from 7e89da2 to a812269 Compare September 17, 2026 15:02

laskoviymishka force-pushed the laskoviymishka/labels-fixture branch 2 times, most recently from 5329a7d to aaa6a3f Compare September 18, 2026 08:29

laskoviymishka force-pushed the laskoviymishka/labels-spark branch from a812269 to 1d767a8 Compare September 18, 2026 08:29

laskoviymishka force-pushed the laskoviymishka/labels-fixture branch from aaa6a3f to 5f72886 Compare September 18, 2026 09:04

nastra reviewed Sep 18, 2026

View reviewed changes

Comment thread open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTServerCatalogAdapter.java Resolved

nastra approved these changes Sep 18, 2026

View reviewed changes

laskoviymishka force-pushed the laskoviymishka/labels-spark branch from cb995c5 to e2c01b6 Compare September 18, 2026 12:19

laskoviymishka force-pushed the laskoviymishka/labels-fixture branch 2 times, most recently from 7467a1f to c909e21 Compare September 18, 2026 12:37

laskoviymishka force-pushed the laskoviymishka/labels-spark branch from e2c01b6 to ef7aeda Compare September 18, 2026 12:37

laskoviymishka force-pushed the laskoviymishka/labels-fixture branch from c909e21 to 29123e6 Compare September 18, 2026 13:03

laskoviymishka force-pushed the laskoviymishka/labels-spark branch from ef7aeda to 769261b Compare September 18, 2026 13:03

laskoviymishka force-pushed the laskoviymishka/labels-fixture branch from 29123e6 to cf3dbb4 Compare September 18, 2026 13:26

laskoviymishka force-pushed the laskoviymishka/labels-spark branch from 769261b to 1148aa5 Compare September 18, 2026 13:26

laskoviymishka force-pushed the laskoviymishka/labels-fixture branch 2 times, most recently from 19854ad to 96db1f7 Compare September 22, 2026 17:07

This was referenced Sep 23, 2026

feat: support catalog labels on tables loaded from REST catalogs apache/iceberg-cpp#960

Merged

Catalog Labels: client-library read support for C++ apache/iceberg-cpp#938

Open

laskoviymishka force-pushed the laskoviymishka/labels-fixture branch from 96db1f7 to 8cd50b6 Compare September 30, 2026 20:20

laskoviymishka mentioned this pull request Oct 13, 2026

[feature](fe) Support iceberg REST catalog labels as the $labels table laskoviymishka/doris#1

Open

4 of 16 tasks

laskoviymishka force-pushed the laskoviymishka/labels-spark branch from d31b289 to dfc44fd Compare October 13, 2026 17:43

laskoviymishka force-pushed the laskoviymishka/labels-fixture branch from 8cd50b6 to a32f4af Compare October 13, 2026 17:43

laskoviymishka removed this pull request from stack #18051 October 13, 2026 17:44

laskoviymishka changed the base branch from laskoviymishka/labels-spark to main October 13, 2026 17:44

laskoviymishka and others added 2 commits October 13, 2026 19:26

OpenAPI: inline the label-name constants in the REST server fixture

cbdd89e

Inline the catalog-name / catalog-host / classification label keys at their single use sites, per review. Co-authored-by: Isaac <no-reply@databricks.com>

laskoviymishka force-pushed the laskoviymishka/labels-fixture branch from a32f4af to cbdd89e Compare October 13, 2026 18:26

laskoviymishka mentioned this pull request Oct 15, 2026

[feature](fe) Support iceberg REST catalog labels as the $labels table apache/doris#69135

Merged

5 of 16 tasks

Fokko mentioned this pull request Nov 9, 2026

Parse catalog labels in REST load table and load view responses apache/iceberg-python#4320

Merged

laskoviymishka merged commit 16c8a32 into main Nov 9, 2026
24 checks passed

laskoviymishka deleted the laskoviymishka/labels-fixture branch November 9, 2026 22:17

yayaha mentioned this pull request Nov 10, 2026

Infra: 'iceberg-rest-fixture' Docker image not published since 2026-04-29 #18877

Open

3 tasks

Navigation Menu

Repository navigation

OpenAPI: emit labels from the REST server test fixture - #18047

OpenAPI: emit labels from the REST server test fixture#18047
laskoviymishka merged 3 commits into
mainfrom
laskoviymishka/labels-fixture

laskoviymishka commented Sep 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

laskoviymishka commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

laskoviymishka commented Sep 10, 2026 •

edited

Loading