[ibrahim halatci (ihalatci)]

Propagate PostgreSQL 19 RC1 EXPLAIN IO to worker nodes.

Stack

Native stack layer 3: #8929 -> #8805 -> #8807

This PR is based on ihalatci-pg19-repack-lock-fix at
805849c8ed7b3fa5e77edf289c5e2b159b4bb529.

Changes

• Propagate ExplainState.io to worker EXPLAIN commands on PostgreSQL 19+.
• Preserve INSTRUMENT_IO through the saved-plan EXPLAIN ANALYZE path.
• Cover both saved-plan and live remote-EXPLAIN paths in pg19 regression tests.
• Keep PostgreSQL 17/18 behavior unchanged through existing version guards.

The diff relative to #8805 is limited to:

• src/backend/distributed/planner/multi_explain.c
• src/test/regress/bin/normalize.sed
• src/test/regress/sql/pg19.sql
• src/test/regress/expected/pg19.out

Validation

• PostgreSQL 19rc1 Citus configure, build, and install: passed.
• src/test/regress/citus_tests/run_test.py pg19: passed (7/7).
• src/test/regress/citus_tests/run_test.py multi_explain: passed (16/16).
• src/test/regress/citus_tests/run_test.py multi_mx_explain: passed (10/10).
• PostgreSQL 17.10 configure/build: passed.
• PostgreSQL 18.4 configure/build: passed.
• citus_indent --check: passed.
• ci/editorconfig.sh: passed.
• git diff --check: passed.
• make check-style: repository-baseline flake8 failures only; reproduced
  unchanged on PG19: use ShareUpdateExclusiveLock for REPACK CONCURRENTLY #8805. The changed C file and editorconfig checks pass.

Fixes #8770

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.82%. Comparing base (a1ba764) to head (436d684).

Additional details and impacted files

@@                        Coverage Diff                        @@
##           ihalatci-pg19-repack-lock-fix    #8807      +/-   ##
=================================================================
- Coverage                          88.82%   88.82%   -0.01%     
=================================================================
  Files                                290      290              
  Lines                              65495    65501       +6     
  Branches                            8261     8262       +1     
=================================================================
+ Hits                               58175    58178       +3     
- Misses                              4937     4939       +2     
- Partials                            2383     2384       +1     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ibrahim halatci (ihalatci)]

Beta 4 disposition: retained. Restack this PR directly onto current main, removing #8805/#8806 ancestry and keeping only the two required EXPLAIN IO propagation sites plus focused Beta 4 tests.

[Onur Tirtir (onurctirtir)]

approving with some suggestions

[Onur Tirtir (onurctirtir)]

This propagates IO for EXPLAIN (ANALYZE, IO) <query>. It does not work for EXPLAIN (ANALYZE, IO) EXECUTE <prepared>.

CurrentDistributedQueryExplainOptions is only filled in here, in the ExplainOneQuery_hook. PostgreSQL does not call that hook for EXPLAIN EXECUTE. So the static global keeps either the defaults or the previous EXPLAIN's options.

Repro on PG19rc1 with this PR. ps is PREPARE ps(int) AS SELECT count(*) FROM t WHERE b > $1 on a distributed table.

1. In a fresh session, run EXPLAIN (ANALYZE, IO) EXECUTE ps(0). The worker receives "io": false, and there is no Prefetch: line.
2. In another session, first run EXPLAIN (ANALYZE, IO, BUFFERS, VERBOSE) SELECT .... Then run EXPLAIN (ANALYZE, IO OFF, BUFFERS OFF, VERBOSE OFF) EXECUTE ps(0). The worker receives "verbose": true, "buffers": true, "io": true. The worker plan shows Prefetch:, Buffers:, and Output:, even though all three were turned off.

This is pre-existing. BUFFERS and VERBOSE already behave the same way on main. So I don't think it has to block this PR. It would be worth a separate issue, though, since IO now inherits the same problem. A possible fix is to capture the options from the ExplainStmt in the utility hook, so that EXECUTE gets them too.

A test that catches it. It currently returns t for the second call, and the correct answer is f:

PREPARE explain_io_ps(int) AS SELECT count(*) FROM explain_io WHERE b > $1;
SELECT explain_has_prefetch($$
	EXPLAIN (ANALYZE, IO, COSTS OFF, TIMING OFF, SUMMARY OFF)
	SELECT count(*) FROM explain_io $$);
-- no IO requested, so must be f
SELECT explain_has_prefetch($$
	EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) EXECUTE explain_io_ps(0) $$);

[Onur Tirtir (onurctirtir)]

This new sentence describes a bug as if it were the design.

For a subplan, every task is supposed to find its saved output. That is what #8017 added to fix #4212, so a subplan is not run twice. A task only falls through to a live EXPLAIN here when its task id in the re-plan differs from its task id in the first run. That is a bug. When it happens:

• one task prints another shard's plan, and
• another task runs the query a second time on the worker. The savepoint rolls it back.

The bug is pre-existing, because this PR does not touch ExtractAnalyzeStats(). Could you open an issue for it, and change this comment to call it a known bug with a link to that issue, instead of describing it as expected? For example:

	 * can be NULL for some cases of es->analyze == true.  A distributed subplan
	 * can also end up here when its re-planned task ids do not match the ids its
	 * saved output was stored under; that is a known bug, see #XXXX.

A repro for the issue. With 2 shards, row 6 goes to shard 8775001 (port 10702), and rows 7 and 8 go to shard 8775000 (port 10701):

SET citus.shard_count TO 2;
SET citus.next_shard_id TO 8775000;
CREATE TABLE t(a int, b int);
SELECT create_distributed_table('t', 'a');
SET citus.explain_all_tasks TO on;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF, BUFFERS OFF)
WITH c AS (INSERT INTO t VALUES (6,66),(7,77),(8,88) RETURNING *) SELECT * FROM c;

Actual output (PG19rc1):

   ->  Distributed Subplan 1_1
         ->  Custom Scan (Citus Adaptive) (actual rows=1.00 loops=1)
               ->  Task
                     Tuple data received from node: 8 bytes
                     Node: host=localhost port=10701 dbname=...
                     ->  Insert on t_8775001 citus_table_alias (actual rows=1.00 loops=1)
               ->  Task
                     Tuple data received from node: 0 bytes
                     Node: host=localhost port=10702 dbname=...
                     ->  Insert on t_8775001 citus_table_alias (actual rows=1.00 loops=1)

Expected:

• port 10701 shows Insert on t_8775000 with 2 rows.
• port 10702 shows its saved plan, not a live re-run ("0 bytes").
• the subplan reports 3 rows, not 1.

[Onur Tirtir (onurctirtir)]

This test's oracle depends on the task id mismatch bug described on the RemoteExplain comment.

The IO TRUE remote command only appears here because one INSERT task fails to find its saved output. It then falls back to a live remote EXPLAIN. If someone fixes that mismatch, this test's output changes, even though IO propagation would still be correct. The comment "a distributed subplan makes Citus build the remote EXPLAIN statement itself instead of reusing a plan saved by the worker" is not true in general either. Read-only subplans reuse the saved plan.

Suggestion: replace this test with a read-only subplan. It goes through the same subplan explain path, and it gives a real positive and negative oracle on the plan itself. I verified both on PG19rc1 with this PR. Each task shows Prefetch: avg=1.00 max=1 capacity=94.

SET citus.explain_all_tasks TO on;
-- t
SELECT explain_has_prefetch($$
	EXPLAIN (ANALYZE, IO, COSTS OFF, TIMING OFF, SUMMARY OFF)
	WITH cte_1 AS MATERIALIZED (SELECT * FROM explain_io_mod)
	SELECT count(*) FROM cte_1 $$);
-- f
SELECT explain_has_prefetch($$
	EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF)
	WITH cte_1 AS MATERIALIZED (SELECT * FROM explain_io_mod)
	SELECT count(*) FROM cte_1 $$);
RESET citus.explain_all_tasks;

Optionally, also cover a non-text format, since the worker uses different keys there. With 2 shards and explain_all_tasks on, this returns 2:

DO $d$
DECLARE j json; n int;
BEGIN
  EXECUTE 'EXPLAIN (ANALYZE, IO, COSTS OFF, TIMING OFF, SUMMARY OFF, FORMAT JSON) SELECT count(*) FROM explain_io_mod' INTO j;
  SELECT count(*) INTO n FROM jsonb_path_query(j::jsonb, 'strict $.**."Average Prefetch Distance"');
  RAISE NOTICE 'prefetch keys: %', n;
END $d$;

For the record, I also checked IO on these paths: multi-shard, router, reference table, local execution on the coordinator, XML, YAML, and IO combined with WAL, MEMORY, and SERIALIZE. All of them work.