FIRST_VALUE, LAST_VALUE and NTH_VALUE build a ScalarValue for every row #26540

Is your feature request related to a problem or challenge?

In BoundedWindowAggExec, FIRST_VALUE, LAST_VALUE and NTH_VALUE produce one ScalarValue per output row. NthValueEvaluator::evaluate turns the selected row into a scalar with ScalarValue::try_from_array (nth_value.rs#L472), and StandardWindowExpr::evaluate_stateful collects the scalars (standard.rs#L213-L244) and turns them back into an array with ScalarValue::iter_to_array (#L252).

That is cheap for primitive values but not for nested ones. Each struct scalar is a one-row slice of every child array, and iter_to_array concatenates all of them again. In datafusion-cli, 1M rows in 64-row partitions, with a four-field struct:

SET datafusion.execution.target_partitions = 1;
CREATE TABLE s AS SELECT value / 64 AS k, value AS i, value AS v,
  named_struct('a', value, 'b', concat('algo-', value), 'c', concat('sub-', value), 'd', value % 7 = 0) AS st
FROM generate_series(1, 1000000);

SELECT count(x) FROM (SELECT last_value(st) OVER (PARTITION BY k ORDER BY i) AS x FROM s);

Describe the solution you'd like

These functions always return one of the frame's rows, or NULL. If the PartitionEvaluator could report the index of that row instead of a scalar, StandardWindowExpr could collect the indices and build each output column with one take. A prototype of this, a PartitionEvaluator method that returns the row index for LAST_VALUE and a gather in StandardWindowExpr when the evaluator supports it, made last_value over a five-field struct with 4 rows per partition 2.1x faster in Comet's window benchmark, with identical output.

Describe alternatives you've considered

A faster iter_to_array for struct scalars would remove the concatenation but keep a slice and an allocation per row.

Additional context

The timings are from datafusion-cli built from 3ee3e39 with the release-nonlto profile on an Apple M5 Max. Found through Apache DataFusion Comet, where Spark's last(struct(...), true) over sorted clickstream rows spends about half of its window time on these scalars.