Add SQL based benchmarking harness, port tpch to use framework by Omega359 · Pull Request #21707 · apache/datafusion

Omega359 · 2026-04-17T19:05:44Z

Which issue does this PR close?

part of [EPIC] Add SQL based benchmarking #21706

Rationale for this change

Add a sql based benchmark framework with tpch as the initial benchmark to use this new framework. The README.md includes notes about other benchmarks which will have individual PR's after the initial work is accepted.

What changes are included in this PR?

benchmarking code only.

Are these changes tested?

Yes

Are there any user-facing changes?

benchmarks/bench.sh now uses the new framework for benchmarking tpch

Additional info

AI assisted with refactoring and writing tests. I have reviewed all AI produced code.

…k to use this new framework. The README.md includes notes about other benchmarks which will be pushed after the initial work is accepted.

Omega359 · 2026-04-18T16:51:14Z

Moving back to draft as there are a number of improvements I want to make

…r refactoring and writing the tests and I have reviewed all changes.

Omega359 · 2026-04-18T21:36:39Z

This should now be ready for review.

Omega359 · 2026-04-18T21:38:06Z

+The sql benchmarks are organized in sub‑directories that correspond to the benchmark suites that are commonly used
+in the community:
+
+| Benchmark Suite       | Description                                                        |


This readme covers all the test suites that I've converted for which I'll have additional PR's for each if this PR is merged.

adriangb · 2026-04-18T21:38:43Z

I will review this coming week 😄

…ven if not in validate mode.

…ution in benchmark files - ${VARIABLE:-default|true value|false value}

adriangb

Generally looks great and is certainly a move in the right direction if we want to be able to run under Codspeed!

Since this doesn't break the current benchmarking setup the only cost to merging this and changing later is code churn, which is low.

My one blocking concern with this is the env vars. I think we should be able to / only support setting them via DATAFUSION_EXECUTION_TARGET_PARTITIONS=1 cargo bench ... and such.

adriangb · 2026-04-21T14:39:14Z

+| PARTITIONS                      | Number of partitions to process in parallel. Defaults to number of available cores.                                                                                                               |
+| BATCH_SIZE                      | Batch size when reading CSV or Parquet files.                                                                                                                                                     |
+| MEM_POOL_TYPE                   | The memory pool type to use, should be one of "fair" or "greedy".                                                                                                                                 |
+| MEMORY_LIMIT                    | Memory limit (e.g. '100M', '1.5G'). If not specified, run all pre-defined memory limits for given query if there's any, otherwise run with no memory limit.                                       |
+| DATAFUSION_RUNTIME_MEMORY_LIMIT | Used if MEMORY_LIMIT is not set.                                                                                                                                                                  |
+| SORT_SPILL_RESERVATION_BYTES    | The amount of memory to reserve for sort spill operations. DataFusion's default value will be used if not specified.                                                                              |


Why are these special? Don't we already have a pattern for setting this via env vars e.g. with DataFusion CLI that we can honor?

Those are basically what bench.sh already supported iirc ... just documented. Maybe not 'special' but not directly related to the sql benchmarks.

Do these env vars actually work? Maybe I'm missing something but I don't see how they get picked up.

It uses the existing code in options.rs via make_ctx() -> args.options.config()? -> self.update_config(config)

It seems datafusion configurations can already get passed into benchmark runs:

(from ./bench.sh --help) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Supported Configuration (Environment Variables) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ DATA_DIR directory to store datasets ... DATAFUSION_* Set the given datafusion configuration

I think the existing benchmark runner configurations that duplicate DataFusion core configurations are redundant. We could instead document the relevant config names and link to the configuration reference page. This might be a good cleanup item during the benchmark suite migration.

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

alamb · 2026-04-22T00:49:28Z

I ran

./benchmarks/bench.sh run tpch

And it seems to work well!

alamb · 2026-04-22T00:49:32Z

run benchmark tpch

adriangbot · 2026-04-22T00:52:52Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4292804307-1728-6qvbk 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing new_sql_benchmark (238d0a6) to a311d14 (merge-base) diff using: tpch
Results will be posted here when complete

File an issue against this benchmark runner

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

alamb · 2026-04-22T14:09:43Z

Thank you for all the work on this @Omega359 and @adriangb

2010YOUY01

Thank you! This looks great.

For future work, I suggest first cleaning up the TPC-H benchmark suite before migrating any other benchmarks.

It would be great to have a single entry point for SQL benchmarks that’s easy to discover and use. A couple of ideas toward that goal:

Move user-facing documentation to benchmarks/README.md, and keep benchmarks/sql_benchmarks/README.md focused on internal runner details
Support a local config file (e.g., toml) with env vars as overrides. I feel managing many env vars locally is cumbersome.

2010YOUY01 · 2026-04-23T07:15:53Z

+
+load sql_benchmarks/tpch/init/load_${TPCH_FILE_TYPE:-parquet}.sql
+
+assert I


The assertion logic is implemented inside sql_benchmark.rs, perhaps we could leverage sqlogictest runner instead.

Interesting idea. Followup PR possibly. The assertion code here doesn't handle types at all - only strings whereas the sqllogictests are obviously a bit more advanced. It's mostly meant to validate that a table was loaded since create external table ... doesn't fail if the file being loaded doesn't exist. That was kinda a WTF moment for me.

2010YOUY01 · 2026-04-23T07:28:14Z

+| PARTITIONS                      | Number of partitions to process in parallel. Defaults to number of available cores.                                                                                                               |
+| BATCH_SIZE                      | Batch size when reading CSV or Parquet files.                                                                                                                                                     |
+| MEM_POOL_TYPE                   | The memory pool type to use, should be one of "fair" or "greedy".                                                                                                                                 |
+| MEMORY_LIMIT                    | Memory limit (e.g. '100M', '1.5G'). If not specified, run all pre-defined memory limits for given query if there's any, otherwise run with no memory limit.                                       |
+| DATAFUSION_RUNTIME_MEMORY_LIMIT | Used if MEMORY_LIMIT is not set.                                                                                                                                                                  |
+| SORT_SPILL_RESERVATION_BYTES    | The amount of memory to reserve for sort spill operations. DataFusion's default value will be used if not specified.                                                                              |


It seems datafusion configurations can already get passed into benchmark runs:

(from ./bench.sh --help) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Supported Configuration (Environment Variables) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ DATA_DIR directory to store datasets ... DATAFUSION_* Set the given datafusion configuration

I think the existing benchmark runner configurations that duplicate DataFusion core configurations are redundant. We could instead document the relevant config names and link to the configuration reference page. This might be a good cleanup item during the benchmark suite migration.

…l_benchmark

… an issue with parsing multiple query statements when the queries were inline in the benchmark file (vs external in another file).

Omega359 · 2026-04-25T03:28:46Z

I've gone ahead and inlined the template file and the query into each of the tpch benchmark files and update the code to allow for multiple statements (previously it was only allowed for external files). Please let me know if this new approach is what is desired.

@alamb @martin-g @adriangb @2010YOUY01

# Conflicts: # Cargo.lock

alamb · 2026-04-28T14:06:16Z

I plan to review this later today

alamb · 2026-04-28T21:30:40Z

I ran out of time today (it is hard to find time for 5K lines!) -- but it is on my list for tomorrow

alamb

I think it looks good -- thank you @Omega359 -- let's merge this one in and then iterate in follow on PR

alamb · 2026-04-29T18:48:04Z

+
+init sql_benchmarks/tpch/init/set_config.sql
+
+load sql_benchmarks/tpch/init/load_${TPCH_FILE_TYPE:-parquet}.sql


I like the new format

adriangb · 2026-04-29T18:53:06Z

Wohoo!!

alamb · 2026-04-29T18:55:44Z

I am also trying to capture additional follow on tasks as a list on

[EPIC] Add SQL based benchmarking #21706

Please feel free to add your own suggestions

adriangb · 2026-04-29T23:46:33Z

I haven’t investigated yet if this change (possibly by revealing pre existing bugs) or the hacky harness is to blame blame but I’m seeing some benches fail now: #21806 (comment)

Upstream apache/datafusion#21707 ported `bench.sh run tpch` to a new Criterion-based SQL harness. The new harness reads parquet from a path relative to ${DATAFUSION_DIR}/benchmarks (where data isn't generated in our layout) and writes timings to target/criterion/, which our `bench.sh compare_detail` step doesn't understand. Recent benchmarks were failing because lineitem resolved to an empty external table. The dfbench tpch subcommand still exists upstream, so for the four tpch variants in our allowlist (tpch, tpch10, tpch_mem, tpch_mem10) invoke the prebuilt dfbench binary directly with the same arguments the old run_tpch used and write JSON to where compare_detail expects it. Other benchmarks still go through bench.sh.

Omega359 added 2 commits April 17, 2026 15:02

Add a sql based benchmark framework with tpch as the initial benchmar…

5ef261f

…k to use this new framework. The README.md includes notes about other benchmarks which will be pushed after the initial work is accepted.

Merge remote-tracking branch 'upstream/main' into new_sql_benchmark

dd498c2

Omega359 marked this pull request as ready for review April 17, 2026 19:37

Omega359 marked this pull request as draft April 18, 2026 16:50

Code cleanup, improvements and added tests. AI assistance was used fo…

1561e99

…r refactoring and writing the tests and I have reviewed all changes.

Omega359 marked this pull request as ready for review April 18, 2026 21:32

Omega359 mentioned this pull request Apr 18, 2026

[EPIC] Benchmark improvements #21165

Open

Merge branch 'main' into new_sql_benchmark

74e3b16

Omega359 commented Apr 18, 2026

View reviewed changes

Omega359 added 3 commits April 20, 2026 11:12

Fixed an issue with missing result files causing benchmarks to fail e…

f847168

…ven if not in validate mode.

Merge remote-tracking branch 'upstream/main' into new_sql_benchmark

dca99d7

Added support for defaults for true/false version of variable substit…

00a34ba

…ution in benchmark files - ${VARIABLE:-default|true value|false value}

Omega359 marked this pull request as draft April 20, 2026 16:20

Omega359 marked this pull request as ready for review April 20, 2026 16:20

Merge branch 'main' into new_sql_benchmark

caf83c3

martin-g reviewed Apr 21, 2026

View reviewed changes

adriangb approved these changes Apr 21, 2026

View reviewed changes

Update benchmarks/sql_benchmarks/tpch/queries/q14.sql

238d0a6

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

alamb mentioned this pull request Apr 21, 2026

fix: improve sort pushdown benchmark data and add DESC LIMIT queries #21711

Merged

alamb changed the title ~~Add SQL based benchmarking~~ Add SQL based benchmarking harness, port tpch to use framework Apr 22, 2026

alamb added the performance Make DataFusion faster label Apr 22, 2026

Omega359 and others added 3 commits April 21, 2026 22:20

Update benchmarks/sql_benchmarks/tpch/queries/q06.sql

0b7c546

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

Update benchmarks/sql_benchmarks/tpch/queries/q01.sql

7bc76dd

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

Update benchmarks/sql_benchmarks/tpch/queries/q04.sql

939895f

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

Omega359 added 2 commits April 22, 2026 17:51

Merge branch 'main' into new_sql_benchmark

aae13b2

Refactored the README.md to address some of the PR feedback.

7b4b93d

2010YOUY01 approved these changes Apr 23, 2026

View reviewed changes

martin-g reviewed Apr 23, 2026

View reviewed changes

Comment thread benchmarks/benches/sql.rs

Comment thread benchmarks/benches/sql.rs Outdated

Comment thread benchmarks/sql_benchmarks/tpch/init/load_csv.sql Outdated

Comment thread benchmarks/benches/sql.rs Outdated

Omega359 added 8 commits April 23, 2026 10:47

Merge branch 'main' into new_sql_benchmark

b107b2f

Enabled env config for options.

e48430e

Removed config that duplicates things that can be set via sessionconfig.

6d4b7df

Updates from PR feedback.

3e928f3

Fix issue with order vs orders.

6b876ce

Drop tables after each benchmark for tpch_mem.

9ca3a79

Merge remote-tracking branch 'refs/remotes/upstream/main' into new_sq…

f16af4e

…l_benchmark

Inline template file and queries into the tpch benchmark files. Fixed…

3e64d65

… an issue with parsing multiple query statements when the queries were inline in the benchmark file (vs external in another file).

Omega359 added 2 commits April 27, 2026 22:18

Merge remote-tracking branch 'upstream/main' into new_sql_benchmark

ffc344c

# Conflicts: # Cargo.lock

Fixed merge conflict with upstream.

6031057

alamb approved these changes Apr 29, 2026

View reviewed changes

alamb mentioned this pull request Apr 29, 2026

[EPIC] Add SQL based benchmarking #21706

Open

15 tasks

alamb added this pull request to the merge queue Apr 29, 2026

Merged via the queue into apache:main with commit 2b95cde Apr 29, 2026
35 checks passed

alamb mentioned this pull request Apr 29, 2026

Conslidate benchmark runner documentation #21935

Open

This was referenced Apr 29, 2026

Consolidate benchmarking options/ENV settings #21936

Open

CLI (non criterion) interface for sql runner #21937

Open


		load sql_benchmarks/tpch/init/load_${TPCH_FILE_TYPE:-parquet}.sql

		assert I


		init sql_benchmarks/tpch/init/set_config.sql

		load sql_benchmarks/tpch/init/load_${TPCH_FILE_TYPE:-parquet}.sql

Conversation

Omega359 commented Apr 17, 2026 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Additional info

Uh oh!

Omega359 commented Apr 18, 2026

Uh oh!

Omega359 commented Apr 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriangb commented Apr 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adriangb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Omega359 Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb commented Apr 22, 2026

Uh oh!

alamb commented Apr 22, 2026

Uh oh!

adriangbot commented Apr 22, 2026

Uh oh!

alamb commented Apr 22, 2026

Uh oh!

2010YOUY01 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Omega359 commented Apr 25, 2026

Uh oh!

alamb commented Apr 28, 2026

Uh oh!

alamb commented Apr 28, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Omega359 commented Apr 17, 2026 •

edited by alamb

Loading

Omega359 Apr 22, 2026 •

edited

Loading

2010YOUY01 left a comment •

edited

Loading