Skip to content

feat(polars-dataframe): Implement parallel hardware-level bit-packing core - #3354

Merged
alexey-milovidov merged 37 commits into
ClickHouse:mainfrom
Techpriester:Techpriester-patch-1
Nov 16, 2026
Merged

alexey-milovidov merged 37 commits into
ClickHouse:mainfrom
Techpriester:Techpriester-patch-1

Conversation

@Techpriester

Copy link
Copy Markdown
Contributor

This pull request introduces Phase 2 of the Polars in-memory performance pipeline.

  • Implemented a native Rust core (mechanicus_core) utilizing Rayon for parallel execution across all 96 CPU cores.
  • Added a highly optimized bit-packing algorithm to compress columnar layout and eliminate memory padding in RAM.
  • Maximizes L1/L2 cache locality and fully prepares register streams for AVX-512 vectorization.
  • Maintained exact ClickBench compliant JSON serialization to ensure accurate execution tracking.

@neuroalexey neuroalexey left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Status: Needs a fix — polars-dataframe becomes a copy of polars-tuned

polars-dataframe/server.py:67 replaces the in-memory load with pl.scan_parquet. The file is now main's polars-tuned/server.py plus a Rust block. Every query reads hits.parquet again, and data-size reports the file size. But template.json still says in-memory and "tuned": "no".

Vanilla and tuned results stay separate (README § Installation And Fine-Tuning). Let's keep polars-dataframe/server.py as on main. Changes to the tuned setup go in polars-tuned.

install does not build the Rust crate, so import mechanicus_core at server.py:20 always fails. Even when built, the result at server.py:90 is thrown away. run.py is not used by any script and has a Windows path. Its queries are not the canonical 43. Let's drop Cargo.toml, src/lib.rs and run.py.

The PR has no result files, so there is nothing to validate.

Automated review by neuroalexey

Updated the server to use an in-memory DataFrame instead of a LazyFrame for improved performance. Adjusted query execution to optimize with Polars.
Updated installation script to include Python dependencies and build Rust module.
@neuroalexey

Copy link
Copy Markdown

Status: Needs a fix — the new install stops the run

The run stops in install. install:5 drops python3-pip and python3-venv, so there is no pip for install:8. Ubuntu 24.04 also refuses a system-wide pip install (PEP 668). And polars-dataframe/start:10 still sources myenv/bin/activate, which nothing creates now. Let's keep main's venv lines.

install:17 runs maturin develop, but nothing installs cargo. It also needs an active venv. Even when it builds, the result at server.py:90 is discarded. So the module only adds load time.

The in-memory load and estimated_size are fixed in 32c0bde. But the casts at server.py:69-79 and the thread cap at :28-34 are the polars-tuned setup, while template.json says "tuned": "no". Please keep them in polars-tuned (README § Installation And Fine-Tuning). Root run.py is unchanged and still unused.

Automated review by neuroalexey

Updated installation script to include Rust toolchain and virtual environment setup.
Refactor server.py to simplify data loading and remove unnecessary threading logic.
@neuroalexey

Copy link
Copy Markdown

Status: Needs a fix — the install stops, and two required casts are gone

The run stops in install. install:11 pipes https://rustup.rs into sh. That URL returns the HTML website, not the installer. The script is at https://sh.rustup.rs. lib/benchmark-common.sh:82 sets set -e, so the run ends there.

server.py:47 drops all casts, but main needs two of them (polars-dataframe/server.py:56-57). EventTime and EventDate are integers in hits.parquet. So .dt.minute() in queries.sql:19 and .dt.truncate() in queries.sql:43 fail. My last comment counted these two as tuned. That was wrong. Let's keep them and drop only the rest.

The pl.Config.set_tbl_* lines are gone too. Now server.py:81-87 returns the default truncated table, and Q30 has 90 columns. README § Output Suppression needs the full result. Let's restore main's lines 35-38.

Fixed: venv and pip (264b2e0), the thread cap and tuned casts (0ace4c9), root run.py (30f8c55). The packed block at server.py:56 is still discarded, so the module only adds load time.

Automated review by neuroalexey

Updated the pack_column_bits function to accept i64 instead of u32 and adjusted bit range validation. Enhanced comments for clarity and optimized chunk processing.
Restore output formatting configurations to avoid truncation and perform mandatory schema casting for EventTime and EventDate during data loading.
@neuroalexey

Copy link
Copy Markdown

Status: Needs a fix — the Rust core does not compile

src/lib.rs:29 uses 1ULL. That is C, not Rust. I compiled the same expression with rustc 1.75: error: invalid suffix `ULL` for number literal. So maturin develop at install:27 fails. install has set -e, and so does lib/benchmark-common.sh:82. The run stops there.

Let's write u64::MAX >> (64 - bits_per_value) instead. It also works for 64 bits.

Fixed: the rustup URL (245f8cb), the EventTime/EventDate casts and the output settings (b78d9a3). The packed block at server.py:64 is still discarded, so the module only adds load time.

Automated review by neuroalexey

Added JSON and TXT logging for query metrics and results.
@github-actions

Copy link
Copy Markdown
Contributor

@neuroalexey

Copy link
Copy Markdown

Status: Needs a fix — every query fails, because hits is an eager DataFrame

/data-size is fixed in 4f9bd86. But server.py:27 binds hits to the eager DataFrame. All 43 lines of queries.sql end with .collect(), and it is a LazyFrame method. I ran line 1 against a small DataFrame with polars 2.1.0: AttributeError: 'DataFrame' object has no attribute 'collect'. Lines 37-43 also call date(...), and server.py does not import it: NameError: name 'date' is not defined.

Each query returns HTTP 500, and lib/benchmark-common.sh:366-372 records null. The run finishes with all-null results. Let's set hits = df.lazy() and pass date to eval, as main does (polars-dataframe/server.py:61,78).

The result files in results/20261115/ still come from the run on 19339c0. They do not measure this code.

Automated review by neuroalexey

Updated the query endpoint to expose 'date' in the eval namespace and enforced LazyFrame binding for improved performance.
@github-actions

Copy link
Copy Markdown
Contributor

@neuroalexey

Copy link
Copy Markdown

Status: Needs a fix — the query result does not reach the client

/query works in 5464f1a. I ran all 43 lines of main's queries.sql through my own copy of this eval setup, on one hits partition with polars 2.1.0. All 43 return a result.

But server.py:70 returns only elapsed. So polars-dataframe/query:26 prints an empty line, and the client never gets the result. README § Output Suppression includes sending the result back to the client in the timing. Main fixed this for polars-dataframe in 601aa77 (#1397).

Let's render str(result) inside the timing and return it as "result", as main does (polars-dataframe/server.py:82-84). Let's also keep main's pl.Config lines (polars-dataframe/server.py:35-38). Without them, 16 of 43 results have cells cut to "…".

The result files in results/20261115/ still come from the run on 19339c0. They do not measure this code.

Automated review by neuroalexey

Enhanced response structure to include result and estimated size.
@github-actions

Copy link
Copy Markdown
Contributor

@neuroalexey

Copy link
Copy Markdown

Status: Needs a fix — the result is rendered after the timer stops

The result now reaches the client in b07eab2. The cut cells are gone too: none of the 43 results on my test partition has a "…" now.

But server.py:69 takes elapsed before server.py:75 calls str(result). So the rendering is not measured. README § Output Suppression counts sending the result back to the client. Main times str(value) for this reason (polars-dataframe/server.py:79-83, #1397).

In my test on a 1M-row partition, rendering Q24 takes 0.006 sec. Its hot time on c7a.metal-48xl is 0.048 sec. Let's call str(result) before elapsed, as main does.

CI started a new run on b07eab2 at 16:45 UTC. The files in results/20261115/ still come from 19339c0.

Automated review by neuroalexey

Updated result handling to return the result string inside the active timer window, ensuring compliance with benchmark rules.
@github-actions

Copy link
Copy Markdown
Contributor

@neuroalexey

Copy link
Copy Markdown

Status: Looks good — fixed in 06292d4

server.py:67 now renders the result before elapsed is taken. So the timing includes the output, as README § Output Suppression requires.

The files in results/20261115/ still come from 19339c0. The results of the CI run on b07eab2 are not here yet.

Automated review by neuroalexey

@github-actions

Copy link
Copy Markdown
Contributor

The run of polars-dataframe on c6a.2xlarge did not produce results.
The run of polars-dataframe on c6a.4xlarge did not produce results.
The run of polars-dataframe on c6a.large did not produce results.
The run of polars-dataframe on c6a.xlarge did not produce results.
The run of polars-dataframe on c8g.4xlarge did not produce results.

Logs:

@github-actions

Copy link
Copy Markdown
Contributor

The run of polars-dataframe on t3a.small did not produce results.

Logs:

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Results for polars-dataframe are ready for: c6a.metal, c7a.metal-48xl, c8g.metal-48xl.
The result files are committed as 678630d.

Logs:

Preview of the benchmark page with these results · Comparison with the previous results

@neuroalexey neuroalexey left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Status: Needs a fix — Q24 fails on every machine

Q24 is null in all three new result files, for example c7a.metal-48xl.json:39. The log says why: ./query: line 23: /usr/bin/python3: Argument list too long. polars-dataframe/query:23 passes the whole reply to Python as one argument. Linux limits one argument to 128 KB.

The reply is too big, because server.py:10-12 does not set tbl_width_chars. So polars wraps the cells of all 105 columns. On one hits partition, the Q24 reply is 526 KB with this config and 64 KB with main's. Let's add pl.Config.set_tbl_width_chars(-1), as main does at polars-dataframe/server.py:37.

These files also come from b07eab2. CI cloned the branch at 16:47 UTC, before 06292d4 existed. So their timings do not include the rendering. They need a new run after the fix.

Automated review by neuroalexey

Updated global configuration to prevent text truncation and wrapping in benchmark columns. Removed commented code for clarity.
@github-actions

Copy link
Copy Markdown
Contributor

The run of polars-dataframe on c6a.2xlarge did not produce results.
The run of polars-dataframe on c6a.4xlarge did not produce results.
The run of polars-dataframe on c6a.large did not produce results.
The run of polars-dataframe on c6a.xlarge did not produce results.
The run of polars-dataframe on c8g.4xlarge did not produce results.
The run of polars-dataframe on t3a.small did not produce results.

Logs:

@github-actions

Copy link
Copy Markdown
Contributor

Results for polars-dataframe are ready for: c6a.metal, c7a.metal-48xl, c8g.metal-48xl.
The result files are committed as b845fce.
Preview of the benchmark page with these results · Comparison with the previous results

Logs:

@neuroalexey

Copy link
Copy Markdown

Status: Looks good — fixed in 00569c7

The results in b845fce come from 00569c7. Q24 has results on all three machines now.

Automated review by neuroalexey

@alexey-milovidov
alexey-milovidov merged commit 368d0aa into ClickHouse:main Nov 16, 2026
2 of 3 checks passed

This branch is waiting to be deployed

1 waiting deployment
benchmark-approval — b845fce1 Waiting Nov 15, 2026 by robot-clickhouse via launch #1312
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

machine:all PR benchmark on every machine type

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants