Skip to content

fix(tdigest): merge into existing dest without OVERRIDE#3412

Open
Tangruilin wants to merge 4 commits intoapache:unstablefrom
Tangruilin:fix#3410
Open

fix(tdigest): merge into existing dest without OVERRIDE#3412
Tangruilin wants to merge 4 commits intoapache:unstablefrom
Tangruilin:fix#3410

Conversation

@Tangruilin
Copy link
Copy Markdown

WIP: fix TDIGEST.MERGE to match Redis behavior

Issue: #3410

When destination key already exists without OVERRIDE flag:

  • Redis: merge dest + sources together
  • Kvrocks (before): return error

Changes:

  1. Remove error return when dest exists without OVERRIDE
  2. Read dest centroids and add to merge list
  3. Add check to prevent duplicate merge if dest in sources
  4. Update tests to verify merge and override behavior

@Tangruilin Tangruilin marked this pull request as draft March 31, 2026 17:04
@Tangruilin Tangruilin marked this pull request as ready for review April 5, 2026 08:13
@Tangruilin
Copy link
Copy Markdown
Author

Tangruilin commented Apr 5, 2026

@jihuayu Hi, this PR is ready for review. Thanks!

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Aligns Kvrocks’ TDIGEST.MERGE semantics with Redis when the destination key already exists and OVERRIDE is not specified, by merging destination + sources instead of returning an error.

Changes:

  • Update TDigest::Merge to include the destination digest’s centroids in the merge set when dest exists without OVERRIDE.
  • Refactor centroid extraction logic into a shared helper (getCentroidsForMerge) to handle buffered/unbuffered digests consistently.
  • Extend Go unit tests to validate merging into an existing destination and OVERRIDE overwrite behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
tests/gocase/unit/type/tdigest/tdigest_test.go Updates merge tests to assert existing-dest merge semantics and override overwrite semantics.
src/types/redis_tdigest.h Declares a new helper for extracting centroids for merge operations.
src/types/redis_tdigest.cc Implements the new merge behavior and the centroid-extraction helper.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@Tangruilin
Copy link
Copy Markdown
Author

@jihuayu "I'm sorry the CI didn't pass.

However, it seems the CI format configuration is not using the default configuration. I ran ./x.py check format locally and it passed, but the CI command is ./x.py check format
--clang-format-path clang-format-18.

This can be confusing. Below is a screenshot of my local run. Perhaps we should keep the CI consistent with the default configuration and use configuration files instead of command-line
arguments."

image

@Tangruilin
Copy link
Copy Markdown
Author

Tangruilin commented Apr 5, 2026

@jihuayu
"The lint issue was introduced by MR #3419 It seems this MR was merged without passing the format check.

Should kvrocks consider enforcing a rule that blocks merging when CI doesn't pass, to avoid similar CI pollution issues?"

@Tangruilin
Copy link
Copy Markdown
Author

@jihuayu "The lint issue was introduced by MR #3419 It seems this MR was merged without passing the format check.

Should kvrocks consider enforcing a rule that blocks merging when CI doesn't pass, to avoid similar CI pollution issues?"

image

@Tangruilin Tangruilin force-pushed the fix#3410 branch 3 times, most recently from 089460d to f8ff985 Compare April 5, 2026 16:27
@jihuayu
Copy link
Copy Markdown
Member

jihuayu commented Apr 6, 2026

@Tangruilin Yes, you are right!

@Tangruilin Tangruilin force-pushed the fix#3410 branch 3 times, most recently from eec4eb0 to 0ffe467 Compare April 7, 2026 15:04
Copy link
Copy Markdown
Member

@LindaSummer LindaSummer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Tangruilin ,

Generally LGTM with some comments. 😊

Could we add some cpp unit tests for this change?

@Tangruilin
Copy link
Copy Markdown
Author

Hi @Tangruilin ,

Generally LGTM with some comments. 😊

Could we add some cpp unit tests for this change?

Sure. I will add cpp unit before merge

@jihuayu
Copy link
Copy Markdown
Member

jihuayu commented Apr 8, 2026

@Tangruilin It looks like the code incorrectly overrides the COMPRESSION parameter in the following scenario. Could you add more test cases for this?

TDIGEST.CREATE src COMPRESSION 200
TDIGEST.ADD src 1

TDIGEST.CREATE dest COMPRESSION 100
TDIGEST.ADD dest 2

TDIGEST.INFO dest
TDIGEST.MERGE dest 1 src
TDIGEST.INFO dest

The Redis results are as follows:

127.0.0.1:6379> TDIGEST.INFO dest
 1) Compression
 2) (integer) 100

@Tangruilin
Copy link
Copy Markdown
Author

@Tangruilin It looks like the code incorrectly overrides the COMPRESSION parameter in the following scenario. Could you add more test cases for this?

TDIGEST.CREATE src COMPRESSION 200
TDIGEST.ADD src 1

TDIGEST.CREATE dest COMPRESSION 100
TDIGEST.ADD dest 2

TDIGEST.INFO dest
TDIGEST.MERGE dest 1 src
TDIGEST.INFO dest

The Redis results are as follows:

127.0.0.1:6379> TDIGEST.INFO dest
 1) Compression
 2) (integer) 100

Done

WIP: fix TDIGEST.MERGE to match Redis behavior

Issue: apache#3410

When destination key already exists without OVERRIDE flag:
- Redis: merge dest + sources together
- Kvrocks (before): return error

Changes:
1. Remove error return when dest exists without OVERRIDE
2. Read dest centroids and add to merge list
3. Add check to prevent duplicate merge if dest in sources
4. Update tests to verify merge and override behavior
@jihuayu jihuayu requested review from LindaSummer and jihuayu April 12, 2026 11:29
@Tangruilin
Copy link
Copy Markdown
Author

@jihuayu ji
The CI seems to fail because of env

jihuayu
jihuayu previously approved these changes Apr 13, 2026
Copy link
Copy Markdown
Member

@jihuayu jihuayu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks OK.
However, the logic in this function is a bit complex, so I’ll need some help from other reviewers.

@sonarqubecloud
Copy link
Copy Markdown

LindaSummer
LindaSummer previously approved these changes Apr 14, 2026
Copy link
Copy Markdown
Member

@LindaSummer LindaSummer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Tangruilin ,

LGTM 🚀

With the compression parameter improvement, its behavior should match the Redis' now.

Thanks very much for your effort!

@LindaSummer LindaSummer enabled auto-merge (squash) April 14, 2026 15:41
@LindaSummer LindaSummer disabled auto-merge April 14, 2026 15:43
@LindaSummer LindaSummer enabled auto-merge (squash) April 14, 2026 15:43
Co-authored-by: Twice <twice@apache.org>
@PragmaTwice PragmaTwice dismissed stale reviews from LindaSummer and jihuayu via cecd0f0 April 14, 2026 15:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants