[CASSANDRA-21138] Add autorepair to Cassandra 5.0 by pauloricardomg · Pull Request #4558 · apache/cassandra

pauloricardomg · 2026-01-12T16:23:40Z

Thanks for sending a pull request! Here are some tips if you're new here:

Ensure you have added or run the appropriate tests for your PR.
Be sure to keep the PR description updated to reflect all changes.
Write your PR title to summarize what this PR proposes.
If possible, provide a concise example to reproduce the issue for a faster review.
Read our contributor guidelines
If you're making a documentation change, see our guide to documentation contribution

Commit messages should follow the following format:

<One sentence description, usually Jira title or CHANGES.txt summary>

<Optional lengthier description (context on patch)>

patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####

Co-authored-by: Name1 <email1>
Co-authored-by: Name2 <email2>

The Cassandra Jira

pauloricardomg · 2026-01-28T15:53:36Z

.build/run-tests.sh

-    for branch in cassandra-4.0 cassandra-4.1 cassandra-5.0 ; do
+    # Note: cassandra-5.0.6 tag is used instead of cassandra-5.0 branch to enable
+    # testing upgrades from 5.0.6 to the current local build for autorepair feature
+    for branch in cassandra-4.0 cassandra-4.1 cassandra-5.0.6 ; do


The cassandra-5.0 dtest jar was unused since we don't do tests upgrading from the current version to the current version so I modified this to build the cassandra-5.0.6 dtest jar where autorepair is not included to test a minor version upgrade on AutoRepairDisabledSchemaUpgradeTest.

pauloricardomg · 2026-01-28T15:53:57Z

.build/run-tests.sh

      echo "Reusing ${TMP_DIR}/cassandra-dtest-jars for past branch dtest jars"
      if [ "x" == "x${OFFLINE}" ] ; then
-        until git -C ${TMP_DIR}/cassandra-dtest-jars fetch --quiet origin ; do echo "git -C ${TMP_DIR}/cassandra-dtest-jars fetch failed… trying again… " ; done
+        until git -C ${TMP_DIR}/cassandra-dtest-jars fetch --quiet --tags origin ; do echo "git -C ${TMP_DIR}/cassandra-dtest-jars fetch failed… trying again… " ; done


the --tags is needed because cassandra-5.0.6 is a tag and not a branch.

pauloricardomg · 2026-01-28T17:46:06Z

src/java/org/apache/cassandra/repair/autorepair/AutoRepairUtils.java

+     * Versions at or below this version (5.0.6) do not support auto-repair.
+     */
+    @VisibleForTesting
+    static final CassandraVersion LAST_UNSUPPORTED_VERSION_FOR_AUTO_REPAIR = new CassandraVersion("5.0.6");


This needs to be updated to the latest version before the release version of autorepair in 5.0. The reason for this is because we want the tests to pass if it's running 5.0.7-SNAPSHOT which is below 5.0.7 but above 5.0.6.

pauloricardomg · 2026-01-28T20:13:16Z

src/java/org/apache/cassandra/repair/autorepair/AutoRepairUtils.java

+            if (currentRepairStatus == null || parallelRepairNumber > currentRepairStatus.hostIdsWithOnGoingRepair.size())
+            {
+                // more repairs can be run, I might be the new one
+                if (autoRepairHistories != null)


@jaydeepkumar1984 do you recall why do we re-fetch the repair history here? In what circumstances can autoRepairHistories can be null, is this just on an uninitialized system ? I am wondering if this could cause a race where multiple nodes will attempt to run repair more than the allowed parallelNumber. Should we not perform this re-fetch outside the if parallelRepairNumber > currentRepairStatus.hostIdsWithOnGoingRepair.size() ?

I do not recall any specific reason, so this seems unnecessary. We can remove ti from here and refresh the autoRepairHistories above in line no. 876. which should be sufficient.
CurrentRepairStatus currentRepairStatus = getCurrentRepairStatus(repairType, getAutoRepairHistory(repairType), myId);

updated on 13f0b29 let me know if this is what you had in mind

jaydeepkumar1984 · 2026-02-09T00:53:36Z

src/java/org/apache/cassandra/config/DatabaseDescriptor.java

+    }
+
+    @VisibleForTesting
+    public static void setInitialTokens(String initial_token)


unused code

jaydeepkumar1984 · 2026-02-09T00:55:54Z

src/java/org/apache/cassandra/db/commitlog/CommitLog.java

        logger.info("Updated CDC block_writes from {} to {}", oldVal, val);
    }

-


nit: unwanted change

jaydeepkumar1984 · 2026-02-09T01:06:57Z

src/java/org/apache/cassandra/schema/SchemaKeyspace.java

        if (row.has("incremental_backups"))
            builder.incrementalBackups(row.getBoolean("incremental_backups"));

+        // auto_repair column was introduced in 5.1


nit: "auto_repair column was introduced in 5.1" -> "auto_repair column was introduced in 5.0.7"

jaydeepkumar1984 · 2026-02-09T01:46:00Z

src/java/org/apache/cassandra/utils/FBUtilities.java

 import com.google.common.collect.ImmutableList;
 import com.vdurmont.semver4j.Semver;
 import com.vdurmont.semver4j.SemverException;
+


nit: unwanted code change

jaydeepkumar1984 · 2026-02-09T01:54:05Z

...istributed/org/apache/cassandra/distributed/upgrade/AutoRepairDisabledSchemaUpgradeTest.java

+            Object[][] result1 = cluster.coordinator(1).execute("SELECT * FROM " + KEYSPACE + ".tbl WHERE pk = 1", ConsistencyLevel.ALL);
+            Object[][] result2 = cluster.coordinator(2).execute("SELECT * FROM " + KEYSPACE + ".tbl WHERE pk = 1", ConsistencyLevel.ALL);
+            assertEquals("Data should be readable from node 1", 1, result1.length);
+            assertEquals("Data should be readable from node 2", 1, result2.length);


Let's extend the test case here by downgrading node1 to 5.0.6, so that would also complete the rollback scenario.

jaydeepkumar1984 · 2026-02-09T02:01:35Z

Overall, the changes look great. A few nits and a comment in the test case. Please take a look at it.

Kokokokoka · 2026-02-12T15:36:22Z

Hello. can an option be added to skip TWCS tables automatically?
Like reaper does.
P.S. Thanks for an awesome change!

pauloricardomg · 2026-02-13T22:55:35Z

@Kokokokoka can you create a new issue requesting this feature? https://issues.apache.org/jira/secure/CreateIssue!default.jspa

pauloricardomg · 2026-02-13T23:38:00Z

@jaydeepkumar1984 addressed review comments on d7eb86f

jaydeepkumar1984 · 2026-02-14T01:30:00Z

@jaydeepkumar1984 addressed review comments on d7eb86f

Thanks. I will do one more round next week and then likely approve it.

src/java/org/apache/cassandra/repair/autorepair/AutoRepairUtils.java

src/java/org/apache/cassandra/cql3/statements/schema/TableAttributes.java

src/java/org/apache/cassandra/schema/SystemDistributedKeyspace.java

test/distributed/org/apache/cassandra/distributed/test/repair/AutoRepairTablePropertyDTest.java

conf/cassandra.yaml

src/java/org/apache/cassandra/config/Config.java

doc/modules/cassandra/pages/managing/operating/auto_repair.adoc

src/java/org/apache/cassandra/service/AutoRepairService.java

jaydeepkumar1984 · 2026-03-01T02:30:14Z

src/java/org/apache/cassandra/schema/SystemDistributedKeyspace.java

+     * gen 7: add auto_repair_history and auto_repair_priority tables for AutoRepair feature
     */
-    public static final long GENERATION = 6;
+    public static final long GENERATION = 7;


Should we also then maintain the same version if CassandraRelevantProperties.AUTOREPAIR_ENABLE.getBoolean() is false?

Good catch, fixed on 81d9547

was sort of worried about this, but I see that 7 is whats used for trunk for AutoRepair anyways, and has since advanced to 8 for zstd dictionary compression, so this shouldn't be a problem.

tolbertam · 2026-03-06T21:21:08Z

Thanks for making the suggested changes! I will review this weekend

tolbertam

This is looking great, played with this a little bit this weekend and it worked well.

I ran through all tests except the python upgrade dtests (which i ran into some configuration issues for) and everything worked except some AutoRepair dtests were missing the property to indicate auto repair is enabled (added comment for that). Other than that, one small cassandra.yaml issue which also exists on trunk that would be nice to fix here.

Did hit a couple known repeated flakes (https://issues.apache.org/jira/browse/CASSANDRA-20933, https://issues.apache.org/jira/browse/CASSANDRA-18098), but everything else appears to have run well.

Kicked off upgrade dtests overnight and will share how that went tomorrow. Assuming the property gets added for the auto repair dtests, i should be +1 on this. I'm wondering if we should just do a last formal vote for this before merging in assuming Jaydeep also +1s?

tolbertam · 2026-03-08T17:29:42Z

conf/cassandra.yaml

+# repair as anti-compaction during repair may contribute to additional space temporarily.
+# For example, setting this to 0.2 means at least 20% of disk must be unused.
+# Set to 0.0 to disable this check. Defaults to 0.0 (disabled) on 5.0 for backward-compatibility.
+# repair_disk_headroom_reject_ratio: 0.0;


Haven't noticed before but as it seems the case on trunk as well, but the semicolon seems extraneous here:

Suggested change

# repair_disk_headroom_reject_ratio: 0.0;

# repair_disk_headroom_reject_ratio: 0.0

reminder on this tiny one ^

...ributed/test/repair/AutoRepairSchedulerDisallowParallelReplicaRepairAcrossSchedulesTest.java

jaydeepkumar1984 · 2026-03-09T03:50:01Z

Thanks a lot @pauloricardomg for incorporating the review comments. I have also just reviewed the updated PR.
I am ready to +1.
Regarding the vote, I think we already had a consensus on this, and Paulo recently fyi'd the dev list. I think it should be ok as long as two of us officially +1 the PR.

tolbertam · 2026-03-09T13:42:03Z

sounds good 👍. I'm having trouble running the upgrade python-dtests but everything else looking good other than the issue with the dtests missing the system property that I mentioned in the review. I think assuming that gets fixed up we should be all set. A final rebase would be good too, there's one new unit test failing (CompactionTaskTest#testFullyExpiredSSTablesAreNotReleasedPrematurely) that was added on 5.0 that addresses a fixed issue on cassandra-5.0 that this branch doesn't have yet. It'd be great to get a clean test run (other than known flakes) if we can.

pauloricardomg · 2026-03-13T22:11:37Z

rebased and added missing system property to dtests on f8b14b0

5.0 pre-CI

There's also #4644 that adds an upgrade test to trunk that check that it's possible to run repair after upgrade from 5.0

Trunk pre-CI (should fail before the 5.0 version is merged)

pauloricardomg · 2026-03-13T22:14:18Z

Once you are good with the changes and CI results can you +1 on https://issues.apache.org/jira/browse/CASSANDRA-21138 ? Thanks

tolbertam · 2026-03-13T22:18:49Z

taking a look now! I think only thing I noticed not addressed was just a tiny suggestion in cassandra.yaml (#4558 (comment)). I'm kicking off test suite now and will finish reviewing and if all looks good i'll give my +1!

tolbertam · 2026-03-14T00:58:50Z

tests ran clean and everything looks great, +1 the PR and will also add comment to the JIRA.

Kokokokoka · 2026-03-22T19:23:09Z

@Kokokokoka can you create a new issue requesting this feature? https://issues.apache.org/jira/secure/CreateIssue!default.jspa

https://issues.apache.org/jira/browse/CASSANDRA-21244

pauloricardomg · 2026-03-23T20:13:09Z

@jaydeepkumar1984 @tolbertam talking with other developers (mck, smiklosovic) we decided to ship this in 5.0.8 as a standalone release, so now that 5.0.7 is out I rebased this and updated the version check and references to 5.0.8(2d18e4b). @tolbertam would you mind taking a quick look and resubmitting CI again?

Also created #4644 to add a new trunk test to validate upgrade from 5.0.X if you can take a look and approve.

tolbertam · 2026-03-24T01:06:51Z

sounds good, will kick off a new CI build and will also try to review 4644 as well!

tolbertam · 2026-03-24T01:09:23Z

.build/run-tests.sh

-    for branch in cassandra-4.0 cassandra-4.1 cassandra-5.0 ; do
+    # Note: cassandra-5.0.6 tag is used instead of cassandra-5.0 branch to enable
+    # testing upgrades from 5.0.6 to the current local build for autorepair feature
+    for branch in cassandra-4.0 cassandra-4.1 cassandra-5.0.6 ; do


nit: can we change this to cassandra-5.0.7 now?

tolbertam · 2026-03-24T01:14:58Z

...istributed/org/apache/cassandra/distributed/upgrade/AutoRepairDisabledSchemaUpgradeTest.java

+ */
+public class AutoRepairDisabledSchemaUpgradeTest extends UpgradeTestBase
+{
+    private static final Semver v506 = new Semver("5.0.6", SemverType.STRICT);


Suggested change

private static final Semver v506 = new Semver("5.0.6", SemverType.STRICT);

private static final Semver v506 = new Semver("5.0.7", SemverType.STRICT);

tolbertam · 2026-03-24T01:15:23Z

...istributed/org/apache/cassandra/distributed/upgrade/AutoRepairDisabledSchemaUpgradeTest.java

+        Version from = versions.get(v506);
+        Version to = AbstractCluster.CURRENT_VERSION;
+
+        assertNotNull("5.0.6 version not available - ensure dtest-5.0.6.jar is built", from);


Suggested change

assertNotNull("5.0.6 version not available - ensure dtest-5.0.6.jar is built", from);

assertNotNull("5.0.7 version not available - ensure dtest-5.0.7.jar is built", from);

tolbertam · 2026-03-24T01:21:02Z

...istributed/org/apache/cassandra/distributed/upgrade/AutoRepairDisabledSchemaUpgradeTest.java

+import static org.junit.Assert.assertNotNull;
+
+/**
+ * Tests that upgrading from 5.0.6 (where auto-repair is not included) to the current version


Suggested change

* Tests that upgrading from 5.0.6 (where auto-repair is not included) to the current version

* Tests that upgrading from 5.0.7 (where auto-repair is not included) to the current version

I guess technically this test is still valid now that i think about it, but would be good to make it to the most proximate to the the version where autorepair is introduced.

tolbertam · 2026-03-24T15:06:35Z

I attached CI results to the JIRA everything came back good. I see a change got committed overnight, hopefully that doesn't impact plans to do a standalone release!

Includes bug fixes and features: - Improved observability in AutoRepair (CASSANDRA-20581) - Stop repair scheduler if two major versions detected (CASSANDRA-20048) - Safeguard Full repair against disk protection (CASSANDRA-20045) - Stop AutoRepair monitoring thread upon shutdown (CASSANDRA-20623) - Fix race condition in auto-repair scheduler (CASSANDRA-20265) - Minimum repair task duration setting (CASSANDRA-20160) - Preview_repaired auto-repair type (CASSANDRA-20046) - Gate auto-repair behind cassandra.autorepair.enable JVM property - Add cassandra.autorepair.check_min_version to gate minimum version enforcement - Prevent auto-repair from running if any node is below 5.0.7 - Make system_distributed auto-repair schema conditional on feature being enabled - Add user-friendly errors for disabled auto-repair and schema incompatibility - Disable repair_disk_headroom_reject_ratio by default and remove deprecated references patch by Paulo Motta; reviewed by Jaydeepkumar Chovatia for CASSANDRA-21138 Co-Authored-By: Andy Tolbert <andy_tolbert@apple.com> Co-Authored-By: Chris Lohfink <clohfink@netflix.com> Co-Authored-By: Francisco Guerrero <frankgh@apache.org> Co-Authored-By: Himanshu Jindal <himanshj@amazon.com> Co-Authored-By: Jaydeepkumar Chovatia <jchovati@uber.com> Co-Authored-By: Kristijonas Zalys <kzalys@uber.com> Co-Authored-By: jaydeepkumar1984 <chovatia.jaydeep@gmail.com>

pauloricardomg · 2026-03-24T19:43:38Z

good catch @tolbertam addressed remaining comments on b514b54 - rebased and submitted a new CI run

I will commit both after you have a chance to check #4644

tolbertam · 2026-03-28T17:13:12Z

Just approved #4644, thanks @pauloricardomg ! I kicked off another round of CI on this branch for good measure, will upload the results later this afternoon.

pauloricardomg changed the base branch from trunk to cassandra-5.0 January 12, 2026 16:24

pauloricardomg force-pushed the autorepair_5_0_7 branch 5 times, most recently from 00f48df to 29b0090 Compare January 27, 2026 21:57

pauloricardomg changed the title ~~[JIRA PENDING] Add autorepair to Cassandra 5.0~~ [CASSANDRA-21138] Add autorepair to Cassandra 5.0 Jan 28, 2026

pauloricardomg commented Jan 28, 2026

View reviewed changes

pauloricardomg force-pushed the autorepair_5_0_7 branch from 29b0090 to b866027 Compare January 28, 2026 16:46

pauloricardomg commented Jan 28, 2026

View reviewed changes

pauloricardomg force-pushed the autorepair_5_0_7 branch from b866027 to 13f0b29 Compare January 30, 2026 21:56

tolbertam self-requested a review January 30, 2026 23:26

jaydeepkumar1984 reviewed Feb 9, 2026

View reviewed changes

pauloricardomg force-pushed the autorepair_5_0_7 branch from 13f0b29 to d7eb86f Compare February 13, 2026 23:36