Skip to content

KAFKA-20395: Support unregistering controllers - #22191

Merged
junrao merged 39 commits into
apache:trunkfrom
kevin-wu24:KAFKA-20395
Aug 3, 2026
Merged

KAFKA-20395: Support unregistering controllers#22191
junrao merged 39 commits into
apache:trunkfrom
kevin-wu24:KAFKA-20395

Conversation

@kevin-wu24

@kevin-wu24 kevin-wu24 commented May 1, 2026

Copy link
Copy Markdown
Contributor

What Changed

  • Introduce UnregisterControllerRequest and
    UnregisterControllerResponse RPC schemas
  • Introduce Admin#unregisterController interface, which the
    kafka-cluster and kafka-metadata-quorum tools call via the
    AdminClient to unregister a controller
  • Add controller-side handling for UnregisterControllerRequest
  • Introduce UnregisterControllerRecord metadata record schema + new
    MetadataVersion IBP_4_4_IV2 to support this feature
  • Add handling for replaying the UnregisterControllerRecord on the
    active controller and in the Cluster/MetadataDelta to remove
    unregistered controllers from the MetadataImage
  • Add the kafka-cluster unregister-controller CLI command
  • Add the --unregister flag to the kafka-metadata-quorum remove-controller command

Testing

  • Unit testing for CLI changes, controller-side changes, and AdminClient
    changes
  • Integration tests in KRaftClusterTest

Reviewers: Jun Rao junrao@gmail.com, Chetan Koneru
(github:ChetanKoneru), Paolo Patierno ppatierno@live.com

@github-actions github-actions Bot added triage PRs from the community streams core Kafka Broker tools kraft clients labels May 1, 2026
@github-actions

github-actions Bot commented May 9, 2026

Copy link
Copy Markdown

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

@github-actions

Copy link
Copy Markdown

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

@github-actions

Copy link
Copy Markdown

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

@kevin-wu24
kevin-wu24 marked this pull request as ready for review May 18, 2026 15:03
@github-actions

Copy link
Copy Markdown

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

@ahuang98 ahuang98 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Kevin!

Comment on lines +9958 to +9965
private void runUnregisterScenario(
AdminClientUnitTestEnv env,
ApiKeys apiKey,
Function<Errors, AbstractResponse> responseFactory,
Function<Admin, KafkaFuture<Void>> adminCall,
List<Errors> responsesToPrepare,
Class<? extends Throwable> expectedException
) throws ExecutionException, InterruptedException {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this added abstraction feels a bit heavy - I see that there are similarities between unregister_broker and unregister_controller but imo this makes figuring out what the tests are doing a little too opaque

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically, I just copied the unregister broker tests, but that resulted in lots of duplicate code.

UnregisterControllerRequest request = new UnregisterControllerRequest.Builder(
new UnregisterControllerRequestData()
).build((short) 0);
String customerErrorMessage = "customer error message";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

customer feels like an odd choice here? did you mean custom?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was copied from the broker test above it haha. I can change both.


@Override
public UnregisterControllerResult unregisterController(int controllerId, UnregisterControllerOptions options) {
if (usingRaftController) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting, seems like we have more dead code to remove

Comment on lines +106 to +107
registrationModifier: RegisterControllerRecord => Option[RegisterControllerRecord],
unregisterModifier: UnregisterControllerRecord => Option[UnregisterControllerRecord]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know you're just extending the same semantics, but man these are so hard to reason about - can we add comments/examples?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree the modifier pattern can be a bit confusing, but I think this is outside of the scope of this PR. This file will get converted to java at some point, and maybe after that we can clean this up. What do you think?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment!

}

@Test
def testReRegistrationAfterDifferentIncarnationId(): Unit = {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just confirming, is this new or existing behavior that we're just adding test coverage for?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is existing behavior. From ControllerRegistrationManager:

} else if (!curRegistration.incarnationId().equals(incarnationId)) {
                            logger.info("Found registration for {} instead of our incarnation.", curRegistration.incarnationId());
                            registeredInLog = false;

setting the registeredInLog = false tells the manage to send ControllerRegistrationRequest.

.action(store())
.required(true)
.help("The ID of the broker to unregister.");
unregisterControllerParser.addArgument("--controller-id", "-i")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thoughts on just using "--id" here given that the subcommand is pretty self-explanatory ("unregister-controller")?

} catch (ExecutionException ee) {
Throwable cause = ee.getCause();
if (cause instanceof UnsupportedVersionException) {
stream.println("The target cluster does not support the controller unregistration API.");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the original exception message not descriptive enough?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also copied from the broker unregistration case.

Instead of dumping a stack trace in the case the cluster does not support the MV for unregistering controllers, it prints a simple message to the user. I think that is better UX for this case. I think we should also handle this in a similar way for other "expected exceptions" like InvalidRequestException, and ControllerIdNotRegisteredException. What do you think?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my comment applies to the broker unregistration case too, the original exception messages feel sufficient (and are more actionable in that they specifically mention MV is insufficient) so it feels odd we don't just include that in the output here. in any case, we don't have to change existing behavior in this PR and we can just have the controller unregistration API match

Comment on lines +536 to +538
+ ". To unregister the controller from the cluster, run "
+ "`kafka-cluster.sh unregister-controller --controller-id "
+ controllerId + "`.");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, should we include the original exception so the client knows why removeRaftVoter failed?

also, if the removal fails due to UnsupportedVersionException then wouldn't unregisterController also be unsupported?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, should we include the original exception so the client knows why removeRaftVoter failed?

Throwing TerseException strips the stack trace, but that only occurs when the user tries to remove AND unregister the controller in the same command. In any case, we still print cause.getMessage(), so we'll see Kafka's message when initializing the exception IIUC.

For these two exceptions specifically, I wanted to have the ability to direct the user to run the other tool if they want to unregister the controller, since removing the voter is not possible if those exceptions are returned.

also, if the removal fails due to UnsupportedVersionException then wouldn't unregisterController also be unsupported?

UnsupportedVersionException being returned by RemoveRaftVoterRequest means the cluster is kraft.version=0. You could still unregister a controller in a static quorum cluster, since kraft.version is a different feature from MV. For example, you can have an observer that does not set controller.quorum.voters and instead sets controller.quorum.bootstrap.servers.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed ": " + cause.getMessage()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, forgot those are gated by different features

Comment on lines +550 to +551
throw new TerseException("Removed KRaft voter " + controllerId
+ " but failed to unregister it: "

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't seem like the right place to log a successful kraft voter removal

@kevin-wu24 kevin-wu24 May 22, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. In handleRemoveController we print out something if removeRaftVoter returns successfully.

Comment on lines +74 to +76
assertTrue(outputs.contains("DRY RUN of removing KRaft controller 2 with directory id _KWDkTahTVaiVVVTaugNew"),
"Failed to find expected output in stdout: " + outputs);
assertTrue(outputs.contains("DRY RUN of unregistering KRaft controller 2"),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, looks like we are printing extra whitespace

@junrao junrao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kevin-wu24 : Thanks for the PR. Made a pass of non-testing files. Just a few minor comments.

Comment thread docs/operations/kraft.md Outdated
$ bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 remove-controller --controller-id <id> --controller-directory-id <directory-id>
```

To also unregister the controller from the cluster metadata after it has been removed from the voter set, pass the `--unregister` flag:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit confusing. It reads as if one can call kafka-metadata-quorum.sh first to remove the node from the voter set and then call it again to unregister it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea was to provide one CLI invocation for operators to allow for the "voter removal + unregistration" to happen in one CLI command. I can reword this comment to make it more clear.

* Options for {@link Admin#unregisterController(int, UnregisterControllerOptions)}.
*/
@InterfaceAudience.Public
public class UnregisterControllerOptions extends AbstractOptions<UnregisterControllerOptions> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not defined in the KIP. Could you add it there?

* <li>{@link org.apache.kafka.common.errors.TimeoutException}
* If the request timed out before the unregister operation could finish.</li>
* <li>{@link org.apache.kafka.common.errors.UnsupportedVersionException}
* If the software is too old to support the unregistration API.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The KIP also includes NOT_CONTROLLER and INVALID_REQUEST.

.action(store())
.required(true)
.help("The ID of the broker to unregister.");
unregisterControllerParser.addArgument("--id", "-i")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The KIP uses --controller-id. Should we update the KIP?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

throw new TerseException("Failed to unregister controller " + controllerId
+ ": " + (cause != null ? cause.getMessage() : e.getMessage())
+ ". To unregister the controller from the cluster, run "
+ "`kafka-cluster.sh unregister-controller --id "

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, if the unregister fails here, will it ever be successful using kafka-cluster.sh unregister-controller?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd argue yes. Many things could cause this second RPC loop to fail after the first succeeds. Some examples include: network partitions, leadership changes, and a cluster MV that does not support unregistering.

My goal for this exception handling is to direct the user to the kafka-cluster unregister-controller command, which only handles unregistering controllers. The kafka-metadata-quorum remove-controller --unregister can do two things. Attempting to unregister is conditioned on voter removal succeeding.

) {
return appendWriteEvent("unregisterController", context.deadlineNs(),
() -> {
if (nodeId == controllerId && isActiveController()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isActiveController() seems redundant since this code runs under ControllerWriteEvent.run() that checks activeController already.

} catch (ExecutionException e) {
Throwable cause = e.getCause();
throw new TerseException("Failed to unregister controller " + controllerId
+ ": " + (cause != null ? cause.getMessage() : e.getMessage())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We check for null here, but not in line 535. Could we be consistent?

@kevin-wu24 kevin-wu24 Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For L535, because we are checking:

(cause instanceof UnsupportedVersionException ||
                cause instanceof VoterNotFoundException))

cause != null is always true.

I wonder if we should make this command's behavior idempotent though in the case of VoterNotFoundException. Meaning that, if the voter is already gone from the voter set, we should try to unregister it, rather than throwing an exception. There is a precedent for this in other CLIs like kafka-storage format --ignore-formatted. What do you think?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, not sure about treating VoterNotFoundException the same as the voter is gone. removeRaftVoter takes both controllerId and directoryId. VoterNotFoundException could also mean the voter is still there, but the directoryId is incorrect.

@kevin-wu24 kevin-wu24 Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VoterNotFoundException could also mean the voter is still there, but the directoryId is incorrect.

Hmm, yeah. I don't think there is a way to differentiate between those two cases. I think I'll just leave the behavior as is.


public void replay(UnregisterControllerRecord record) {
int controllerId = record.controllerId();
ControllerRegistration registration = controllerRegistrations.get(controllerId);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can just call remove() here since it returns the previous value if present.

@junrao junrao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kevin-wu24 : Thanks for the updated PR. Looks good to me overall. Just a minor comment.

} catch (ExecutionException e) {
Throwable cause = e.getCause();
throw new TerseException("Failed to unregister controller " + controllerId
+ ": " + (cause != null ? cause.getMessage() : e.getMessage())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, not sure about treating VoterNotFoundException the same as the voter is gone. removeRaftVoter takes both controllerId and directoryId. VoterNotFoundException could also mean the voter is still there, but the directoryId is incorrect.

@ChetanKoneru

Copy link
Copy Markdown

Thank you for working on this—this is directly relevant to a production stale-controller recovery case we have.

see - https://github.com/orgs/strimzi/discussions/13001

We run a Strimzi Kafka with KRaft on AKS. After upgrading Strimzi 0.42.0 -> 0.45.2 and Kafka 3.7.1 -> 3.9.2, the cluster is healthy (Ready=True) but cannot finalize the metadata upgrade from 3.7-IV4 to 3.9-IV0.

Strimzi: 0.45.2
Kafka: 3.9.2
KRaft metadata state: KRaft
Finalized metadata: 3.7-IV4
Requested metadata: 3.9-IV0

Active brokers: 0,1,2
Active controllers: 90,91,92
status.registeredNodeIds: [0,1,2,90,91,92]

During the original ZooKeeper-to-KRaft migration, the controller NodePool did not have reserved node IDs, so controllers were assigned 3,4,5. We later reserved IDs explicitly:

broker NodePool
strimzi.io/next-node-ids: "[0-49]"

controller NodePool
strimzi.io/next-node-ids: "[90-99]"

The active controller NodePool now uses 90,91,92. Controllers 3,4,5 are no longer Pods or quorum voters, but their registrations remain in __cluster_metadata:

ControllerRegistration(id=3, supportedFeatures={metadata.version: 1-19})
ControllerRegistration(id=4, supportedFeatures={metadata.version: 1-19})
ControllerRegistration(id=5, supportedFeatures={metadata.version: 1-19})

The active controllers and brokers support metadata feature levels 1-21. The metadata upgrade fails because the stale registration still participates in validation:

InvalidUpdateVersionException:
Invalid update version 21 for feature metadata.version.
Controller 5 only supports versions 1-19

As the safe mitigation, we retain Kafka 3.9.2 but keep the desired metadata version at 3.7-IV4. This stops repeated failed update attempts, but leaves the expected incomplete-upgrade warning.

We understand this PR adds Admin#unregisterController, kafka-cluster.sh unregister-controller, and UnregisterControllerRecord, which is exactly the missing capability.

Could you clarify whether this operation requires the cluster to have already finalized the metadata version that supports UnregisterControllerRecord (currently proposed as IBP_4_4_IV1)?

If so, this existing-cluster recovery case appears circular:

  1. Stale controller registrations must be removed to advance metadata.
  2. The proposed controller-unregistration record requires newer metadata.
  3. Advancing metadata is blocked by those same stale registrations.

Is an in-place repair path for clusters stuck at an older finalized metadata version in scope for this PR or planned separately? In particular, is there a supported way to unregister obsolete controller registrations while retaining metadata.version=3.7-IV4?

@kevin-wu24

Copy link
Copy Markdown
Contributor Author

Hi @ChetanKoneru,

Thanks for the questions. Please see the compatibility section and rejected alternatives sections of KIP-1312: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406623954#KIP1312:Supportunregisteringcontrollers-Compatibility,Deprecation,andMigrationPlan. This feature only applies to new clusters going forward. In many cases, operators can "refresh" a stale controller registration by bringing up a controller with the same node id as the stale registration. I'm not familiar with Strimzi's operator, so I cannot speak to your specific case unfortunately.

Could you clarify whether this operation requires the cluster to have already finalized the metadata version that supports UnregisterControllerRecord (currently proposed as IBP_4_4_IV1)?

Yes, clusters need a metadata version that supports the record to serialize and deserialize it. New metadata records must be gated behind new metadata versions. This is because a given metadata version must be supported by all nodes registered with the cluster to be finalized.

Is an in-place repair path for clusters stuck at an older finalized metadata version in scope for this PR or planned separately?

This is not in scope for this PR or planned separately.

@junrao junrao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kevin-wu24 : Thanks for the updated PR. A few more comments.

if (nodeId == controllerId) {
throw new InvalidRequestException("Controller cannot unregister itself while it is active.");
}
return clusterControl.unregisterController(controllerId);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finding from Claude.

Should we remove the controllerId from FeatureControlManager's quorumFeatures.quorumNodeIds()? Otherwise, this unregistered controller can block future MV upgrades.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we remove the controllerId from FeatureControlManager's quorumFeatures.quorumNodeIds()?

I'm looking at FeatureControlManager#reasonNotSupported, which is how we validate the cluster members support a feature upgrade via their registrations. It looks like we do look through the broker + controller registrations, which is good.

From my reading, the quorumNodeIds check on FeatureControlManager L360 serves as a way to check that the static voter set members from controller.quorum.voters are all registered with the cluster prior to an upgrade. If the config is not set, then this field is an empty list.

From https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406623954#KIP1312:Supportunregisteringcontrollers-UserExperience, and assuming the voter in question is not able to send a registration, we should not be unregistering a member of the static voter set without removing it from that config first. The UX guide recommends removing the controller from the voter set config prior to unregistering it. This means the lack of a registration from a static voter SHOULD block feature upgrades in my opinion. Therefore, I think the quorumNodeIds check is correct here. What do you think?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, one check we can add is rejecting the unregister request if the controller id appears in quorumFeatures.quorumNodeIds(), since that means the id is part of controller.quorum.voters. This will help ensure users remove the controller id from the static voter set before unregistering it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The KIP says the following.

Unregister a voter in a static KRaft quorum when the static voter set is mistakenly configured
1. Stop the voter who was mistakenly put in controller.quorum.voters 
2. Run kafka-cluster unregister-controller
3. Ensure the stopped voter is not part of controller.quorum.voters on every Kafka node

What will be the new sequence?

@kevin-wu24 kevin-wu24 Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Run kafka-cluster unregister-controller would be step 3. Stopping the voter and removing it from all controller.quorum.voters configs can be done interchangeably I believe.

That is to say, the preconditions for unregistering a controller with id X in a static quorum are:

  1. X is not present in controller.quorum.voters
  2. X is not running (if it is it will re-register)

Ideally, we should have the same pre-requisites for dynamic quorum WRT the voters set. I think the approach would be analogous to using RaftControllerNodeProvider or KRaftVersionAccessor to extract raft internal state. Maybe it is okay to file a JIRA for this as a follow-up instead, since unregistering a live voter in either quorum mode will cause the voter to register itself again, so the cluster can recover from that "incorrect" operation. What do you think @junrao?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Sounds good. Could you update the PR and the KIP?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

int controllerId = record.controllerId();
ControllerRegistration registration = controllerRegistrations.remove(controllerId);
if (registration == null) {
throw new RuntimeException(String.format("Unable to replay %s: no controller " +

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finding from Claude.

UnregisterBrokerRecord has a brokerEpoch field, which is checked during replay. UnregisterControllerRecord doesn't have a brokerEpoch field. Why the inconsistency?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UnregisterControllerRecord doesn't have a brokerEpoch field. Why the inconsistency?

The main reasons are that brokerEpoch is used for broker lifecycle management between broker and active controller, such as fencing, unfencing, and clean shutdown detection.brokerEpoch also provides the active controller with a mechanism for detecting a request from a "zombie-broker" incarnation/registration and reject it.

However, controller registration is more "simple" IMO, in that they only really are needed by the metadata layer to know each controller's endpoints + supported features. These are properties which only change on a process restart, and controllers will refresh their registrations on restart because they will not see their incarnation ID in the existing registration.

// initial state with controller registered in log
val image = doMetadataUpdate(MetadataImage.EMPTY,
manager,
MetadataVersion.IBP_3_7_IV0,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we use an old MV? Ditto below.

@kevin-wu24 kevin-wu24 Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. Adding a MV check. The test below doesn't test unregistering, just re-registering when the existing registration has the wrong incarnation ID, so I will leave it as is, since that old MV supports registration.

int controllerIdToUnregister = cluster.controllers().keySet().iterator().next();
cluster.controllers().get(controllerIdToUnregister).shutdown();

try (Admin admin = createAdminClient(cluster, usingBootstrapControllers)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we guarantee that the active controller is elected when we get here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a cluster.waitForActiveController call in both tests.


@Override
public boolean shouldClientThrottle(short version) {
return true;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to override the method now that we merged #22908

"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "ControllerId", "type": "int32", "versions": "0+",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add "entityType" following UnregisterBrokerRequest?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not super clear on the purpose of entityType, but I did not add it because RegisterControllerRecord does not use it either. This is what claude said:

entityType is nearly inert in codegen. The only consumers are FieldSpec.java:100, which calls verifyTypeMatches to assert the declared type matches the entity's base type (brokerId → int32), and Target.java:50, which just copies it through.

* returned {@link UnregisterControllerResult}:
* <ul>
* <li>{@link org.apache.kafka.common.errors.TimeoutException}
* If the request timed out before the unregister operation could finish.</li>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add </li> for the remaining <li>?

@ppatierno

Copy link
Copy Markdown
Contributor

This feature only applies to new clusters going forward. In many cases, operators can "refresh" a stale controller registration by bringing up a controller with the same node id as the stale registration. I'm not familiar with Strimzi's operator, so I cannot speak to your specific case unfortunately.

@kevin-wu24 sorry if I missed that in the rejected alternatives during KIP discussion but let me ask a question. If the Kafka cluster is running by using static quorum, the KIP suggest to "refresh" the node ID by scaling up a new controller with same ID and then scale it down again. Doesn't it go against the way the static quorum works and cannot be changed? (so dynamic quorum was developed).

@kevin-wu24

Copy link
Copy Markdown
Contributor Author

This feature only applies to new clusters going forward. In many cases, operators can "refresh" a stale controller registration by bringing up a controller with the same node id as the stale registration. I'm not familiar with Strimzi's operator, so I cannot speak to your specific case unfortunately.

@kevin-wu24 sorry if I missed that in the rejected alternatives during KIP discussion but let me ask a question. If the Kafka cluster is running by using static quorum, the KIP suggest to "refresh" the node ID by scaling up a new controller with same ID and then scale it down again. Doesn't it go against the way the static quorum works and cannot be changed? (so dynamic quorum was developed).

@ppatierno not necessarily. Having observer controllers is still a valid cluster configuration in static quorum. For example, you can bring a up a node in a static quorum cluster whose configuration uses controller.quorum.bootstrap.servers to discover the quorum, rather than controller.quorum.voters. Stale controller registrations only apply to observers, because otherwise, your cluster has a voter set failure/misconfiguration. The difference is that in a static quorum, you cannot promote the observer controller to a voter, which you can do in dynamic quorum.

@junrao junrao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kevin-wu24 : Thanks for the updated PR. Just a couple of minor comments.

}
}
if (metadataVersion.isControllerUnregistrationSupported) {
for (i <- Seq(1, 2, 3)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

identation

@kevin-wu24 kevin-wu24 Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 61446f5

if (nodeId == controllerId) {
throw new InvalidRequestException("Controller cannot unregister itself while it is active.");
}
return clusterControl.unregisterController(controllerId);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The KIP says the following.

Unregister a voter in a static KRaft quorum when the static voter set is mistakenly configured
1. Stop the voter who was mistakenly put in controller.quorum.voters 
2. Run kafka-cluster unregister-controller
3. Ensure the stopped voter is not part of controller.quorum.voters on every Kafka node

What will be the new sequence?

@junrao junrao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kevin-wu24 : Thanks for the updated PR. LGTM Do you know why the CI failed?

@kevin-wu24

kevin-wu24 commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the reviews @junrao.

Do you know why the CI failed?

Not sure. It seems unrelated. I think the Java 17 unit test step timed out with 2 flaky tests? This is what I see:

Gradle task had a timeout. Failing this script. These are partial results!

Are you able to re-run that task? I don't think I can.

@junrao

junrao commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

I triggered a rerun of all jobs. You can also rebase to trigger a rerun yourself.

@junrao

junrao commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Hmm, my rerun also failed. Could you rebase to trigger another rerun?

@kevin-wu24

Copy link
Copy Markdown
Contributor Author

Could you rebase to trigger another rerun?

Sure. It looks like the re-run passed.

@junrao
junrao merged commit c274a73 into apache:trunk Aug 3, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants