KAFKA-20395: Support unregistering controllers - #22191
Conversation
…adata-quorum tool error handling
|
A label of 'needs-attention' was automatically added to this PR in order to raise the |
|
A label of 'needs-attention' was automatically added to this PR in order to raise the |
|
A label of 'needs-attention' was automatically added to this PR in order to raise the |
|
A label of 'needs-attention' was automatically added to this PR in order to raise the |
| private void runUnregisterScenario( | ||
| AdminClientUnitTestEnv env, | ||
| ApiKeys apiKey, | ||
| Function<Errors, AbstractResponse> responseFactory, | ||
| Function<Admin, KafkaFuture<Void>> adminCall, | ||
| List<Errors> responsesToPrepare, | ||
| Class<? extends Throwable> expectedException | ||
| ) throws ExecutionException, InterruptedException { |
There was a problem hiding this comment.
this added abstraction feels a bit heavy - I see that there are similarities between unregister_broker and unregister_controller but imo this makes figuring out what the tests are doing a little too opaque
There was a problem hiding this comment.
Basically, I just copied the unregister broker tests, but that resulted in lots of duplicate code.
| UnregisterControllerRequest request = new UnregisterControllerRequest.Builder( | ||
| new UnregisterControllerRequestData() | ||
| ).build((short) 0); | ||
| String customerErrorMessage = "customer error message"; |
There was a problem hiding this comment.
customer feels like an odd choice here? did you mean custom?
There was a problem hiding this comment.
This was copied from the broker test above it haha. I can change both.
|
|
||
| @Override | ||
| public UnregisterControllerResult unregisterController(int controllerId, UnregisterControllerOptions options) { | ||
| if (usingRaftController) { |
There was a problem hiding this comment.
interesting, seems like we have more dead code to remove
| registrationModifier: RegisterControllerRecord => Option[RegisterControllerRecord], | ||
| unregisterModifier: UnregisterControllerRecord => Option[UnregisterControllerRecord] |
There was a problem hiding this comment.
I know you're just extending the same semantics, but man these are so hard to reason about - can we add comments/examples?
There was a problem hiding this comment.
I agree the modifier pattern can be a bit confusing, but I think this is outside of the scope of this PR. This file will get converted to java at some point, and maybe after that we can clean this up. What do you think?
| } | ||
|
|
||
| @Test | ||
| def testReRegistrationAfterDifferentIncarnationId(): Unit = { |
There was a problem hiding this comment.
just confirming, is this new or existing behavior that we're just adding test coverage for?
There was a problem hiding this comment.
This is existing behavior. From ControllerRegistrationManager:
} else if (!curRegistration.incarnationId().equals(incarnationId)) {
logger.info("Found registration for {} instead of our incarnation.", curRegistration.incarnationId());
registeredInLog = false;
setting the registeredInLog = false tells the manage to send ControllerRegistrationRequest.
| .action(store()) | ||
| .required(true) | ||
| .help("The ID of the broker to unregister."); | ||
| unregisterControllerParser.addArgument("--controller-id", "-i") |
There was a problem hiding this comment.
thoughts on just using "--id" here given that the subcommand is pretty self-explanatory ("unregister-controller")?
| } catch (ExecutionException ee) { | ||
| Throwable cause = ee.getCause(); | ||
| if (cause instanceof UnsupportedVersionException) { | ||
| stream.println("The target cluster does not support the controller unregistration API."); |
There was a problem hiding this comment.
is the original exception message not descriptive enough?
There was a problem hiding this comment.
This is also copied from the broker unregistration case.
Instead of dumping a stack trace in the case the cluster does not support the MV for unregistering controllers, it prints a simple message to the user. I think that is better UX for this case. I think we should also handle this in a similar way for other "expected exceptions" like InvalidRequestException, and ControllerIdNotRegisteredException. What do you think?
There was a problem hiding this comment.
my comment applies to the broker unregistration case too, the original exception messages feel sufficient (and are more actionable in that they specifically mention MV is insufficient) so it feels odd we don't just include that in the output here. in any case, we don't have to change existing behavior in this PR and we can just have the controller unregistration API match
| + ". To unregister the controller from the cluster, run " | ||
| + "`kafka-cluster.sh unregister-controller --controller-id " | ||
| + controllerId + "`."); |
There was a problem hiding this comment.
hm, should we include the original exception so the client knows why removeRaftVoter failed?
also, if the removal fails due to UnsupportedVersionException then wouldn't unregisterController also be unsupported?
There was a problem hiding this comment.
hm, should we include the original exception so the client knows why removeRaftVoter failed?
Throwing TerseException strips the stack trace, but that only occurs when the user tries to remove AND unregister the controller in the same command. In any case, we still print cause.getMessage(), so we'll see Kafka's message when initializing the exception IIUC.
For these two exceptions specifically, I wanted to have the ability to direct the user to run the other tool if they want to unregister the controller, since removing the voter is not possible if those exceptions are returned.
also, if the removal fails due to UnsupportedVersionException then wouldn't unregisterController also be unsupported?
UnsupportedVersionException being returned by RemoveRaftVoterRequest means the cluster is kraft.version=0. You could still unregister a controller in a static quorum cluster, since kraft.version is a different feature from MV. For example, you can have an observer that does not set controller.quorum.voters and instead sets controller.quorum.bootstrap.servers.
There was a problem hiding this comment.
I missed ": " + cause.getMessage()
There was a problem hiding this comment.
nice, forgot those are gated by different features
| throw new TerseException("Removed KRaft voter " + controllerId | ||
| + " but failed to unregister it: " |
There was a problem hiding this comment.
this doesn't seem like the right place to log a successful kraft voter removal
There was a problem hiding this comment.
I see. In handleRemoveController we print out something if removeRaftVoter returns successfully.
| assertTrue(outputs.contains("DRY RUN of removing KRaft controller 2 with directory id _KWDkTahTVaiVVVTaugNew"), | ||
| "Failed to find expected output in stdout: " + outputs); | ||
| assertTrue(outputs.contains("DRY RUN of unregistering KRaft controller 2"), |
There was a problem hiding this comment.
nit, looks like we are printing extra whitespace
junrao
left a comment
There was a problem hiding this comment.
@kevin-wu24 : Thanks for the PR. Made a pass of non-testing files. Just a few minor comments.
| $ bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 remove-controller --controller-id <id> --controller-directory-id <directory-id> | ||
| ``` | ||
|
|
||
| To also unregister the controller from the cluster metadata after it has been removed from the voter set, pass the `--unregister` flag: |
There was a problem hiding this comment.
This is a bit confusing. It reads as if one can call kafka-metadata-quorum.sh first to remove the node from the voter set and then call it again to unregister it.
There was a problem hiding this comment.
The idea was to provide one CLI invocation for operators to allow for the "voter removal + unregistration" to happen in one CLI command. I can reword this comment to make it more clear.
| * Options for {@link Admin#unregisterController(int, UnregisterControllerOptions)}. | ||
| */ | ||
| @InterfaceAudience.Public | ||
| public class UnregisterControllerOptions extends AbstractOptions<UnregisterControllerOptions> { |
There was a problem hiding this comment.
This is not defined in the KIP. Could you add it there?
| * <li>{@link org.apache.kafka.common.errors.TimeoutException} | ||
| * If the request timed out before the unregister operation could finish.</li> | ||
| * <li>{@link org.apache.kafka.common.errors.UnsupportedVersionException} | ||
| * If the software is too old to support the unregistration API. |
There was a problem hiding this comment.
The KIP also includes NOT_CONTROLLER and INVALID_REQUEST.
| .action(store()) | ||
| .required(true) | ||
| .help("The ID of the broker to unregister."); | ||
| unregisterControllerParser.addArgument("--id", "-i") |
There was a problem hiding this comment.
The KIP uses --controller-id. Should we update the KIP?
| throw new TerseException("Failed to unregister controller " + controllerId | ||
| + ": " + (cause != null ? cause.getMessage() : e.getMessage()) | ||
| + ". To unregister the controller from the cluster, run " | ||
| + "`kafka-cluster.sh unregister-controller --id " |
There was a problem hiding this comment.
Hmm, if the unregister fails here, will it ever be successful using kafka-cluster.sh unregister-controller?
There was a problem hiding this comment.
I'd argue yes. Many things could cause this second RPC loop to fail after the first succeeds. Some examples include: network partitions, leadership changes, and a cluster MV that does not support unregistering.
My goal for this exception handling is to direct the user to the kafka-cluster unregister-controller command, which only handles unregistering controllers. The kafka-metadata-quorum remove-controller --unregister can do two things. Attempting to unregister is conditioned on voter removal succeeding.
| ) { | ||
| return appendWriteEvent("unregisterController", context.deadlineNs(), | ||
| () -> { | ||
| if (nodeId == controllerId && isActiveController()) { |
There was a problem hiding this comment.
isActiveController() seems redundant since this code runs under ControllerWriteEvent.run() that checks activeController already.
| } catch (ExecutionException e) { | ||
| Throwable cause = e.getCause(); | ||
| throw new TerseException("Failed to unregister controller " + controllerId | ||
| + ": " + (cause != null ? cause.getMessage() : e.getMessage()) |
There was a problem hiding this comment.
We check for null here, but not in line 535. Could we be consistent?
There was a problem hiding this comment.
For L535, because we are checking:
(cause instanceof UnsupportedVersionException ||
cause instanceof VoterNotFoundException))
cause != null is always true.
I wonder if we should make this command's behavior idempotent though in the case of VoterNotFoundException. Meaning that, if the voter is already gone from the voter set, we should try to unregister it, rather than throwing an exception. There is a precedent for this in other CLIs like kafka-storage format --ignore-formatted. What do you think?
There was a problem hiding this comment.
Hmm, not sure about treating VoterNotFoundException the same as the voter is gone. removeRaftVoter takes both controllerId and directoryId. VoterNotFoundException could also mean the voter is still there, but the directoryId is incorrect.
There was a problem hiding this comment.
VoterNotFoundException could also mean the voter is still there, but the directoryId is incorrect.
Hmm, yeah. I don't think there is a way to differentiate between those two cases. I think I'll just leave the behavior as is.
|
|
||
| public void replay(UnregisterControllerRecord record) { | ||
| int controllerId = record.controllerId(); | ||
| ControllerRegistration registration = controllerRegistrations.get(controllerId); |
There was a problem hiding this comment.
We can just call remove() here since it returns the previous value if present.
junrao
left a comment
There was a problem hiding this comment.
@kevin-wu24 : Thanks for the updated PR. Looks good to me overall. Just a minor comment.
| } catch (ExecutionException e) { | ||
| Throwable cause = e.getCause(); | ||
| throw new TerseException("Failed to unregister controller " + controllerId | ||
| + ": " + (cause != null ? cause.getMessage() : e.getMessage()) |
There was a problem hiding this comment.
Hmm, not sure about treating VoterNotFoundException the same as the voter is gone. removeRaftVoter takes both controllerId and directoryId. VoterNotFoundException could also mean the voter is still there, but the directoryId is incorrect.
|
Thank you for working on this—this is directly relevant to a production stale-controller recovery case we have. see - https://github.com/orgs/strimzi/discussions/13001 We run a Strimzi Kafka with KRaft on AKS. After upgrading Strimzi During the original ZooKeeper-to-KRaft migration, the controller NodePool did not have reserved node IDs, so controllers were assigned The active controller NodePool now uses The active controllers and brokers support metadata feature levels As the safe mitigation, we retain Kafka We understand this PR adds Could you clarify whether this operation requires the cluster to have already finalized the metadata version that supports If so, this existing-cluster recovery case appears circular:
Is an in-place repair path for clusters stuck at an older finalized metadata version in scope for this PR or planned separately? In particular, is there a supported way to unregister obsolete controller registrations while retaining |
|
Hi @ChetanKoneru, Thanks for the questions. Please see the compatibility section and rejected alternatives sections of KIP-1312: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406623954#KIP1312:Supportunregisteringcontrollers-Compatibility,Deprecation,andMigrationPlan. This feature only applies to new clusters going forward. In many cases, operators can "refresh" a stale controller registration by bringing up a controller with the same node id as the stale registration. I'm not familiar with Strimzi's operator, so I cannot speak to your specific case unfortunately.
Yes, clusters need a metadata version that supports the record to serialize and deserialize it. New metadata records must be gated behind new metadata versions. This is because a given metadata version must be supported by all nodes registered with the cluster to be finalized.
This is not in scope for this PR or planned separately. |
junrao
left a comment
There was a problem hiding this comment.
@kevin-wu24 : Thanks for the updated PR. A few more comments.
| if (nodeId == controllerId) { | ||
| throw new InvalidRequestException("Controller cannot unregister itself while it is active."); | ||
| } | ||
| return clusterControl.unregisterController(controllerId); |
There was a problem hiding this comment.
Finding from Claude.
Should we remove the controllerId from FeatureControlManager's quorumFeatures.quorumNodeIds()? Otherwise, this unregistered controller can block future MV upgrades.
There was a problem hiding this comment.
Should we remove the controllerId from FeatureControlManager's quorumFeatures.quorumNodeIds()?
I'm looking at FeatureControlManager#reasonNotSupported, which is how we validate the cluster members support a feature upgrade via their registrations. It looks like we do look through the broker + controller registrations, which is good.
From my reading, the quorumNodeIds check on FeatureControlManager L360 serves as a way to check that the static voter set members from controller.quorum.voters are all registered with the cluster prior to an upgrade. If the config is not set, then this field is an empty list.
From https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406623954#KIP1312:Supportunregisteringcontrollers-UserExperience, and assuming the voter in question is not able to send a registration, we should not be unregistering a member of the static voter set without removing it from that config first. The UX guide recommends removing the controller from the voter set config prior to unregistering it. This means the lack of a registration from a static voter SHOULD block feature upgrades in my opinion. Therefore, I think the quorumNodeIds check is correct here. What do you think?
There was a problem hiding this comment.
Actually, one check we can add is rejecting the unregister request if the controller id appears in quorumFeatures.quorumNodeIds(), since that means the id is part of controller.quorum.voters. This will help ensure users remove the controller id from the static voter set before unregistering it.
There was a problem hiding this comment.
The KIP says the following.
Unregister a voter in a static KRaft quorum when the static voter set is mistakenly configured
1. Stop the voter who was mistakenly put in controller.quorum.voters
2. Run kafka-cluster unregister-controller
3. Ensure the stopped voter is not part of controller.quorum.voters on every Kafka node
What will be the new sequence?
There was a problem hiding this comment.
Run kafka-cluster unregister-controller would be step 3. Stopping the voter and removing it from all controller.quorum.voters configs can be done interchangeably I believe.
That is to say, the preconditions for unregistering a controller with id X in a static quorum are:
- X is not present in
controller.quorum.voters - X is not running (if it is it will re-register)
Ideally, we should have the same pre-requisites for dynamic quorum WRT the voters set. I think the approach would be analogous to using RaftControllerNodeProvider or KRaftVersionAccessor to extract raft internal state. Maybe it is okay to file a JIRA for this as a follow-up instead, since unregistering a live voter in either quorum mode will cause the voter to register itself again, so the cluster can recover from that "incorrect" operation. What do you think @junrao?
There was a problem hiding this comment.
Thanks. Sounds good. Could you update the PR and the KIP?
There was a problem hiding this comment.
| int controllerId = record.controllerId(); | ||
| ControllerRegistration registration = controllerRegistrations.remove(controllerId); | ||
| if (registration == null) { | ||
| throw new RuntimeException(String.format("Unable to replay %s: no controller " + |
There was a problem hiding this comment.
Finding from Claude.
UnregisterBrokerRecord has a brokerEpoch field, which is checked during replay. UnregisterControllerRecord doesn't have a brokerEpoch field. Why the inconsistency?
There was a problem hiding this comment.
UnregisterControllerRecord doesn't have a brokerEpoch field. Why the inconsistency?
The main reasons are that brokerEpoch is used for broker lifecycle management between broker and active controller, such as fencing, unfencing, and clean shutdown detection.brokerEpoch also provides the active controller with a mechanism for detecting a request from a "zombie-broker" incarnation/registration and reject it.
However, controller registration is more "simple" IMO, in that they only really are needed by the metadata layer to know each controller's endpoints + supported features. These are properties which only change on a process restart, and controllers will refresh their registrations on restart because they will not see their incarnation ID in the existing registration.
| // initial state with controller registered in log | ||
| val image = doMetadataUpdate(MetadataImage.EMPTY, | ||
| manager, | ||
| MetadataVersion.IBP_3_7_IV0, |
There was a problem hiding this comment.
Why do we use an old MV? Ditto below.
There was a problem hiding this comment.
Thanks for catching this. Adding a MV check. The test below doesn't test unregistering, just re-registering when the existing registration has the wrong incarnation ID, so I will leave it as is, since that old MV supports registration.
| int controllerIdToUnregister = cluster.controllers().keySet().iterator().next(); | ||
| cluster.controllers().get(controllerIdToUnregister).shutdown(); | ||
|
|
||
| try (Admin admin = createAdminClient(cluster, usingBootstrapControllers)) { |
There was a problem hiding this comment.
Do we guarantee that the active controller is elected when we get here?
There was a problem hiding this comment.
Added a cluster.waitForActiveController call in both tests.
|
|
||
| @Override | ||
| public boolean shouldClientThrottle(short version) { | ||
| return true; |
There was a problem hiding this comment.
We don't need to override the method now that we merged #22908
| "validVersions": "0", | ||
| "flexibleVersions": "0+", | ||
| "fields": [ | ||
| { "name": "ControllerId", "type": "int32", "versions": "0+", |
There was a problem hiding this comment.
Should we add "entityType" following UnregisterBrokerRequest?
There was a problem hiding this comment.
I'm not super clear on the purpose of entityType, but I did not add it because RegisterControllerRecord does not use it either. This is what claude said:
entityType is nearly inert in codegen. The only consumers are FieldSpec.java:100, which calls verifyTypeMatches to assert the declared type matches the entity's base type (brokerId → int32), and Target.java:50, which just copies it through.
| * returned {@link UnregisterControllerResult}: | ||
| * <ul> | ||
| * <li>{@link org.apache.kafka.common.errors.TimeoutException} | ||
| * If the request timed out before the unregister operation could finish.</li> |
There was a problem hiding this comment.
Should we add </li> for the remaining <li>?
@kevin-wu24 sorry if I missed that in the rejected alternatives during KIP discussion but let me ask a question. If the Kafka cluster is running by using static quorum, the KIP suggest to "refresh" the node ID by scaling up a new controller with same ID and then scale it down again. Doesn't it go against the way the static quorum works and cannot be changed? (so dynamic quorum was developed). |
@ppatierno not necessarily. Having observer controllers is still a valid cluster configuration in static quorum. For example, you can bring a up a node in a static quorum cluster whose configuration uses |
junrao
left a comment
There was a problem hiding this comment.
@kevin-wu24 : Thanks for the updated PR. Just a couple of minor comments.
| } | ||
| } | ||
| if (metadataVersion.isControllerUnregistrationSupported) { | ||
| for (i <- Seq(1, 2, 3)) { |
| if (nodeId == controllerId) { | ||
| throw new InvalidRequestException("Controller cannot unregister itself while it is active."); | ||
| } | ||
| return clusterControl.unregisterController(controllerId); |
There was a problem hiding this comment.
The KIP says the following.
Unregister a voter in a static KRaft quorum when the static voter set is mistakenly configured
1. Stop the voter who was mistakenly put in controller.quorum.voters
2. Run kafka-cluster unregister-controller
3. Ensure the stopped voter is not part of controller.quorum.voters on every Kafka node
What will be the new sequence?
junrao
left a comment
There was a problem hiding this comment.
@kevin-wu24 : Thanks for the updated PR. LGTM Do you know why the CI failed?
|
Thanks for the reviews @junrao.
Not sure. It seems unrelated. I think the Java 17 unit test step timed out with 2 flaky tests? This is what I see: Are you able to re-run that task? I don't think I can. |
|
I triggered a rerun of all jobs. You can also rebase to trigger a rerun yourself. |
|
Hmm, my rerun also failed. Could you rebase to trigger another rerun? |
Sure. It looks like the re-run passed. |
What Changed
UnregisterControllerRequestandUnregisterControllerResponseRPC schemasAdmin#unregisterControllerinterface, which thekafka-clusterandkafka-metadata-quorumtools call via theAdminClientto unregister a controllerUnregisterControllerRequestUnregisterControllerRecordmetadata record schema + newMetadataVersion
IBP_4_4_IV2to support this featureUnregisterControllerRecordon theactive controller and in the
Cluster/MetadataDeltato removeunregistered controllers from the
MetadataImagekafka-cluster unregister-controllerCLI command--unregisterflag to thekafka-metadata-quorum remove-controllercommandTesting
changes
KRaftClusterTestReviewers: Jun Rao junrao@gmail.com, Chetan Koneru
(github:ChetanKoneru), Paolo Patierno ppatierno@live.com