Repository navigation
Add kubectl-byoc-migrate to migrate the CloudPrem and OPW Helm charts - #3755
Draft
shuheiktgw wants to merge 10 commits into
Draft
shuheiktgw wants to merge 10 commits into
shuheiktgw wants to merge 10 commits into
Conversation
shuheiktgw
force-pushed
the
shuheiktgw/byoc_migrate_cli
branch
from
November 10, 2026 13:26
61de01c to
fd6ce0e
Compare
🛑 Gate Violations
ℹ️ Info🎯 Code Coverage (details) Useful? React with 👍 / 👎 This comment will be updated automatically if new data arrives.🔗 Commit SHA: fd6e62d | Docs | Give us feedback! |
shuheiktgw
force-pushed
the
shuheiktgw/byoc_migrate_cli
branch
from
November 10, 2026 13:58
fd6ce0e to
403cd98
Compare
kubectl-byoc-migrate gives the fields of the CloudPrem objects that it migrates from the Helm charts to the field manager of the controller, so that the controller keeps applying them with the same manager.
kubectl-byoc-migrate converts the objects of the CloudPrem and Observability Pipelines Worker Helm charts to a DatadogBYOCCluster, compares them with the resources of the cluster, adopts the CloudPrem objects in place, and aborts the migration. It does not create, change, or delete the cluster: the user applies and deletes it, for example from the deployment pipeline that applies the other resources. adopt and abort do not pause the cluster. The controller applies an object only when the cluster controls it, and with the resource version that it read, so it does not write to an object before adopt gives it to the cluster, nor give back an object that abort removed from the owners of the cluster. adopt and abort check that the user of the context may patch each CloudPrem object, as kubectl auth can-i does, and write nothing otherwise. The converter treats a missing gossip_protocol_version as version 0, which older CloudPrem charts run with, so that nodeConfigOverrides keeps the version of the cluster.
adopt kept every field manager of a Service, so that load balancer settings survive, but that also kept the helm entry: the Services kept the meta.helm.sh annotations and the chart labels after the migration, and the controller could not remove a port or label that Helm still owned. adopt now gives the fields of the helm and kubectl-client-side-apply managers of a Service to the controller, and keeps the other managers. diff predicts the Service as the resource of the cluster plus the fields of the kept managers.
…ation test adopt now gives the fields of the helm manager of a Service to the DatadogBYOCCluster controller, so the chart label is removed and the helm manager no longer appears in the managed fields.
awsSpec read the IRSA annotation of the ServiceAccount of the indexer before checking that the manifest has it, so Convert panicked when the chart was rendered with serviceAccount.create=false.
TestConvert now also converts a CloudPrem manifest without pipelines, with two pipelines, with labels, annotations, scheduling, init containers, and volumes, and with an image tag from a private registry and an existing ServiceAccount. The CloudPrem manifests are rendered from commit 025ffd4 of the chart, which reproduces cloudprem.yaml and still has version 0.5.4.
…controller After adopt, diff applied every resource that the cluster or a Worker controls with the field manager of the DatadogBYOCCluster controller. The Worker controller applies its resources with its own manager, so the dry-run kept the fields that the Worker controller had set but no longer applies, and diff did not show their removal, for example of a port removed from a pipeline. diff now applies the resources of a Worker with the field manager of the Worker controller, which the utils package exports next to the one of the cluster. diff also leaves out the kind and the API version of the objects, which the header of each diff names: only the objects applied with a dry-run set them, so their diffs showed them as added.
abort refused to write until the StatefulSet of each Observability Pipelines Worker manifest had all its replicas ready, because deleting the cluster next deletes the Workers. The check could only see the Pods, not whether the clients moved back to them, and moving the traffic back is a step of the user, like scaling the StatefulSets. abort now only removes the cluster from the owners of the CloudPrem objects, and reads only the CloudPrem manifest, so it no longer accepts --opw-manifest. Its help, its last message, and the documentation still tell to move the clients back before deleting the cluster.
…den files The operator now sets the env vars and the metastore PostgreSQL connections that the CloudPrem Helm chart sets by default, so the converter no longer copies them. The chart's max_num_concurrent_split_searches no longer matches the operator default and is kept.
shuheiktgw
force-pushed
the
shuheiktgw/byoc_migrate_cli
branch
from
November 13, 2026 09:41
0db4978 to
4093610
Compare
It builds the binaries for Linux and macOS, uploads them to a folder named after the commit and to latest, and prints the commands that install them.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds
kubectl-byoc-migrate, a kubectl plugin that migrates the objects of thecloudpremandobservability-pipelines-workerHelm charts to a DatadogBYOCCluster. It reads the objects from manifest files, ashelm templateorhelm get manifestprints them.convertprints the DatadogBYOCCluster that matches the manifests.diffcompares the resources that the cluster renders with the objects of the manifests.adoptgives the CloudPrem objects to the live cluster in place, so the data, the PersistentVolumeClaims and the Service addresses are kept. Each pipeline runs a new Worker next to the objects of its OPW manifest; the read-only Metastore is recreated next to the one of the chart.abortremoves the cluster from the owners of the CloudPrem objects, after checking that the OPW StatefulSets of the manifests run their Pods again, and checks that no object still references the cluster.The plugin does not create, change or delete the DatadogBYOCCluster: the user applies and deletes it, for example from their deployment pipeline.
adoptandabortcheck that the user of the context may patch each CloudPrem object (SelfSubjectAccessReview, askubectl auth can-i) and write nothing otherwise.Operator changes:
adoptgives the fields of the adopted objects to it.gossip_protocol_version, so that the cluster keeps the version it runs with.The user guide is
docs/byoc_helm_migration.md.Motivation
Let CloudPrem customers who installed the Helm charts move to a DatadogBYOCCluster without losing data and without an ingestion outage.
Additional Notes
This PR is not intended to be merged upstream, as the CLI is more of a one-off script for migrating customers, and we don't plan to maintain it long-term.
Checklist
bug,enhancement,refactoring,documentation,tooling, and/ordependenciesqa/skip-qalabel