test: (scriptless) Enable scriptless phase 3 in AB e2es by lilypan26 · Pull Request #8453 · Azure/AgentBaker

lilypan26 · 2026-05-05T16:25:30Z

What this PR does / why we need it:

Enables scriptless phase 3 in ab e2es under tag EnableScriptlessANC
- This tag means both ANC and NBC cse cmd will be provided, tests should validate that there are no diffs between the generated provisioning env vars
- For now, NBC cse cmd will be used for provisioning until there are no more diffs, after which we can switch to using ANC
Adds CustomDataPhase3 to provide both ANC and NBC cse cmd to AKS node controller

Which issue(s) this PR fixes:

Fixes #

…scriptless/phase-3-e2e

Copilot

Pull request overview

Enables “scriptless phase 3” coverage in the AgentBaker e2e suite by adding a new scriptless_anc subtest path that provisions nodes using AKSNodeConfig/aks-node-controller, plus wiring many existing scenarios to provide an AKSNodeConfigMutator.

Changes:

Added a new scriptless_anc subtest variant and runtime flag (EnableScriptlessANC) to drive scriptless phase-3 execution.
Refactored/expanded the e2e “aks-node-controller hack” customData generation to optionally include AKSNodeConfig and/or an nbc-cmd script.
Updated many scenarios to set equivalent AKSNodeConfigMutator fields alongside existing NBC mutators.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

File	Description
e2e/vmss.go	Refactors customData hack generation and wires scriptless ANC + NBC cmd hack paths into VMSS creation.
e2e/types.go	Adds `EnableScriptlessANC` and adjusts kubelet-config-file detection logic for scriptless ANC scenarios.
e2e/test_helpers.go	Adds `scriptless_anc` subtest generation and new gating helper.
e2e/scenario_test.go	Adds `AKSNodeConfigMutator` coverage across many existing scenarios.

…scriptless/phase-3-e2e

Co-authored-by: lilypan26 <106703606+lilypan26@users.noreply.github.com>

…e/AgentBaker into lily/scriptless/phase-3-e2e

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 20 out of 23 changed files in this pull request and generated 2 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 20 out of 23 changed files in this pull request and generated 4 comments.

…e/AgentBaker into lily/scriptless/phase-3-e2e

Devinwong · 2026-05-27T23:31:12Z

-    [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia-container-runtime.options]
-      BinaryName = "/usr/bin/nvidia-container-runtime"
-      SystemdCgroup = true
+  [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia-container-runtime.options]


I don't know why there is no icm on this before your change? maybe no one is using containerd.cri.v1?

Devinwong · 2026-05-27T23:32:48Z

  snapshotter = "tardev"
  runtime_type = "io.containerd.kata-cc.v2"
  privileged_without_host_devices = true
+  pod_annotations = ["io.katacontainers.*"]


Many indents error are related to Kata. I am wondering if kata is never in production or these configs are never used.

Devinwong · 2026-05-28T23:00:01Z

-			FeatureGates:          map[string]bool{},
-			FailSwapOn:            to.Ptr(false),
+			FeatureGates: map[string]bool{
+				"RotateKubeletServerCertificate": true,


We already have this set to true in KubeletFlags["--feature-gates".
If we've set EnableKubeletConfigFile: false,, do we need this?

Devinwong · 2026-05-28T23:08:27Z

 }

-const sysctlTemplateString = `# This is a partial workaround to this upstream Kubernetes issue:
-# https://github.com/kubernetes/kubernetes/issues/41916#issuecomment-312428731


No one dared to remove this although it's 9 years ago. 😂
Indeed, do you know if the issue no longer exists?

Devinwong · 2026-05-28T23:12:15Z

 				nbc.IsARM64 = true
 			},
+			AKSNodeConfigMutator: func(_ *Cluster, config *aksnodeconfigv1.Configuration) {
+				config.VmSize = "Standard_D2pds_v6"


Why the vmsize needs explicit set for some cases only?

awesomenix · 2026-05-29T21:01:32Z

 		"IDENTITY_BINDINGS_LOCAL_AUTHORITY_SNI":        config.GetServiceAccountImagePullProfile().GetLocalAuthoritySni(),
 		"CSE_TIMEOUT":                                  getCSETimeout(config),
-		"SKIP_WAAGENT_HOLD":                            "true",
+		"SKIP_WAAGENT_HOLD":                            "false",


this needs to be true, because scriptless mode we always never hold waagent

Yep this would be a difference compared to non scriptless path, may be we can have a list of keys which are ok to ignore

awesomenix · 2026-05-29T21:13:39Z

 	VM                        *ScenarioVM
 	VMSSName                  string
 	EnableScriptlessNBCCSECmd bool
+	EnableScriptlessANC       bool


Do we need a new flag. Because RP needs to provide aksnodeconfig input somewhere which means we are changing the signature of GetBootstrappingData(). We will probably be switching over to aks-node-controller.utils.CustomData instead based on some feature flag.

lilypan26 added 4 commits April 28, 2026 10:54

enable aksnodeconfig in all e2es

042c0f4

Merge branch 'main' of https://github.com/Azure/AgentBaker into lily/…

ffa89ce

…scriptless/phase-3-e2e

refactor customdatahack and get e2e passing

2c493c8

fix scriptless vs nonscriptless e2e logic

1bb406d

Copilot AI review requested due to automatic review settings May 5, 2026 16:25

lilypan26 requested review from AbelHu, Devinwong, SriHarsha001, awesomenix, calvin197, cameronmeissner, djsly, ganeshkumarashok, junjiezhang1997, mxj220, pdamianov-dev, phealy, r2k1, sulixu, surajssd, timmy-wright and zachary-bailey as code owners May 5, 2026 16:25

lilypan26 temporarily deployed to test May 5, 2026 16:25 — with GitHub Actions Inactive

lilypan26 changed the title ~~Lily/scriptless/phase 3 e2e~~ test(scriptless): Enable scriptless phase 3 in AB e2es May 5, 2026

Copilot started reviewing on behalf of lilypan26 May 5, 2026 16:26 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

Comment thread e2e/vmss.go Outdated

Comment thread e2e/vmss.go Outdated

Comment thread e2e/vmss.go Outdated

Comment thread e2e/vmss.go

Comment thread e2e/test_helpers.go

Comment thread e2e/vmss.go

restore

02b46dc

lilypan26 temporarily deployed to test May 6, 2026 00:17 — with GitHub Actions Inactive

lilypan26 added 2 commits May 10, 2026 13:38

Merge branch 'main' of https://github.com/Azure/AgentBaker into lily/…

85702a0

…scriptless/phase-3-e2e

add phase 3 custom data and fix e2e logic

dba6573

Copilot started work on behalf of lilypan26 May 27, 2026 20:59 View session

fix: propagate gzip helper errors in e2e custom data

e7c279c

Co-authored-by: lilypan26 <106703606+lilypan26@users.noreply.github.com>

Copilot AI review requested due to automatic review settings May 27, 2026 21:04

lilypan26 review requested due to automatic review settings May 27, 2026 21:04

Copilot finished work on behalf of lilypan26 May 27, 2026 21:08

fix: gzip phase3 custom data payload

f4adeb9

Co-authored-by: lilypan26 <106703606+lilypan26@users.noreply.github.com>

Copilot AI review requested due to automatic review settings May 27, 2026 21:09

lilypan26 review requested due to automatic review settings May 27, 2026 21:09

Copilot finished work on behalf of lilypan26 May 27, 2026 21:09

lilypan26 added 3 commits May 27, 2026 14:22

fix http proxy vars quotations

50768f5

Merge branch 'lily/scriptless/phase-3-e2e' of https://github.com/Azur…

428dbcf

…e/AgentBaker into lily/scriptless/phase-3-e2e

address comments

d4ed38a

Copilot AI review requested due to automatic review settings May 27, 2026 22:15

Copilot started reviewing on behalf of lilypan26 May 27, 2026 22:15 View session

Potential fix for pull request finding

2ef3c72

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI reviewed May 27, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/cse_cmd.sh

Comment thread e2e/validators.go Outdated

improve env compare validation

425c7ff

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings May 27, 2026 22:41

Copilot started reviewing on behalf of lilypan26 May 27, 2026 22:42 View session

Copilot AI reviewed May 27, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/cse_cmd.sh

Comment thread e2e/vmss.go

Comment thread aks-node-controller/pkg/nodeconfigutils/utils.go

Comment thread e2e/test_helpers.go

lilypan26 added 3 commits May 27, 2026 15:54

fix proxy var comapre

442c525

do not log values

2126d61

Merge branch 'lily/scriptless/phase-3-e2e' of https://github.com/Azur…

c6675f0

…e/AgentBaker into lily/scriptless/phase-3-e2e

Devinwong reviewed May 27, 2026

View reviewed changes

Devinwong reviewed May 28, 2026

View reviewed changes

awesomenix reviewed May 29, 2026

View reviewed changes

Conversation

lilypan26 commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Devinwong May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Devinwong May 27, 2026

Choose a reason for hiding this comment

Uh oh!

Devinwong May 28, 2026

Choose a reason for hiding this comment

Uh oh!

Devinwong May 28, 2026

Choose a reason for hiding this comment

Uh oh!

Devinwong May 28, 2026

Choose a reason for hiding this comment

Uh oh!

awesomenix May 29, 2026

Choose a reason for hiding this comment

Uh oh!

awesomenix May 29, 2026

Choose a reason for hiding this comment

Uh oh!

awesomenix May 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

lilypan26 commented May 5, 2026 •

edited

Loading

Devinwong May 27, 2026 •

edited

Loading