CORS-4336: Add CI jobs for AWS European Sovereign Cloud (EUSC) by liweinan · Pull Request #75568 · openshift/release

liweinan · 2026-03-02T18:04:40Z

Implement continuous integration support for AWS EUSC partition (aws-eusc) in eusc-de-east-1 region. Includes cluster profile definition, service endpoints configuration, custom AMI handling, and periodic test jobs.

This enables OpenShift testing on AWS's new European Sovereign Cloud infrastructure, which requires explicit endpoint configuration and custom RHCOS AMIs not available in public regions.

openshift-ci-robot · 2026-03-02T18:04:45Z

@liweinan: This pull request references CORS-4336 which is a valid jira issue.

Details

In response to this:

Implement continuous integration support for AWS EUSC partition (aws-eusc) in eusc-de-east-1 region. Includes cluster profile definition, service endpoints configuration, custom AMI handling, and periodic test jobs.

This enables OpenShift testing on AWS's new European Sovereign Cloud infrastructure, which requires explicit endpoint configuration and custom RHCOS AMIs not available in public regions.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2026-03-02T18:06:05Z

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[125]: invalid cluster profile "aws-eusc-qe"
 * tests[126]: invalid cluster profile "aws-eusc-qe"

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml

ci-operator/step-registry/cluster-profiles/cluster-profiles-config.yaml

...ipi/private/provision/cucushift-installer-rehearse-aws-eusc-ipi-private-provision-chain.yaml

ci-operator/step-registry/ipi/conf/aws/eusc-ami/ipi-conf-aws-eusc-ami-commands.sh

yunjiang29 · 2026-03-05T02:26:01Z

@liweinan as we discussed offline, for the new partition we need three types of cluster:

common IPI cluster
private cluster
disconnected (private) cluster
and based on above basic cluster, we also need to cover STS, custom KMS key, FIPS and minimum permission, you can refer to existing jobs.

...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml

liweinan · 2026-03-05T02:38:24Z

@yunjiang29 Thanks for the review! I'll refactor this PR today.

openshift-ci-robot · 2026-03-05T12:18:26Z

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[126]: invalid cluster profile "aws-eusc"
 * tests[127]: invalid cluster profile "aws-eusc"

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

liweinan · 2026-03-06T03:16:23Z

@yunjiang29 Thanks for the detailed review! I'll update the PR recordingly.

openshift-ci-robot · 2026-03-06T06:37:31Z

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[126]: invalid cluster profile "aws-eusc"
 * tests[127]: invalid cluster profile "aws-eusc"

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

Address yunfei's review comments on PR openshift#75568: 1. Job naming convention: - Rename jobs from -f60 to -f7 suffix (non-destructive tests) - Update cron schedule to standard f7 pattern: 7,14,23,30 2. Private cluster configuration: - Add complete private cluster setup with bastion host - Add VPC, security groups, and proxy configuration - Set PUBLISH=Internal for private cluster access - Add minimal IAM permission provisioning - Follow pattern from cucushift-installer-rehearse-aws-ipi-private-provision 3. AMI configuration fix: - Replace deprecated compute.platform.aws.amiID field - Use platform.aws.defaultMachinePlatform.amiID instead

openshift-ci-robot · 2026-03-06T06:40:37Z

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[126]: invalid cluster profile "aws-eusc"
 * tests[127]: invalid cluster profile "aws-eusc"

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

1. Job naming convention: - Rename jobs from -f60 to -f7 suffix (non-destructive tests) - Update cron schedule to standard f7 pattern: 7,14,23,30 2. Private cluster configuration: - Add complete private cluster setup with bastion host - Add VPC, security groups, and proxy configuration - Set PUBLISH=Internal for private cluster access - Add minimal IAM permission provisioning - Follow pattern from cucushift-installer-rehearse-aws-ipi-private-provision 3. AMI configuration fix: - Replace deprecated compute.platform.aws.amiID field - Use platform.aws.defaultMachinePlatform.amiID instead 4. Generalize step registry components for reusability: - Enhance ipi-conf-aws-custom-endpoints to support multiple AWS partitions * Add AWS_DOMAIN_SUFFIX env var (defaults to amazonaws.com) * Support amazonaws.eu for EUSC, amazonaws.com.cn for China * Allow full URLs for maximum flexibility - Make ipi-conf-aws-eusc-ami more generic * Support AWS_CUSTOM_AMI_ID for general use * Maintain AWS_EUSC_AMI_ID for backward compatibility * Can be used for EUSC, China, GovCloud, or custom AMI scenarios - Use generic steps in EUSC provision chain with partition-specific config - Remove obsolete ipi-conf-aws-eusc-endpoints (replaced by generic version)

openshift-ci-robot · 2026-03-06T07:00:15Z

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[126]: invalid cluster profile "aws-eusc"
 * tests[127]: invalid cluster profile "aws-eusc"

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

...erator/step-registry/ipi/conf/aws/custom-endpoints/ipi-conf-aws-custom-endpoints-commands.sh

1. Job naming convention: - Rename jobs from -f60 to -f7 suffix (non-destructive tests) - Update cron schedule to standard f7 pattern: 7,14,23,30 2. Private cluster configuration: - Add complete private cluster setup with bastion host - Add VPC, security groups, and proxy configuration - Set PUBLISH=Internal for private cluster access - Add minimal IAM permission provisioning - Follow pattern from cucushift-installer-rehearse-aws-ipi-private-provision 3. Generalize step registry components for maximum reusability: a) Enhance ipi-conf-aws-custom-endpoints for all AWS partitions: - Add AWS_DOMAIN_SUFFIX env var (defaults to amazonaws.com) - Support amazonaws.eu (EUSC), amazonaws.com.cn (China) - Allow full URLs for maximum flexibility - Remove obsolete ipi-conf-aws-eusc-endpoints step b) Extend ipi-conf-aws to support custom AMI configuration: - Add AWS_AMI_ID env var for custom RHCOS AMI - Useful for EUSC, China, GovCloud, or any partition without public AMIs - Fix deprecated amiID field -> defaultMachinePlatform.amiID - Auto-detection still works for C2S/SC2S - Remove obsolete ipi-conf-aws-eusc-ami step c) EUSC provision chain now uses only generic steps with env config This refactoring reduces code duplication (net -59 lines) and makes step components reusable across all AWS partitions.

openshift-ci-robot · 2026-03-06T07:12:44Z

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[126]: invalid cluster profile "aws-eusc"
 * tests[127]: invalid cluster profile "aws-eusc"

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

liweinan · 2026-03-10T13:59:36Z

Relative PRs merged: #75441 / openshift/ci-tools#4973

liweinan · 2026-03-21T20:37:54Z

cluster destroyed:

vagrant@10:~/works$ ./openshift-install destroy cluster
INFO Credentials loaded from the AWS config using "SharedConfigCredentials: /home/vagrant/.aws/credentials" provider
INFO elbv2 endpoint is empty, using elb endpoint: https://elasticloadbalancing.eusc-de-east-1.amazonaws.eu
INFO Terminated                                    instance=i-0027fb4f664a14756
INFO Deleted                                       id=net/weli-eusc-s92nv-int/e62ff1a531385316/48eb2b4b76fb6b5b resourceType=listener
INFO Deleted                                       id=apiserver-target-dvsq7/fc5408d474c9cb57 resourceType=targetgroup
INFO Deleted                                       id=weli-eusc-s92nv-cloud-credential-operator-iam-ro-28fln policy=weli-eusc-s92nv-cloud-credential-operator-iam-ro-28fln-policy
INFO Deleted                                       id=weli-eusc-s92nv-cloud-credential-operator-iam-ro-28fln
INFO Disassociated                                 id=weli-eusc-s92nv-worker-profile name=weli-eusc-s92nv-worker-profile role=weli-eusc-s92nv-worker-role
INFO Deleted                                       InstanceProfileName=weli-eusc-s92nv-worker-profile arn=arn:aws-eusc:iam::082250599274:instance-profile/weli-eusc-s92nv-worker-profile id=weli-eusc-s92nv-worker-profile
INFO Deleted                                       id=weli-eusc-s92nv-openshift-machine-api-aws-7fg92 policy=weli-eusc-s92nv-openshift-machine-api-aws-7fg92-policy
INFO Deleted                                       id=weli-eusc-s92nv-openshift-machine-api-aws-7fg92
INFO Deleted                                       id=net/weli-eusc-s92nv-int/e62ff1a531385316 resourceType=loadbalancer
INFO Deleted                                       id=weli-eusc-s92nv-openshift-cloud-network-config-contro-l85qw policy=weli-eusc-s92nv-openshift-cloud-network-config-contro-l85qw-policy
INFO Deleted                                       id=weli-eusc-s92nv-openshift-cloud-network-config-contro-l85qw
INFO Deleted                                       id=net/weli-eusc-s92nv-ext/e23a4a340de1c8b0 resourceType=loadbalancer
INFO Not found or already deleted                  id=net/weli-eusc-s92nv-ext/e23a4a340de1c8b0/d467a0b1f4d6fd63 resourceType=listener
INFO Deleted                                       id=nat-0ab41a56f2a9286a9 resourceType=natgateway
INFO Deleted                                       id=weli-eusc-s92nv-master-role name=weli-eusc-s92nv-master-role policy=weli-eusc-s92nv-master-policy
INFO Disassociated                                 id=weli-eusc-s92nv-master-role name=weli-eusc-s92nv-master-profile role=weli-eusc-s92nv-master-role
INFO Deleted                                       InstanceProfileName=weli-eusc-s92nv-master-profile arn=arn:aws-eusc:iam::082250599274:instance-profile/weli-eusc-s92nv-master-profile id=weli-eusc-s92nv-master-role name=weli-eusc-s92nv-master-role
INFO Deleted                                       id=weli-eusc-s92nv-master-role name=weli-eusc-s92nv-master-role
INFO Deleted                                       id=weli-eusc-s92nv-openshift-image-registry-bbqng policy=weli-eusc-s92nv-openshift-image-registry-bbqng-policy
INFO Deleted                                       id=weli-eusc-s92nv-openshift-image-registry-bbqng
INFO Disassociated                                 id=rtbassoc-09d02f92b6d4172e7 resourceType=route-table
INFO Deleted                                       id=rtb-0fcb44fa352346c64 resourceType=route-table
INFO Deleted                                       id=weli-eusc-s92nv-aws-ebs-csi-driver-operator-wfj2l policy=weli-eusc-s92nv-aws-ebs-csi-driver-operator-wfj2l-policy
INFO Deleted                                       id=weli-eusc-s92nv-aws-ebs-csi-driver-operator-wfj2l
INFO Disassociated                                 id=rtbassoc-0d9990447edbb515d resourceType=route-table
INFO Deleted                                       id=rtb-029c5de9d3be5cd24 resourceType=route-table
INFO Not found or already deleted                  id=net/weli-eusc-s92nv-int/e62ff1a531385316/a936d3a16b432d3e resourceType=listener
INFO Deleted                                       id=sg-0b3c4db8b7c06fa4d resourceType=security-group
INFO Deleted                                       id=apiserver-target-r9flz/417b4a988db3dce4 resourceType=targetgroup
INFO Deleted                                       id=weli-eusc-s92nv-openshift-ingress-vbxnn policy=weli-eusc-s92nv-openshift-ingress-vbxnn-policy
INFO Deleted                                       id=weli-eusc-s92nv-openshift-ingress-vbxnn
INFO Deleted                                       id=subnet-0016e00f58695ffac resourceType=subnet
INFO Deleted                                       id=weli-eusc-s92nv-worker-role name=weli-eusc-s92nv-worker-role policy=weli-eusc-s92nv-worker-policy
INFO Deleted                                       id=weli-eusc-s92nv-worker-role name=weli-eusc-s92nv-worker-role
INFO Deleted
INFO Deleted                                       classic load balancer=a2028d926bfdf4d1c86ac8a24131ffc3 id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       NAT gateway=nat-0ab41a56f2a9286a9 id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       NAT gateway=nat-0187425548704793e id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       id=sg-076c19d61a09ab930 resourceType=security-group
INFO Deleted                                       id=rtb-0e40882cab21e13f6 resourceType=route-table
INFO Deleted                                       id=subnet-0d46147c3e680416a resourceType=subnet
INFO Deleted                                       id=nat-0187425548704793e resourceType=natgateway
INFO Deleted                                       id=additional-listener-v9699/76c7b6bd09fb46d3 resourceType=targetgroup
INFO Disassociated                                 id=rtbassoc-0326cc88c13d95250 resourceType=route-table
INFO Deleted                                       id=rtb-057870fbf836380d9 resourceType=route-table
INFO Deleted                                       id=a2028d926bfdf4d1c86ac8a24131ffc3 resourceType=loadbalancer
INFO Deleted                                       id=vpce-02f160f39bedbac0e resourceType=vpc-endpoint
WARNING could not determine whether hosted zone is private  hosted zone=weli-eusc.ci-eusc.devcluster.openshift.com. id=Z06024201CZSGSQHJJ3CY
INFO Deleted                                       id=Z06024201CZSGSQHJJ3CY record set=A api-int.weli-eusc.ci-eusc.devcluster.openshift.com.
INFO Deleted                                       id=Z06024201CZSGSQHJJ3CY public zone=/hostedzone/Z04057337TKVHDWNA7XB record set=A api.weli-eusc.ci-eusc.devcluster.openshift.com.
INFO Deleted                                       id=Z06024201CZSGSQHJJ3CY public zone=/hostedzone/Z04057337TKVHDWNA7XB record set=A \052.apps.weli-eusc.ci-eusc.devcluster.openshift.com.
WARNING could not determine whether hosted zone is private  hosted zone=weli-eusc.ci-eusc.devcluster.openshift.com. id=Z06024201CZSGSQHJJ3CY
INFO Deleted                                       id=Z06024201CZSGSQHJJ3CY record set=A api.weli-eusc.ci-eusc.devcluster.openshift.com.
INFO Deleted                                       id=Z06024201CZSGSQHJJ3CY record set=A \052.apps.weli-eusc.ci-eusc.devcluster.openshift.com.
INFO Deleted                                       id=Z06024201CZSGSQHJJ3CY
INFO Released                                      id=eipalloc-0ddd9a3d2e94b9df4 resourceType=elastic-ip
INFO Deleted                                       id=subnet-0b806b5c7b9e2def9 resourceType=subnet
INFO Deleted                                       id=sg-02738de8acbe8ca39 resourceType=security-group
INFO Deleted                                       id=sg-0031209cdc8d80186 resourceType=security-group
INFO Deleted                                       id=sg-0847162268706887c resourceType=security-group
INFO Deleted                                       NAT gateway=nat-0ab41a56f2a9286a9 id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       NAT gateway=nat-0187425548704793e id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       id=vpc-05f0a100aa18a5abe resourceType=vpc subnet=subnet-0b0d11cbc3ee3e7f8
INFO Released                                      id=eipalloc-0fc6a065c3622a91d resourceType=elastic-ip
INFO Deleted                                       NAT gateway=nat-0ab41a56f2a9286a9 id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       NAT gateway=nat-0187425548704793e id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       id=igw-0ee6005c1878801f2 resourceType=internet-gateway
INFO Deleted                                       NAT gateway=nat-0ab41a56f2a9286a9 id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       NAT gateway=nat-0187425548704793e id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Time elapsed: 5m46s
INFO Uninstallation complete!
vagrant@10:~/works$

...t/openshift-tests-private/openshift-openshift-tests-private-release-4.23__multi-nightly.yaml

...ft/openshift-tests-private/openshift-openshift-tests-private-release-5.0__multi-nightly.yaml

...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__multi-nightly.yaml

yunjiang29 · 2026-03-23T05:24:19Z

@tthvo's initial EUSC PR has merged, so we might actually be able to rehearse this, if credentials are setup:

I guess the credentials are not yet ready? The installer is asking for the access key and secrets in the ci/rehearse/openshift/installer/main/e2e-aws-eusc-techpreview.

level=warning msg=Found override for release image (registry.build10.ci.openshift.org/ci-op-n8lpz8gb/release@sha256:02c954c753c39c185094a8253214485c9dcd4aa17202bf680bb158d06db15032). Release Image Architecture is unknown
? AWS Access Key ID [? for help] �[?25l�[?25l�7�8�[?25h�[?25h? AWS Access Key ID [? for help] �[?25l�[?25l�7�8�[?25h�[?25h? AWS Access Key ID [? for help] �[?25l�[?25l�7�8�[?25h�[?25hlevel=error msg=failed to fetch Master Machines: failed to load asset "Install Config": failed to create install config: [platform.aws.region: Internal error: failed to get list of regions: failed to create EC2 client: EOF, controlPlane.platform.aws: Internal error: failed to create EC2 client: EOF, compute[0].platform.aws: Internal error: failed to create EC2 client: EOF]
Create manifests exit code: 3

@yunjiang29 Do you have any idea on this?

We can see Unsupported cluster type 'aws-eusc' in the build log, aws-eusc is new cluster type, but install steps can't recognize it, it needs to be added to

release/ci-operator/step-registry/ipi/install/install/aws/ipi-install-install-aws-commands.sh

Line 233 in a8b5667

aws|aws-arm64|aws-usgov)

and

release/ci-operator/step-registry/ipi/install/install/ipi-install-install-commands.sh

Line 621 in a8b5667

aws|aws-arm64|aws-usgov)

yunjiang29 · 2026-03-23T05:29:42Z

@tthvo @patrickdillon btw, this PR is not ready for testing because it hasn't configured AMI_ID yet.

ami-0c5e051fb7dc39f8d is ready for test, you can config it in job config via:

CONTROL_PLANE_AMI: ami-0c5e051fb7dc39f8d
COMPUTE_AMI: ami-0c5e051fb7dc39f8d

liweinan · 2026-03-23T10:28:07Z

ami-0b78302f83217d149

After discussing with Yunfei locally, we decided to use the AMI noted here: #75568 (comment)

@yunjiang29

This refactors the AWS European Sovereign Cloud (EUSC) CI configuration to maximize reuse of standard AWS workflows and reduce maintenance burden. Changes based on @yunjiang29's review feedback: - Reduced from 9 to 6 jobs following the pattern: 3 cluster types × 2 test types - Improved FIPS coverage from 1/9 (11%) to 2/6 (33%) jobs: * aws-eusc-ipi-fips-f7 (IPI + FIPS) * aws-eusc-ipi-private-sts-fips-f7 (Private + STS + FIPS) - Combined features across jobs: * aws-eusc-ipi-f28-destructive (destructive testing) * aws-eusc-ipi-private-mini-perm-f28 (Private + minimal permissions) * aws-eusc-ipi-disc-priv-kms-f7 (Disconnected + KMS) * aws-eusc-ipi-disc-priv-f28 (Disconnected destructive) - All jobs cover: FIPS, STS, KMS, minimal permissions across 3 cluster types - Deleted 15 EUSC-specific files, created 8 new ones (net reduction: -7 files) - Maximized reuse of standard AWS workflows: * Basic IPI: reuses cucushift-installer-rehearse-aws-ipi-deprovision * Private: reuses cucushift-installer-rehearse-aws-ipi-private-deprovision * Disconnected: reuses cucushift-installer-rehearse-aws-ipi-disconnected-private-provision * Private-STS: reuses cucushift-installer-rehearse-aws-ipi-private-cco-manual-security-token-service - EUSC-specific changes limited to: * Inserting ipi-conf-aws-custom-endpoints ref for service endpoint configuration * Custom provision chain for disconnected-private-kms (combines disconnected + KMS) - Deleted all EUSC-specific deprovision chains (reuse standard chains) - Removed unnecessary byo-kms and STS specific directory structures 1. Custom endpoints (ipi-conf-aws-custom-endpoints-commands.sh): - Removed auto-detection logic for AWS_DOMAIN_SUFFIX - Simplified to use environment variable or default to "amazonaws.com" - Removed Route53 endpoint configuration (global service) - Designed for easy removal when installer adds native EUSC support 2. AMI configuration (ipi-conf-aws-commands.sh): - Simplified from split variables (CONTROL_PLANE_AMI/COMPUTE_AMI) to single CONTROL_PLANE_AMI - Preserved C2S/SC2S auto-detection logic - Removed complex heredoc patching, kept simple approach - Updated documentation for clarity 1. **Minimize EUSC-specific code**: Only 8 workflow files vs 15 previously 2. **Maximize standard workflow reuse**: Follows USGov pattern, not C2S pattern 3. **Prepare for future evolution**: Custom endpoints easy to remove when installer supports EUSC natively 4. **FIPS coverage aligned with USGov**: 33% vs USGov's 18%, not C2S's 100% - make update completed successfully - All 6 jobs generated in ci-operator/jobs/.../periodics.yaml - Step registry validation passed Addresses: openshift#75568

liweinan · 2026-03-23T11:02:41Z

@yunjiang29 Thanks for the detailed review! I have updated the PR accordingly.

@patrickdillon @tthvo As openshift/cluster-ingress-operator#1360 is verified locally: #75568 (comment), after that PR is merged, I guess we can use the job here for testing then.

...nshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22-periodics.yaml

patrickdillon · 2026-03-23T15:11:16Z

Rehearsal run shows that the credentials are not being found:

https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/75568/rehearse-75568-pull-ci-openshift-installer-main-e2e-aws-eusc-techpreview/2035055582121037824

@yunjiang29

This refactors the AWS European Sovereign Cloud (EUSC) CI configuration to maximize reuse of standard AWS workflows and reduce maintenance burden. Changes based on @yunjiang29's review feedback: - Reduced from 9 to 6 jobs following the pattern: 3 cluster types × 2 test types - Improved FIPS coverage from 1/9 (11%) to 2/6 (33%) jobs: * aws-eusc-ipi-fips-f7 (IPI + FIPS) * aws-eusc-ipi-private-sts-fips-f7 (Private + STS + FIPS) - Combined features across jobs: * aws-eusc-ipi-f28-destructive (destructive testing) * aws-eusc-ipi-private-mini-perm-f28 (Private + minimal permissions) * aws-eusc-ipi-disc-priv-kms-f7 (Disconnected + KMS) * aws-eusc-ipi-disc-priv-f28 (Disconnected destructive) - All jobs cover: FIPS, STS, KMS, minimal permissions across 3 cluster types - Deleted 15 EUSC-specific files, created 8 new ones (net reduction: -7 files) - Maximized reuse of standard AWS workflows: * Basic IPI: reuses cucushift-installer-rehearse-aws-ipi-deprovision * Private: reuses cucushift-installer-rehearse-aws-ipi-private-deprovision * Disconnected: reuses cucushift-installer-rehearse-aws-ipi-disconnected-private-provision * Private-STS: reuses cucushift-installer-rehearse-aws-ipi-private-cco-manual-security-token-service - EUSC-specific changes limited to: * Inserting ipi-conf-aws-custom-endpoints ref for service endpoint configuration * Custom provision chain for disconnected-private-kms (combines disconnected + KMS) - Deleted all EUSC-specific deprovision chains (reuse standard chains) - Removed unnecessary byo-kms and STS specific directory structures 1. Custom endpoints (ipi-conf-aws-custom-endpoints-commands.sh): - Removed auto-detection logic for AWS_DOMAIN_SUFFIX - Simplified to use environment variable or default to "amazonaws.com" - Removed Route53 endpoint configuration (global service) - Designed for easy removal when installer adds native EUSC support 2. AMI configuration (ipi-conf-aws-commands.sh): - Simplified from split variables (CONTROL_PLANE_AMI/COMPUTE_AMI) to single CONTROL_PLANE_AMI - Preserved C2S/SC2S auto-detection logic - Removed complex heredoc patching, kept simple approach - Updated documentation for clarity 1. **Minimize EUSC-specific code**: Only 8 workflow files vs 15 previously 2. **Maximize standard workflow reuse**: Follows USGov pattern, not C2S pattern 3. **Prepare for future evolution**: Custom endpoints easy to remove when installer supports EUSC natively 4. **FIPS coverage aligned with USGov**: 33% vs USGov's 18%, not C2S's 100% - make update completed successfully - All 6 jobs generated in ci-operator/jobs/.../periodics.yaml - Step registry validation passed Addresses: openshift#75568

liweinan · 2026-03-24T02:01:46Z

@tthvo's initial EUSC PR has merged, so we might actually be able to rehearse this, if credentials are setup:

I guess the credentials are not yet ready? The installer is asking for the access key and secrets in the ci/rehearse/openshift/installer/main/e2e-aws-eusc-techpreview.
level=warning msg=Found override for release image (registry.build10.ci.openshift.org/ci-op-n8lpz8gb/release@sha256:02c954c753c39c185094a8253214485c9dcd4aa17202bf680bb158d06db15032). Release Image Architecture is unknown
? AWS Access Key ID [? for help] �[?25l�[?25l�7�8�[?25h�[?25h? AWS Access Key ID [? for help] �[?25l�[?25l�7�8�[?25h�[?25h? AWS Access Key ID [? for help] �[?25l�[?25l�7�8�[?25h�[?25hlevel=error msg=failed to fetch Master Machines: failed to load asset "Install Config": failed to create install config: [platform.aws.region: Internal error: failed to get list of regions: failed to create EC2 client: EOF, controlPlane.platform.aws: Internal error: failed to create EC2 client: EOF, compute[0].platform.aws: Internal error: failed to create EC2 client: EOF]
Create manifests exit code: 3
@yunjiang29 Do you have any idea on this?
We can see Unsupported cluster type 'aws-eusc' in the build log, aws-eusc is new cluster type, but install steps can't recognize it, it needs to be added to

release/ci-operator/step-registry/ipi/install/install/aws/ipi-install-install-aws-commands.sh

Line 233 in a8b5667

aws|aws-arm64|aws-usgov)

and

release/ci-operator/step-registry/ipi/install/install/ipi-install-install-commands.sh

Line 621 in a8b5667

aws|aws-arm64|aws-usgov)

@patrickdillon This change should have fixed the problem. I'll trigger a rehearse test to verify it.

@yunjiang29

This refactors the AWS European Sovereign Cloud (EUSC) CI configuration to maximize reuse of standard AWS workflows and reduce maintenance burden. Changes based on @yunjiang29's review feedback: - Reduced from 9 to 6 jobs following the pattern: 3 cluster types × 2 test types - Improved FIPS coverage from 1/9 (11%) to 2/6 (33%) jobs: * aws-eusc-ipi-fips-f7 (IPI + FIPS) * aws-eusc-ipi-private-sts-fips-f7 (Private + STS + FIPS) - Combined features across jobs: * aws-eusc-ipi-f28-destructive (destructive testing) * aws-eusc-ipi-private-mini-perm-f28 (Private + minimal permissions) * aws-eusc-ipi-disc-priv-kms-f7 (Disconnected + KMS) * aws-eusc-ipi-disc-priv-f28 (Disconnected destructive) - All jobs cover: FIPS, STS, KMS, minimal permissions across 3 cluster types - Deleted 15 EUSC-specific files, created 8 new ones (net reduction: -7 files) - Maximized reuse of standard AWS workflows: * Basic IPI: reuses cucushift-installer-rehearse-aws-ipi-deprovision * Private: reuses cucushift-installer-rehearse-aws-ipi-private-deprovision * Disconnected: reuses cucushift-installer-rehearse-aws-ipi-disconnected-private-provision * Private-STS: reuses cucushift-installer-rehearse-aws-ipi-private-cco-manual-security-token-service - EUSC-specific changes limited to: * Inserting ipi-conf-aws-custom-endpoints ref for service endpoint configuration * Custom provision chain for disconnected-private-kms (combines disconnected + KMS) - Deleted all EUSC-specific deprovision chains (reuse standard chains) - Removed unnecessary byo-kms and STS specific directory structures 1. Custom endpoints (ipi-conf-aws-custom-endpoints-commands.sh): - Removed auto-detection logic for AWS_DOMAIN_SUFFIX - Simplified to use environment variable or default to "amazonaws.com" - Removed Route53 endpoint configuration (global service) - Designed for easy removal when installer adds native EUSC support 2. AMI configuration (ipi-conf-aws-commands.sh): - Simplified from split variables (CONTROL_PLANE_AMI/COMPUTE_AMI) to single CONTROL_PLANE_AMI - Preserved C2S/SC2S auto-detection logic - Removed complex heredoc patching, kept simple approach - Updated documentation for clarity 1. **Minimize EUSC-specific code**: Only 8 workflow files vs 15 previously 2. **Maximize standard workflow reuse**: Follows USGov pattern, not C2S pattern 3. **Prepare for future evolution**: Custom endpoints easy to remove when installer supports EUSC natively 4. **FIPS coverage aligned with USGov**: 33% vs USGov's 18%, not C2S's 100% - make update completed successfully - All 6 jobs generated in ci-operator/jobs/.../periodics.yaml - Step registry validation passed Addresses: openshift#75568

liweinan · 2026-03-24T06:53:44Z

The previous conflicts are caused by the new commits in the main branch. I have rebased the PR for several rounds, and I'll rebase the branch after all the other issues are fixed.

This PR adds comprehensive CI support for AWS EUSC (European Sovereign Cloud) partition, including: - Add aws-eusc cluster type support in install scripts: - ipi-install-install-aws-commands.sh - ipi-install-install-commands.sh - ipi-conf-aws-commands.sh - Add aws-eusc cluster profile configuration - Add EUSC periodic jobs for openshift-tests-private (4.22, 4.23, 5.0): - 8 EUSC IPI jobs per version with various configurations - Hash-based cron schedule distribution to avoid resource contention - AMI configuration: ami-0b78302f83217d149 - Custom DNS jobs with qe.gcp.devcluster.openshift.com domain - Add EUSC installer rehearse jobs (4.22, 4.23, 5.0, main): - IPI installation workflow - FEATURE_SET: TechPreviewNoUpgrade All generated job files updated via make update.

openshift-ci-robot · 2026-03-24T12:06:01Z

[REHEARSALNOTIFIER]
@liweinan: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-openshift-assisted-installer-agent-master-okd-scos-e2e-aws-ovn	openshift/assisted-installer-agent	presubmit	Registry content changed
pull-ci-openshift-assisted-installer-agent-release-4.21-okd-scos-e2e-aws-ovn	openshift/assisted-installer-agent	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-master-okd-scos-e2e-aws-ovn	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.21-okd-scos-e2e-aws-ovn	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-master-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-5.0-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.23-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.22-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.21-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.20-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.19-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.18-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.17-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.16-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.15-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.14-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.13-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.12-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.11-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.10-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.9-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.8-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.7-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-release-4.6-e2e-aws-csi	openshift/aws-ebs-csi-driver	presubmit	Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-master-e2e-aws-csi-extended	openshift/aws-ebs-csi-driver	presubmit	Registry content changed

A total of 32145 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 2, 2026

openshift-ci bot requested review from neisw and xingxingxia March 2, 2026 18:05