The metal-operator test framework provides a set of tests that can be used to verify that a given BMC and server are compatible with the metal-operator. Its goal is to enable BMC vendors or users to run these tests against a BMC with minimal effort and more easily determine what capabilities their BMC may be missing or implementing incorrectly.
Refer to the capability table for a list of capabilities and their descriptions. Refer to the coverage document for what tests cover which capabilities. Determine what your BMC might not implement by referring to a failed test.
The tests directory contains a suite of tests based on chainsaw. Every test case creates k8s resources in steps and asserts their status before proceeding. Tests are independent of from the infrastructure they run on and respect KUBECONFIG.
- chainsaw
- A metal-operator installation and BMC to run tests against. This repository usually uses the locally virtualised metal-lab to develop or verify functionality of the actual tests. You can use the lab setup as a reference.
The server to run a test against is configured by passing a values file to chainsaw via VALUES. To try the tests without using your own BMC, clone the metal-lab repository and refer to its README to bring it up.
# Examples
export KUBECONFIG=/path/to/metal-lab/kubeconfig.yaml
make test VALUES=/path/to/metal-lab/values-containerlab-node1.yaml # Run all tests
make test/01-bmc-registration VALUES=/path/to/metal-lab/values-containerlab-node1.yaml # Run a specific test
make test/03-power-annotation VALUES=/path/to/your-bmc.yaml # Run against your own BMCTo run tests against your own BMC, copy these values into a new file:
bmcIP: "172.16.100.11"
bmcPort: 443
bmcScheme: https
username: admin
password: password
firmwareVersion: "1.0.0"
biosVersion: "1.0.0"
01-bmc-registration:
model: "Standard PC (Q35 + ICH9, 2009)"
powerState: "On"
06-biossettings-noreboot:
key: "AssetTag"
value: "compat-test-06"
baselineValue: "compat-base"
07-biossettings-reboot:
key: "BootMode"
value: "Bios"
baselineValue: "Uefi"
08-bmcsettings:
key: "EmailAlert.1.Address"
value: "alerts@example.com"
baselineValue: "compat-base@example.com"
12-force-power-off: # optionally override the defaults
osBootWait: 120 # time to sleep before assuming the boot has reached the OS
powerPollingTimeout: 15 # must equal the metal-operator's --power-polling-timeout to the the Job
forceOffDelay: 90 # seconds allowed beyond powerPollingTimeout for the ForceOff to take effect
13-event-subscriptions:
testMessageId: "TestEventId" # a MessageId your BMC accepts in SubmitTestEvent, e.g. "SYS1000" on Dell iDRAC
testSeverity: "OK"
testOriginOfCondition: "/redfish/v1/Managers/1"
14-biosversion-upgrade:
transferProtocol: HTTP
baselineImageURI: "http://repo.example.com/bios-1.0.0.bin" # The URL to the current BIOS image. Will be flashed at least once during the test
version: "1.1.0"
imageURI: "http://repo.example.com/bios-1.1.0.bin" # The URL to the new BIOS version that will be flashed during the test
15-bmcversion-upgrade:
transferProtocol: HTTP
baselineImageURI: "http://repo.example.com/bmc-1.0.0.bin" # The URL to the current BMC firmware image. Will be flashed at least once during the test
version: "1.1.0"
imageURI: "http://repo.example.com/bmc-1.1.0.bin" # The URL to the new BMC firmware imageAdjust the credentials and expectations to your respective values and point the tests to it:
make ASSERT_TIMEOUT=2m VALUES=/path/to/new/file.yaml test/01-bmc-registrationYou will probably encounter an error like:
chainsaw test --values /home/user/src/metal-operator-test-framework/values-t160.yaml --parallel 1 --assert-timeout 2m tests/01-bmc-registration
Version: 0.2.15
Loading default configuration...
- Using test file: chainsaw-test
- TestDirs [tests/01-bmc-registration]
- Quiet false
- SkipDelete false
- FailFast false
- Namespace ''
- FastNamespaceDeletion false
- FullName false
- IncludeTestRegex ''
- ExcludeTestRegex ''
- ApplyTimeout 5s
- AssertTimeout 2m0s
- CleanupTimeout 30s
- DeleteTimeout 15s
- ErrorTimeout 30s
- ExecTimeout 5s
- DeletionPropagationPolicy Background
- Parallel 1
- Values [/home/user/src/metal-operator-test-framework/values-t160.yaml]
- Template true
- NoCluster false
- PauseOnFailure false
Loading tests...
- 01-bmc-registration (tests/01-bmc-registration)
Loading values...
Running tests...
=== RUN chainsaw
=== PAUSE chainsaw
=== CONT chainsaw
=== RUN chainsaw/01-bmc-registration
| 06:44:56 | 01-bmc-registration | @chainsaw | CREATE | OK | v1/Namespace @ chainsaw-elegant-puma
| 06:44:56 | 01-bmc-registration | register-bmc | TRY | BEGIN |
| 06:44:56 | 01-bmc-registration | register-bmc | APPLY | RUN | metal.ironcore.dev/v1alpha1/BMCSecret @ compatibility-chainsaw-elegant-puma
| 06:44:56 | 01-bmc-registration | register-bmc | CREATE | OK | metal.ironcore.dev/v1alpha1/BMCSecret @ compatibility-chainsaw-elegant-puma
| 06:44:56 | 01-bmc-registration | register-bmc | APPLY | DONE | metal.ironcore.dev/v1alpha1/BMCSecret @ compatibility-chainsaw-elegant-puma
| 06:44:56 | 01-bmc-registration | register-bmc | APPLY | RUN | metal.ironcore.dev/v1alpha1/BMC @ compatibility-chainsaw-elegant-puma
| 06:44:56 | 01-bmc-registration | register-bmc | CREATE | OK | metal.ironcore.dev/v1alpha1/BMC @ compatibility-chainsaw-elegant-puma
| 06:44:56 | 01-bmc-registration | register-bmc | APPLY | DONE | metal.ironcore.dev/v1alpha1/BMC @ compatibility-chainsaw-elegant-puma
| 06:44:56 | 01-bmc-registration | register-bmc | TRY | END |
| 06:44:56 | 01-bmc-registration | assert-bmc-enabled | TRY | BEGIN |
| 06:44:56 | 01-bmc-registration | assert-bmc-enabled | ASSERT | RUN | metal.ironcore.dev/v1alpha1/BMC @ compatibility-chainsaw-elegant-puma
| 06:46:56 | 01-bmc-registration | assert-bmc-enabled | ASSERT | ERROR | metal.ironcore.dev/v1alpha1/BMC @ compatibility-chainsaw-elegant-puma
=== ERROR
-------------------------------------------------------------------
metal.ironcore.dev/v1alpha1/BMC/compatibility-chainsaw-elegant-puma
-------------------------------------------------------------------
* status.firmwareVersion: Invalid value: "7.30.10.50": Expected value: "1.0.0"
* status.model: Invalid value: "16G Monolithic": Expected value: "Standard PC (Q35 + ICH9, 2009)"
--- expected
+++ actual
@@ -3,8 +3,8 @@
metadata:
name: compatibility-chainsaw-elegant-puma
status:
- firmwareVersion: 1.0.0
- model: Standard PC (Q35 + ICH9, 2009)
+ firmwareVersion: 7.30.10.50
+ model: 16G Monolithic
powerState: "On"
state: Enabled
| 06:46:56 | 01-bmc-registration | assert-bmc-enabled | TRY | END |
| 06:46:56 | 01-bmc-registration | register-bmc | CLEANUP | BEGIN |
| 06:46:56 | 01-bmc-registration | register-bmc | DELETE | OK | metal.ironcore.dev/v1alpha1/BMC @ compatibility-chainsaw-elegant-puma
| 06:46:57 | 01-bmc-registration | register-bmc | DELETE | OK | metal.ironcore.dev/v1alpha1/BMCSecret @ compatibility-chainsaw-elegant-puma
| 06:46:57 | 01-bmc-registration | register-bmc | CLEANUP | END |
| 06:46:57 | 01-bmc-registration | @chainsaw | CLEANUP | BEGIN |
| 06:46:57 | 01-bmc-registration | @chainsaw | DELETE | OK | v1/Namespace @ chainsaw-elegant-puma
| 06:47:02 | 01-bmc-registration | @chainsaw | CLEANUP | END |
--- FAIL: chainsaw (125.32s)
--- FAIL: chainsaw/01-bmc-registration (125.32s)
FAIL
Tests Summary...
- Passed tests 0
- Failed tests 1
- Skipped tests 0
Done with failures.
Error: some tests failed
make: *** [Makefile:59: test/01-bmc-registration] Error 1A mismatch like this means the expectations in your VALUES file need to be adjusted to reality.