ci(android): run Gutenberg-module instrumented tests#472
Open
ci(android): run Gutenberg-module instrumented tests#472
Conversation
github-actions
Bot
added
the
[Type] Build Tooling
jkmassel
force-pushed
the
jkmassel/wire-gutenberg-instrumentation
branch
5 times, most recently
from
908aa16 to
a8eb19b
Compare
Runs `:Gutenberg:connectedDebugAndroidTest` on the `mac-metal` queue and uploads results to Buildkite Test Engine (suite: `gutenbergkit-android-library-e2e`). ## Wiring - `buildSrc` helper `configureBuildkiteTestCollector` registers the Buildkite JUnit listener when `BUILDKITE_ANALYTICS_TOKEN_ANDROID_LIBRARY_E2E` is present. Taking `testInstrumentationRunnerArguments` as a plain `MutableMap<String, String>` keeps `buildSrc` free of an AGP dependency — avoids a plugin-classpath conflict with each module's `alias(libs.plugins.android.*)`. - `:app` module migrated to the same helper with its existing `BUILDKITE_ANALYTICS_TOKEN_ANDROID_E2E` token. - `:Gutenberg:androidTest` uses `kotlin.test.*` — consistent with `src/test/`. JUnit 4's `assertEquals(message, expected, actual)` is a footgun versus `kotlin.test.assertEquals(expected, actual, message)`. - Deletes the `ExampleInstrumentedTest` placeholder. ## Fail-loud instead of silent green The collector library silently swallows config and upload errors — a missing token makes the listener inert; HTTP errors only log. Without extra wiring the suite goes green while Test Engine receives nothing. - `configureBuildkiteTestCollector` fails configuration on Buildkite (detected via `BUILDKITE_BUILD_ID`) if the token is missing or blank. - `BUILDKITE_ANALYTICS_DEBUG_ENABLED=true` so the happy path also emits traces, not just errors. - `test-android-library-e2e` clears logcat, runs the gradle task, dumps device logcat to `build/outputs/buildkite-logs/device-logcat.txt` (uploaded as an artifact), and greps for the library's `*-ERROR` log prefixes. Any on-device upload error fails the step. - Test APK manifest declares `android.permission.INTERNET`, scoped to `androidTest/` so published-library consumers don't inherit it. Without it the library logs `Error uploading test analytics data: Permission denied (missing INTERNET permission?).` Verified end-to-end: Test Engine run `f3f3afb4-efa4-876c-ba05-4a4c9621d64a` received on build 2042.
jkmassel
force-pushed
the
jkmassel/wire-gutenberg-instrumentation
branch
from
a8eb19b to
ba4d724
Compare
Contributor
Author
|
e2e test failures will be resolved by #473 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test-android-library-e2emake target that runs:Gutenberg:connectedDebugAndroidTest, and wires it into Buildkite as a new:android: Test Android Library Instrumentedstep on themac-metalqueue — parallel to the existing:appE2E step.instrumented-test-collector, gated on the newBUILDKITE_ANALYTICS_TOKEN_ANDROID_LIBRARY_E2Esecret so timings and flakes show up in the dashboard alongside the existing E2E suite.:appand:Gutenberginto abuildSrchelper, and adopts it in both modules.:Gutenbergmodule's androidTest assertions fromorg.junit.Assert.*tokotlin.test.*, and addsandroidTestImplementation(kotlin("test")). This resolves the pre-existing failures that lighting up the CI job exposed.Context
The
:Gutenbergmodule has carried instrumented test sources underandroidTest/since #367, but no CI job has ever executed them —make test-android-e2eonly covers:app:connectedDebugAndroidTest. Turning them on here surfaced failures inorg.wordpress.gutenberg.http.InstrumentedFixtureTests(multipartParsingErrorCases,requestParsingErrorCases) that this PR is the right place to fix.New renderer instrumented tests from #470 will also get coverage once both PRs land.
Root Cause
The failures produced comparisons that looked wrong but weren't:
InstrumentedFixtureTestsimported JUnit 4'sorg.junit.Assert.assertEquals, whose signature is(message, expected, actual)— message first. Twocatchblocks called it in Jupiter/kotlin.test order(expected, actual, message), so the description string landed in theactualslot and got compared against anerrorId. The parser behavior is fine.Fix
Rather than just reordering the two bad call sites, migrate the androidTest sources to
kotlin.test.*. That matches what a Kotlin author naturally writes ((expected, actual, message)/(actual, message)), so the bug can't re-occur the same way.android/buildSrc/src/main/kotlin/BuildkiteTestCollector.kt: New top-levelconfigureBuildkiteTestCollector(args: MutableMap<String, String>, tokenEnvName: String). Registers the listener, forwards Buildkite metadata env vars, and is inert when the token isn't set. Takes thetestInstrumentationRunnerArgumentsmap as a parameter rather than extending an AGP type — this keepsbuildSrcAGP-free and avoids a plugin-classpath conflict with each module'salias(libs.plugins.android.*)application.android/Gutenberg/build.gradle.kts/android/app/build.gradle.kts: Call the helper fromdefaultConfig {}withtestInstrumentationRunnerArgumentsand their respective token env vars (_ANDROID_LIBRARY_E2Eand_ANDROID_E2E). Replaces ~23 lines of duplicated config in each.android/Gutenberg/build.gradle.kts: AddsandroidTestImplementation(kotlin("test")).kotlin.testis already on the JVMtestImplementationclasspath; this just adds it for androidTest.InstrumentedFixtureTests.kt/ExampleInstrumentedTest.kt: Swaporg.junit.Assert.*→kotlin.test.*and reorder all call sites. Also collapses a fewassertNotNull(...); x!!pairs intoval x = assertNotNull(...)since kotlin.test'sassertNotNullreturns the non-null value.AndroidX's JUnit 4 test runner is unchanged — that's what
AndroidJUnitRunnerdrives; only the assertion style changes. JVM unit tests undersrc/test/already usekotlin.testand are untouched.Commits
ci(android): run Gutenberg-module instrumented tests— new Buildkite step, Test Engine collector wiring, andbuildSrchelper adopted by both Android modules.refactor(android): migrate androidTest assertions to kotlin.test— the assertion migration that fixes the failures.Test plan
:android: Test Android Library Instrumentedstep runs on this PR and passes.:appE2E step still passes in parallel — Test Engine results should still show up for it under its existing suite, since the helper is functionally equivalent to the inline block it replaced.Android Library E2Esuite.