Feat/sdk 7063 report skipped tests - #158
Merged
Merged
Conversation
Tests that never run because of `bail` were absent from the Test Run entirely. Report them as `skipped` so the report accounts for every declared test. WebdriverIO has two unrelated bail mechanisms, handled separately: - `mochaOpts.bail` halts the failing spec. `service.afterTest` walks `test.ctx.test.parent` to the spec's root suite and reuses the existing `reportSuiteSkipped` cascade, so sibling describes are covered. Gated strictly on mocha's own option -- under wdio's `bail` every in-spec test still runs, so triggering on it would report tests that are about to execute. - wdio's `bail: <n>` counts failed spec files and stops scheduling more. `launcher.onComplete` diffs dispatched specs against the run's spec list and enumerates the rest via mocha's declaration pass, without executing any test body. Launcher-side reporting only happens when bail demonstrably fired (`failedRunners >= bail`, counted the way wdio counts it -- a failed spec with retries left is requeued, not counted). It is skipped entirely under sharding, when capabilities define their own specs/exclude, and for excluded specs. A spec can fail to run for many reasons unrelated to bail, and reporting one of those as skipped would corrupt the report; under-reporting is recoverable, phantom skips are not. Events are relayed through the binary's EnqueueTestEvent RPC. The launcher's TESTHUB_JWT is the binary's account token on the CLI path and carries no build claim, so posting to the collector directly returns 401; the binary holds the build-scoped credential. The RPC already existed in the proto -- only the client wrapper was missing. Reported identity matches `getUniqueIdentifier` so a test reported skipped and the same test on a later run share one identity. Covers the CLI/gRPC path only. Enumerating never-started specs re-executes their top-level code.
Wire evidence showed a test that runs emits `name` and `identifier` as the bare title, with only `scope` carrying the parent: identifier: 'MA-TC1 passes' scope: 'Multi Spec A - MA-TC1 passes' Skipped tests reported for never-launched specs were setting `identifier` to the parent-qualified form, so the same test reported skipped and later run would not have matched on that field. Verified on build bpt0jvulwfonuoucepdu892xhzborork2hy7tgqb: executed and skipped tests now emit identical conventions for all three fields.
The customer's config is `bail: 0` with `mochaOpts.bail: true`, so only Mocha's in-spec bail applies to them. WebdriverIO's spec-file-level bail is disabled in their setup and the launcher-side reporting never runs. Drop the never-launched-spec work from this PR so the change that unblocks them is small and built on the already-proven `reportSuiteSkipped` path. That work is preserved on `feat/sdk-7063-unlaunched-specs` and can land separately on its own review. Removes: specEnumerator, unlaunchedSpecReporter, the launcher hooks, the EnqueueTestEvent client wrapper, and the mocha devDependency they needed. README and changeset narrowed to match.
`results.retries` only tracks wdio's spec-file retries. @wdio/utils builds
it as `{attempts: 0, limit: repeatTest}` and @wdio/mocha-framework never
feeds `mochaOpts.retries` into it, so under mocha-level retries it stays
`{0, 0}` and the guard evaluated `0 < 0` — it never suppressed anything.
The cascade therefore fired on the first failed attempt, before the retry
ran. When the retry then passed, those tests reported `skipped` and later
`passed` — two TestRunFinished events with different uuids for one test.
The dashboard looked correct only because TRA collapses same-name rows and
the later result wins.
Read mocha's own runnable state (`currentRetry()` vs `retries()`) and keep
the wdio check for spec-file retries.
Verified on build zynxn48dbs3fsbfhd975tghdrsj2550xdtwenqlm: zero `skipped`
emissions on the wire where there were previously three, dashboard 5/0/0.
hasRetryPending reads mocha's own runnable via currentRetry()/retries(). It ran before the try block, so a throw from either escaped uncaught into the awaited afterTest hook and would surface as a framework-level error in the user's run. Move the check inside the boundary that already wraps every other call in the method. The tests asserted only trackEvent call counts, so a cascade that swept the wrong tests still produced the expected total. Assert which titles were reported skipped as well — counts catch a test emitted twice, titles catch the wrong test emitted. Adds the two uncovered cases: the results.skipped gate branch, and a runnable whose retry accessors throw. A mocha root can span several spec files after all: MochaAdapter creates one Mocha and addFile()s every spec it is handed, so grouped specs share a root. Cascading across them stays correct, since bail aborts that whole runner. Correct the comment that claimed the opposite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rahulpsq
requested review from
osho-20 and
pranay-v29
and removed request for
a team
August 18, 2026 15:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is this about?
Related Jira task/s
Release (mandatory for every PR — required for the
ready-for-reviewlabel)Version bump: (required — tick exactly one)
Release notes type: (optional)
Release notes (customer-facing): (optional but encouraged)
Release notes (internal): (required — engineer-facing; what actually changed / why)
Checklist
PR Validations
Run Tests: Comment RUN_TESTS to trigger sanity tests.