Run Apex VS integration tests on maintainer-requested PRs by abonie · Pull Request #20625 · dotnet/fsharp

abonie · 2026-09-23T17:20:01Z

Run F# Apex integration tests in DartLab against a freshly installed Visual Studio, requested through /dart or /pr-val on PRs targeting main. Requests require repository write access and Microsoft-org membership, including for fork PRs.

Load pipeline definitions from trusted main, fetch source directly from GitHub, and reject PR merges that no longer match the captured head/base revisions. Pushes, PR updates and VS builds do not trigger runs automatically.

DevDiv registration, identity/resource authorization and lab-isolation approval are still required before live rollout.

…n step The azure-pipelines-PR.yml 'Detect CI VS Roslyn version' step and the FSHARP_APEX_ROSLYN_VERSION override in eng/Versions.props were committed, but the script itself was left untracked, so on CI 'powershell: eng\SetApexRoslynVersion.ps1' could not find the file and failed with 'The module ''eng'' could not be loaded'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fe2b3239-4176-4f13-b6f6-486c8799765a

main (via darc) bumped the dotnet/runtime System.* packages to 10.0.8 (assembly 10.0.0.8). The F# VSIX then references System.Composition.AttributedModel 10.0.0.8, which the CI scout VS (18.7, ships 8.0.0.0) cannot bind -- its binding-redirect ceiling is below 10.0.0.8 -- so every FSharp.Editor MEF part fails to load, IFSharpWorkspaceService is null, and all Apex tests fail with an NRE in LanguageService.fs. This is the same forward-pinned-dependency vs older-CI-VS problem already handled for Roslyn. Extend the FSHARP_APEX_ROSLYN_VERSION-gated override in eng/Versions.props to also roll System.Composition and System.Collections.Immutable back to 10.0.2 for the Apex leg only. Verified: with a simulated 10.0.8 base the override forces 10.0.2, and FSharp.Editor compiles and references System.Composition.AttributedModel 10.0.0.2. The committed pin and product/main builds are unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fe2b3239-4176-4f13-b6f6-486c8799765a

… leg" This reverts commit c0c3dd7.

Scaffolds an internal DevDiv DartLab pipeline (modeled on dotnet/roslyn's azure-pipelines-integration-dartlab.yml) that installs a matching Visual Studio via the VS bootstrapper on a VS-Platform test machine, deploys the F# VSIX, and runs the existing Apex tests. This is the robust fix for the version-skew that blocks running the VSIX on the older pre-baked public scout VS (the VS-SDK floor, e.g. RpcContracts requiring System.Composition >= 10.0.8, prevents downgrading to match it). Files (all trigger:none / inert until registered): - azure-pipelines-integration-dartlab.yml: entry, extends the VS real-sign template. - eng/pipelines/apex-integration/{stage,integration-job}.yml: provision + install VS + deploy VSIX + run Apex via the existing -testApex path. - eng/pipelines/apex-integration/README.md: external prerequisites + a concrete request to the DartLab/VS team. - eng/setup-pr-validation.ps1: PR-branch setup on the test machine (adapted from Roslyn). Every access-dependent value is marked '# TODO(P0):'. Requires DevDiv/DartLab access, an internal dotnet-fsharp mirror, and a VS-Platform pool before it can be registered and run; those are outside this repo. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fe2b3239-4176-4f13-b6f6-486c8799765a

…mirror The internal mirror already exists at dnceng/internal/dotnet-fsharp, so the FSharpMirror resource name (internal/dotnet-fsharp via dnceng-internal-code-access) is correct as-is. Update the TODO(P0) markers and README to note the mirror is confirmed; the only remaining mirror-related item is authorizing the service connection for the pipeline once registered. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fe2b3239-4176-4f13-b6f6-486c8799765a

Roslyn's azure-pipelines-integration-dartlab.yml evolved since the initial scaffold. Match it: - Add the VisualStudioBuildUnderTest pipeline resource (source: DD-CB-ReleaseVS, trigger: true) as the VS build whose drop supplies the bootstrapper. - Add the DevDiv/DartLab core repo resource (ref refs/tags/Production). - Extend /DartLab/1ES/pipelines/ci/build.yml@VSTemplates (Roslyn moved off real-sign.yml) and pass dartLabTemplatesRepositoryAlias: DartLab. - Enable the DownloadBuildArtifacts + Get-VisualStudioDropName steps in stage.yml so the '(default)' visualStudioBootstrapperURI resolves from the VS-build-under-test drop. - README: document the wired VS-build gate and note DD-CB-ReleaseVS must be confirmed for F#. Pool remains VS-Platform (matches Roslyn). Still gated on P0 access/registration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fe2b3239-4176-4f13-b6f6-486c8799765a

Per request, the DartLab VS-integration tests should run only on demand when an F# team member asks, mirroring dotnet/roslyn's /dart /pr-val flow. - azure-pipelines-integration-dartlab.yml: set the VisualStudioBuildUnderTest resource to trigger: none, so a new VS build supplies the matching VS drop but never auto-runs the pipeline. - .github/workflows/pr-validation.yml (new): on an issue_comment of /pr-val (or /dart), authorize the commenter (repo write access AND microsoft org membership), then trigger the DevDiv pipeline over an OIDC-authenticated Azure DevOps REST call with prNumber/sha/ EnforceLatestCommit. External-authored PRs must pass an explicit reviewed commit hash. Adapted from Roslyn's pr-validation.yml. - README: document the comment-trigger model and the extra P0 wiring (pipeline ID, OIDC secrets, fsharp_pr_validation environment). Pipeline ID and AZURE_* OIDC secrets are TODO(P0) until the pipeline is registered. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fe2b3239-4176-4f13-b6f6-486c8799765a

# Conflicts: # eng/Versions.props

DartLab installs a matching VS via bootstrapper, so the VSIX pins and the VS-under-test align by construction and no Roslyn/System.* override is needed. Remove the WindowsApexIntegration job from azure-pipelines-PR.yml, delete eng/SetApexRoslynVersion.ps1, and drop the now-dead FSHARP_APEX_ROSLYN_VERSION override group from eng/Versions.props. Update the DartLab README and integration-job comments accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

T-Gro

I propose the following to be added:

Explicitly invokable skill "collect VS tests errors" (i.e. model invocable=false), teaching models how to find the errors, logs etc.

Skill on authoring Apex tests for F#, pointing to both Apex info as well as listing various F# IDE features and pointing to file(s) with basic helpers ("framework" unifying how tests are written - just pointing to the right files)

Instruction triggering /vsintegration (or the even just the Editor sub path) hinting that any addition/change should consider test coverage, and listing the appropriate layers we have (ComponentTests, service tests, editor tests and the newly added Apex tests) - and that logic should be tested at the lowest layer possible (cheapest tests, best coverage across IDEs/OSes), but any new IDE feature should come with integration test smoke test to guard against regressions.

(preventing the most common and dangerous IDE scenarios - failed NGEN, .dll dependency mismatch, broken compatibility coming via a library etc)

abonie · 2026-09-24T08:59:53Z

Let's hold off with merging until we get a review from DartLab if we can

You are using the legacy DTL DartLab template. If you are in a rush to get this pipeline running then using this template for now is ok, but we are in the process of migrating all pipelines to DartLab1ES. The deadline for this is end of 2026

More info: https://dev.azure.com/devdiv/DevDiv/_wiki/wikis/DevDiv.wiki/50321/Migrate-a-DartLab-DTL-Test-Pipeline-to-DartLab-1ES-(CloudTest)

The template changes look good. I would wait to check it in until the new pipeline is created, and you have a successful run in case there are unexpected failures in scripts.

Thanks @nijulia - We don't want to migrate it again this year, so I changed it to DartLab1ES.

What value should be used for testMachineLab?

Thanks for doing that! You can use DotNet-Project-System-AMD64. You will need to create a ticket https://portal.microsofticm.com/imp/v3/incidents/create?tmpl=p2Pa3q to grant the permissions needed for your new pipeline.

Can it be done before merging this? In which case the required link to the pipeline would have to point to this PR?

Yes, this pool is ready and already available in production. I ended up not creating a designated FSharp lab since there was already a DotNet one.

abonie · 2026-09-24T09:43:10Z

I propose the following to be added:

Explicitly invokable skill "collect VS tests errors" (i.e. model invocable=false), teaching models how to find the errors, logs etc.

You mean from the DartLab run? Would be good, I won't know what to put in that skill until we actually have the DartLab wired up though.

Skill on authoring Apex tests for F#, pointing to both Apex info as well as listing various F# IDE features and pointing to file(s) with basic helpers ("framework" unifying how tests are written - just pointing to the right files)

Yeah, that would be good. I used Apex tests from VS repo as a direct reference, but can probably distill something into a skill.

github-actions · 2026-09-29T02:57:01Z

🔍 Tooling Safety Check — Affects-Build-Infra, Affects-Restore, Affects-Test-Tooling
Affects-Build-Infra: Changes test build projects.
Affects-Restore: Changes test package dependencies.
Affects-Test-Tooling: Changes test execution infrastructure.

Generated by PR Tooling Safety Check · gpt56 2.5M · ◷

Replace the legacy DTL stage contract with DartLab-1ES CloudTest operations while preserving trusted PR snapshot validation and the existing Apex build path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 34f3e808-c404-4337-8e8c-76fcea75b3ed

Use the CloudTest lab confirmed by the DartLab team and remove the provisional activation TODO. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 34f3e808-c404-4337-8e8c-76fcea75b3ed

Preserve the Apex test runner alongside main's TestFromManifest implementation and retain current dependency updates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 34f3e808-c404-4337-8e8c-76fcea75b3ed

abonie and others added 22 commits July 12, 2026 12:16

Add Code Action Apex tests based on TS

2ddb037

Copy other integration tests to apex

b6fdb0d

Add orchestration for CI and fix missing assertion

6c4ca07

Remove Apex tests from slnx and gate on Insiders

0c8b741

List VS instances

d1757c7

Pin Apex 18.7

f414963

Add more VS setup

2d4dbe9

Debug logging

d2fd5fd

Skip VS sign in

2a772b0

Start VS with an empty env

fccb0c7

Pin Roslyn packages differently for CI

19675db

Revert "Roll System.Composition/Immutable back to 10.0.2 for the Apex…

1f2d505

… leg" This reverts commit c0c3dd7.

Merge remote-tracking branch 'origin/main' into apex-int-tests

3fd75ac

# Conflicts: # eng/Versions.props

Drop stale P4/P5 note from Apex README (both already done)

8cf5f8a

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Run requested DartLab PR validation directly from GitHub

8efe122

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

abonie requested a review from a team as a code owner September 23, 2026 17:20

github-project-automation Bot added this to F# Compiler and Tooling Sep 23, 2026

github-project-automation Bot moved this to New in F# Compiler and Tooling Sep 23, 2026

github-actions Bot added ⚠️ Affects-Test-Tooling Tooling check: PR touches test framework infrastructure ⚠️ Affects-Build-Infra Tooling check: PR touches build infrastructure ⚠️ Affects-Restore Tooling check: PR touches NuGet packages or feeds labels Sep 23, 2026

This comment has been minimized.

Sign in to view

T-Gro reviewed Sep 24, 2026

View reviewed changes

T-Gro approved these changes Sep 24, 2026

View reviewed changes

github-project-automation Bot moved this from New to In Progress in F# Compiler and Tooling Sep 24, 2026

This comment has been minimized.

Sign in to view

abonie commented Sep 24, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

github-actions Bot added the AI-Tooling-Check-Scanned-Clean Tooling check: diff analyzed, no interesting infrastructure files label Sep 27, 2026

This comment has been minimized.

Sign in to view

abonie and others added 3 commits September 30, 2026 18:02

Set F# Apex CloudTest lab

997906c

Use the CloudTest lab confirmed by the DartLab team and remove the provisional activation TODO. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 34f3e808-c404-4337-8e8c-76fcea75b3ed

Merge upstream main into apex-dartlab-github

12b6261

Preserve the Apex test runner alongside main's TestFromManifest implementation and retain current dependency updates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 34f3e808-c404-4337-8e8c-76fcea75b3ed

Conversation

abonie commented Sep 23, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

T-Gro left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

abonie Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

nijulia Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nijulia Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

abonie Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

nijulia Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abonie Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

nijulia Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

abonie commented Sep 24, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions Bot commented Sep 29, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nijulia Sep 29, 2026 •

edited

Loading

nijulia Oct 2, 2026 •

edited

Loading