Add sealed ETL denial analysis (replacement for #699) by richiemsft · Pull Request #709 · microsoft/mxc

richiemsft · 2026-07-30T05:36:04Z

📖 Description

Replacement for #699, which was mistakenly marked merged when the parent
branch temporarily received child-stack history during a stack-handling
mistake. This PR restores the intended denial-analysis review boundary after
merged Integration PR #696.

Adds the denial-analysis layer following #696:

Introduces the cross-platform learning_mode_core denial model, summary, analysis, framing, and emission primitives.
Adds the Windows sealed-ETL decoder using OpenTrace/ProcessTrace and TDH metadata.
Decodes and normalizes file, registry, UI, and capability-related Learning Mode events.
Caches manifest TDH schemas per trace while bypassing the cache for self-describing TraceLogging events.
Contains all Rust panics inside the ETW callback boundary and resumes them only after the trace handle is closed.
Bounds property counts, struct depth, total decode work, metadata references, and payload reads.
Skips malformed individual payload events while preserving fatal schema/API errors.
Handles fixed-width unsupported InTypes without losing payload alignment.
Uses deterministic synthetic drive maps for path-normalization tests.
Streams raw diagnostics and adds lm_analyze as a developer-only trace diagnostic.

PR #696 has merged. This PR now targets main directly.

🔗 References

🔍 Validation

cargo test -p learning_mode_core
cargo test -p learning_mode_windows --all-targets — 82 tests
cargo clippy -p learning_mode_core -p learning_mode_windows --all-targets -- -D warnings
Feature-enabled V1 Windows VM 172.17.7.153:
- captured a fresh 2,713-byte ETL
- production analysis exited 0 and emitted 9 unique denials
- raw diagnostic traversal exited 0

✅ Checklist

Signed the Contributor License Agreement
Linked to an issue
Updated documentation (if applicable)
Updated Copilot instructions (if build, architecture, or conventions changed)
If this PR changes Cargo.lock, the dependency-feed-check check passes (see docs/pull-requests.md)

📋 Issue Type

Bug fix
Feature
Task

GitHub Actions runs the PR validation build automatically. The ADO pipeline
(MXC-PR-Build) is the Azure version of the PR pipeline, kept in parity with the GitHub
Actions build; it runs on merge to main, and Microsoft reviewers with write access can trigger it
on a PR with /azp run. See docs/pull-requests.md.

azure-pipelines · 2026-07-30T05:36:13Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Pull request overview

Adds cross-platform denial primitives and Windows sealed-ETL analysis for Learning Mode captures.

Changes:

Adds denial models, summaries, framing, and emission.
Implements bounded ETL decoding, extraction, normalization, and deduplication.
Adds the Windows-only lm_analyze diagnostic.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`src/core/learning_mode_core/src/summary.rs`	Defines capture summaries.
`src/core/learning_mode_core/src/model.rs`	Defines denial wire models.
`src/core/learning_mode_core/src/lib.rs`	Exposes the core API.
`src/core/learning_mode_core/src/frame.rs`	Defines framed records.
`src/core/learning_mode_core/src/emit.rs`	Emits RFC 7464 streams.
`src/core/learning_mode_core/src/analyze.rs`	Defines analyzer abstractions.
`src/core/learning_mode_core/Cargo.toml`	Configures the new crate.
`src/Cargo.toml`	Registers the workspace crate.
`src/Cargo.lock`	Records dependency changes.
`src/backends/learning_mode/windows/src/tdh_decode.rs`	Decodes TDH properties.
`src/backends/learning_mode/windows/src/path_norm.rs`	Normalizes Windows paths.
`src/backends/learning_mode/windows/src/lib.rs`	Exports analyzer APIs.
`src/backends/learning_mode/windows/src/extractors.rs`	Extracts typed denials.
`src/backends/learning_mode/windows/src/etl_decode.rs`	Processes sealed ETL files.
`src/backends/learning_mode/windows/examples/lm_analyze.rs`	Adds the diagnostic CLI.
`src/backends/learning_mode/windows/Cargo.toml`	Adds the core dependency.

richiemsft · 2026-07-30T18:07:18Z

/azp run

azure-pipelines · 2026-07-30T18:07:36Z

Azure Pipelines: Successfully started running 1 pipeline(s).

MGudgin

Adversarial review findings for the sealed ETL denial-analysis layer.

richiemsft · 2026-07-31T20:05:38Z

/azp run

azure-pipelines · 2026-07-31T20:05:55Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Introduce the learning_mode_core crate as the cross-platform hinge of the captureDenials pipeline (PR4/PR5 foundation): - model: DeniedResource / ResourceType (file|ui|network|capability|other) / AccessType, camelCase wire contract with round-trip + key-stability tests. - summary: DenialSummary terminator (exitCode, totalDenials, deniedResourcesTruncated). - frame: self-describing type-tagged DenialFrame (denial|summary). - emit: RFC 7464 (0x1E-framed) NDJSON stream writer over any io::Write. - analyze: DenialAnalyzer decode trait + AnalyzeError, implemented per-OS by the backend ETL decoders (Windows first). No OS-specific code; 16 unit tests + 1 doc-test, clippy/fmt clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Decode the learning-mode `.etl` that CaptureSession::finish seals into cross-platform learning_mode_core::DeniedResource values, implementing the DenialAnalyzer trait. - path_norm: kernel-form -> drive-letter, incl. the `\??\C:\...` / `\??\UNC\...` DOS-devices prefix the trace actually emits. - tdh_decode: TDH EVENT_RECORD -> (event_id, props). - extractors: route event 14/4907 (access check) by ObjectType (File -> file, Key -> registry/other, empty -> capability) and derive accessType from AccessMask with a File/Registry right vocabulary (Write > Execute > Read); event 27 -> Ui; event 28 -> capability denial (Denied gate, payload ProcessId). Missing/unparseable mask degrades to Unknown rather than dropping the denial. - etl_decode: file-mode OpenTraceW/ProcessTrace loop + dedup by (user-visible path, accessType). - examples/lm_analyze: NDJSON emit + `--raw` event dump for validation. Event vocabulary and AccessMask classification confirmed on the GE_CURRENT x64 VM against real block-and-log (Mode="Normal") and allow-and-log (Mode="Permissive") captures: File/registry denials decode identically in both modes; capability denials arrive as empty-ObjectType event 14 (permissive) or event 28 (block). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Factor the pure decode composition (extract_denial routing -> path normalisation -> dedup) out of EtlDenialAnalyzer::analyze into resources_from_events, and cover it with tests built from the event shapes captured on hardware for both learning modes: - block-and-log (Mode="Normal"): file/registry event 14 + capability event 28 - allow-and-log (Mode="Permissive"): capability folded into empty-ObjectType event 14 (no event 28) - empty-path capabilities from event 14 and event 28 collapse to one resource - non-actionable object types / not-denied records / unknown event IDs dropped A live ETW/TDH fixture read is intentionally not used: it needs the provider manifests registered on the machine, and a raw capture embeds token SIDs that don't belong in a public repo. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Allocate TRACE_EVENT_INFO storage with its required alignment and decode UTF-16 payloads from byte pairs so unaligned ETW data never forms invalid Rust references. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Stream raw ETL diagnostics, honor producer pointer width, normalize Windows path aliases, correct access masks and empty identifiers, and serialize FILETIME losslessly for JSON consumers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Describe the record-separator-framed output as an RFC 7464 JSON text sequence rather than NDJSON across the core API and diagnostic example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Normalize DOS devices, mapped drives, MUP redirectors, and DFS paths to absolute user-visible forms while omitting pipes, relative paths, and unknown device namespaces. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Document lm_analyze as a developer-only tool and explain why the sealed, policy-oriented learning-mode decoder does not directly reuse the diagnostic console's real-time display consumer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a8860a9-bb20-48ef-91a1-3da8e34b92fb

MGudgin

You'll need to retarget at main (which may then require new signoff, not sure...)

The base branch was changed.

richiemsft · 2026-08-01T02:07:50Z

@MGudgin, yup it resetted the sign-off.

richiemsft · 2026-08-01T02:07:56Z

/azp run

azure-pipelines · 2026-08-01T02:08:10Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a8860a9-bb20-48ef-91a1-3da8e34b92fb

richiemsft · 2026-08-03T16:31:38Z

/azp run

azure-pipelines · 2026-08-03T16:31:53Z

Azure Pipelines: Successfully started running 1 pipeline(s).

richiemsft requested a review from a team as a code owner July 30, 2026 05:36

This was referenced Jul 30, 2026

Emit captureDenials JSON output (replacement for #701) #710

Merged

Wire captureDenials to Windows Learning Mode runtime #696

Merged

richiemsft requested a review from Copilot July 30, 2026 05:42

Copilot started reviewing on behalf of richiemsft July 30, 2026 05:42 View session

Copilot AI reviewed Jul 30, 2026

View reviewed changes

Comment thread src/backends/learning_mode/windows/src/extractors.rs Outdated

Comment thread src/backends/learning_mode/windows/src/tdh_decode.rs Outdated

Comment thread src/backends/learning_mode/windows/examples/lm_analyze.rs Outdated

Comment thread src/backends/learning_mode/windows/Cargo.toml

MGudgin reviewed Jul 31, 2026

View reviewed changes

richiemsft force-pushed the user/saulg/learning-mode-analyze-replacement branch from f5c0f96 to e8cb988 Compare July 31, 2026 20:04

An error occurred while trying to automatically change base from user/saulg/learning-mode-endtoend to main July 31, 2026 23:57

richiemsft requested a review from MGudgin July 31, 2026 23:57

richiemsft and others added 13 commits July 31, 2026 17:00

Update decoder terminology for capture modes

780eca1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Update PR4 workspace lockfile

eb8d30f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Harden Learning Mode ETL analysis

1588939

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Address PR 709 review feedback

fbea9af

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a8860a9-bb20-48ef-91a1-3da8e34b92fb

Harden sealed ETL decoding

4491606

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a8860a9-bb20-48ef-91a1-3da8e34b92fb

richiemsft force-pushed the user/saulg/learning-mode-analyze-replacement branch 2 times, most recently from 4491606 to e8cb988 Compare August 1, 2026 00:06

MGudgin approved these changes Aug 1, 2026

View reviewed changes

MGudgin previously approved these changes Aug 1, 2026

View reviewed changes

richiemsft force-pushed the user/saulg/learning-mode-analyze-replacement branch from e8cb988 to 4491606 Compare August 1, 2026 02:05

richiemsft changed the base branch from user/saulg/learning-mode-endtoend to main August 1, 2026 02:06

Trigger GitHub Actions

74c410b

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a8860a9-bb20-48ef-91a1-3da8e34b92fb

MGudgin approved these changes Aug 3, 2026

View reviewed changes

richiemsft merged commit e8dcd48 into main Aug 3, 2026
40 checks passed

richiemsft deleted the user/saulg/learning-mode-analyze-replacement branch August 3, 2026 17:23

richiemsft mentioned this pull request Aug 3, 2026

Update Learning Mode to V2 API contracts #739

Open

8 tasks

Uh oh!

Conversation

richiemsft commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📖 Description

🔗 References

🔍 Validation

✅ Checklist

📋 Issue Type

Uh oh!

azure-pipelines Bot commented Jul 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

richiemsft commented Jul 30, 2026

Uh oh!

azure-pipelines Bot commented Jul 30, 2026

Uh oh!

MGudgin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

richiemsft commented Jul 31, 2026

Uh oh!

azure-pipelines Bot commented Jul 31, 2026

Uh oh!

MGudgin left a comment

Choose a reason for hiding this comment

Uh oh!

richiemsft commented Aug 1, 2026

Uh oh!

richiemsft commented Aug 1, 2026

Uh oh!

azure-pipelines Bot commented Aug 1, 2026

Uh oh!

richiemsft commented Aug 3, 2026

Uh oh!

azure-pipelines Bot commented Aug 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

richiemsft commented Jul 30, 2026 •

edited

Loading