Skip to content

[issue-labeler] Use env var for prediction threshold; default to 0.05#124435

Merged
jeffhandley merged 2 commits intodotnet:mainfrom
jeffhandley:jeffhandley/issue-labeler-threshold
Feb 16, 2026
Merged

[issue-labeler] Use env var for prediction threshold; default to 0.05#124435
jeffhandley merged 2 commits intodotnet:mainfrom
jeffhandley:jeffhandley/issue-labeler-threshold

Conversation

@jeffhandley
Copy link
Member

We ran several training runs of the issue-labeler to evaluate different prediction thresholds. With input from several team members and a vote, we are lowering the threshold down to 0.05 to minimize the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners occurrences.

This PR adds support for configuring the threshold via an ISSUE_LABELER_PREDICTION_THRESHOLD actions env var, applying the 0.05 value as the default for this repo.

Threshold Matches Mismatches No Prediction
1️⃣ 0.05 (test run) 91.91% (42,866) 8.06% (3,760) 0.03% (13)
2️⃣ 0.10 (test run) 91.71% (42,770) 7.81% (3,640) 0.48% (224)
3️⃣ 0.15 (test run) 91.21% (42,520) 7.40% (3,451) 1.39% (646)
4️⃣ 0.20 (test run) 90.40% (42,140) 6.69% (3,117) 2.92% (1,359)
5️⃣ 0.28 (test run) 88.61% (41,307) 5.41% (2,521) 5.98% (2,788)
6️⃣ 0.40 (test run) [current setting] 84.59% (39,431) 3.62% (1,687) 11.79% (5,498)

…SSUE_LABELER_PREDICTION_THRESHOLD

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeffhandley jeffhandley self-assigned this Feb 15, 2026
@jeffhandley jeffhandley requested a review from a team as a code owner February 15, 2026 00:26
Copilot AI review requested due to automatic review settings February 15, 2026 00:26
@dotnet-policy-service
Copy link
Contributor

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes the issue-labeler prediction threshold configurable and lowers the default from 0.40 to 0.05 based on evaluation of multiple training runs. The change aims to minimize occurrences of issues/PRs needing manual area-label assignment.

Changes:

  • Makes prediction threshold configurable via repository variable ISSUE_LABELER_PREDICTION_THRESHOLD
  • Changes default threshold from 0.40 to 0.05
  • Updates three labeler workflow files consistently

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
.github/workflows/labeler-train.yml Updated THRESHOLD to use configurable vars.ISSUE_LABELER_PREDICTION_THRESHOLD with 0.05 default
.github/workflows/labeler-predict-pulls.yml Updated THRESHOLD to use configurable vars.ISSUE_LABELER_PREDICTION_THRESHOLD with 0.05 default
.github/workflows/labeler-predict-issues.yml Updated THRESHOLD to use configurable vars.ISSUE_LABELER_PREDICTION_THRESHOLD with 0.05 default

…beler.md

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeffhandley jeffhandley merged commit 393c9c4 into dotnet:main Feb 16, 2026
17 checks passed
@jeffhandley jeffhandley deleted the jeffhandley/issue-labeler-threshold branch February 16, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants