feat(scorecard): introduce scorecard dora plugin - #4235
Conversation
* Add github collectors Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add dora Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add workflow runs to github Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update dora provider schemas Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update dora to use scorecardCollectorsServiceRef Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update names Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Calculate all PRs between deployments Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update thresholds Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update deployment freq and lead time for changes names Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Use first commit Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add docs Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Simplify config Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Unify schemas Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add typings Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Validate deployments ascending order Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add description hover Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update docs Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Fix description Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * fix update to rebased new provider interface Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add df and mltc translations Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update gh collector descriptions Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update gh pr collector description Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Page commit shas Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Use warn Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Generate api reports Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Fix review comments for docs, link and readme Co-authored-by: Patrick Knight <pknight@redhat.com> Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update alpha api reports Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Handle null types Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Fix description Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Throw on not enough data Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add productionEnvironments option to provider, move under options Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Fix rename Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Restrict CollectorInput type Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Batch commit prs Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update to max 10 Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Rename deploymentRangePullRequests to deploymentPullRequests Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add app config examples Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Fix api report Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Simplify collector descriptions Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update api report Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Fix median lead time thresholds wording for translations Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Skip deployment interval if no PRs found Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> --------- Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> Co-authored-by: Patrick Knight <pknight@redhat.com>
* Add jira incident collector Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Introduce dora mttr and cfr Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * dora mttr and cfr name update Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Simplify config Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add tests Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Simplify since validated via collector schema Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add docs Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update median time to resolve to mean time to restore Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Rename resolutionDate to resolutionAt Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Remove median ttr Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * fix update jira to rebased new provider interface Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add cfr and mttr translations Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update jira description Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Simplify config setup and add prod env tp cft Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add examples to config Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Simplify collector description Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Use catalog filter Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update api reports Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Fail when not enough deployments for cfr Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Handle edge cases Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Use same format the result is for remaining dora metrics Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Fix links Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Page jira results Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update docs Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Support more Jira annotations Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Fix docs grammar Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add thresholds translations Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add fetchItemsLimit Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Throw error when no incident data for mttr Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add deployments cap Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update docs with cap Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Use GITHUB_BATCH_SIZE Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Add configurable incident issue type Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Move custom jql entity filter handling from jira client to respective providers and collectors Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update descriptions Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Move incidents collector setting under scorecard.plugins.jira Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Rename getIncidentIssues to getIssues Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Ignore blank app config mandatoryFilter for open issues Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Extract getAnnotationFiltersFromEntity from jira client Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Use clear dummy value Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Update indentation in README.md * Fix indentation README.md * Fix indentation in README.md Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Use isLast Co-authored-by: Patrick Knight <pknight@redhat.com> * Log reached gh fetch limit Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Move client to options Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Fix prettier Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> * Log warning for jira max fetch items Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> --------- Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com> Co-authored-by: Patrick Knight <pknight@redhat.com>
* feat: implement time series api, pass visualisation for metric Signed-off-by: Diana Janickova <djanicko@redhat.com> * fix: cast to number Signed-off-by: Diana Janickova <djanicko@redhat.com> * ref: rename graph to sparkline, use zod Signed-off-by: Diana Janickova <djanicko@redhat.com> * ref: add defaultVisualization to mcp action schemas Signed-off-by: Diana Janickova <djanicko@redhat.com> --------- Signed-off-by: Diana Janickova <djanicko@redhat.com>
Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com>
Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com>
Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com>
Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com>
Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com>
Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com>
|
Important This PR includes changes that affect public-facing API. Please ensure you are adding/updating documentation for new features or behavior. Changed Packages
|
|
🤖 Review · Commit: |
Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com>
|
🤖 Finished Review · ✅ Success · Started 5:40 PM UTC · Completed 6:00 PM UTC Commit: |
PR Summary by QodoAdd DORA scorecard backend module with collectors and time-series API
AI Description
Diagram
High-Level Assessment
Files changed (83)
|
Code Review by Qodo
1. Unsafe ID numeric compare
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4235 +/- ##
==========================================
+ Coverage 58.39% 58.64% +0.25%
==========================================
Files 2436 2466 +30
Lines 96935 97691 +756
Branches 26984 27105 +121
==========================================
+ Hits 56607 57294 +687
- Misses 40136 40205 +69
Partials 192 192
*This pull request uses carry forward flags. Click here to find out more. Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
| name: dora-scorecard | ||
| annotations: | ||
| github.com/project-slug: dzemanov/test-scorecard-github-dora | ||
| backstage.io/source-location: url:https://github.com/dzemanov/test-scorecard-github-dora |
There was a problem hiding this comment.
Will update source to official rhdh repo in following PR
ReviewFindingsHigh
Medium
Low
Next steps:
Previous runReviewFindingsMedium
Low
Labels: PR introduces a new DORA metrics plugin module with new API endpoints and collectors framework |
Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com>
Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com>
Assisted-By: Cursor Desktop Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com>
|
🤖 Review · Commit: |
Signed-off-by: Dominika Zemanovicova <dzemanov@redhat.com>
|
|
🤖 Finished Review · ✅ Success · Started 3:23 PM UTC · Completed 3:45 PM UTC Commit: |
| } | ||
| } | ||
|
|
||
| if (filterAnnotations.customFilter) { |
There was a problem hiding this comment.
[high] JQL Injection
The customFilter annotation value (from entity annotation jira/custom-filter) is passed through to the JQL query without any sanitization or validation. Unlike all other annotation fields (project, component, label, team) which go through sanitizeValue and validateJQLValue/validateIdentifier, the customFilter slot bypasses all sanitization. An attacker who can control entity annotations could inject arbitrary JQL.
Suggested fix: Either remove the customFilter annotation passthrough entirely, or apply validation to restrict its content.
| @@ -35,3 +35,42 @@ export function validateIdentifier(value: string, fieldName: string): string { | |||
| export function sanitizeValue(value: string): string { | |||
| return value.replace(/\\/g, '\\\\').replace(/"/g, '\\"'); | |||
| } | |||
There was a problem hiding this comment.
[low] JQL Injection
sanitizeValue is redundant given the strict allowlist regex in validateJQLValue. While safe, the redundancy makes the security contract confusing.
| entity: Entity, | ||
| ): string { | ||
| const from = toJiraDateTime(options.from); | ||
| const to = toJiraDateTime(options.to); |
There was a problem hiding this comment.
[low] JQL Injection
issueType in incidentsCollectorInputSchema lacks a JQL-safe regex constraint; validateJQLValue catches unsafe characters at JQL build time rather than at config parsing time.
| * Provider-specific options. | ||
| */ | ||
| options?: { | ||
| collectors?: { |
There was a problem hiding this comment.
[low] config / schema mismatch
config.d.ts declares thresholds for each DORA metric but parseDoraXxxConfig functions do not read them. ThresholdResolver may handle this independently.
| > | ||
| { | ||
| static readonly inputSchema = incidentsCollectorInputSchema; | ||
| static readonly outputSchema = incidentsCollectorOutputSchema; |
There was a problem hiding this comment.
[low] constructor-visibility
JiraIncidentsCollector uses a public constructor while GitHub collector classes use private constructors with static fromConfig() factories.



Hey, I just made a Pull Request!
Introduces new
backstage-plugin-scorecard-backend-module-dora.Incorporates PRs:
Additional changes:
issueTypeapp-config configuration under dora collector input in e7c248f, so configuration looks like this:Fixes
Fixes https://redhat.atlassian.net/browse/RHIDP-13982
Incorporates https://redhat.atlassian.net/browse/RHIDP-13978, https://redhat.atlassian.net/browse/RHIDP-13979, https://redhat.atlassian.net/browse/RHIDP-13980
DORA Metrics
dora.deploymentFrequency
(successfulProductionDeployments / 30) * 7(to deployments/week)Thresholds deploymentFrequency
Updated (medium to 1-7 to cover the whole real line):
dora.medianLeadTimeForChanges
MEDIAN for (deployedAtTimestamp - firstCommitAtTimestamp) / 3_600_000(to hours)Thresholds medianLeadTimeForChanges
Updated to use single unit (h):
dora.changeFailureRate
(failedDeployments / successfulDeployments) * 100Thresholds changeFailureRate
dora.meanTimeToRestore
MEAN for (incidentResolutionAtTimestamp - incidentCreatedAtTimestamp) / 3_600_000(to hours)Thresholds meanTimeToRestore
DORA collectors
DORA metrics are composite metrics that require data from different third parties for their computation. They use Scorecard Collectors, reusable components designed to gather data from various datasources, such as Jira or GitHub. Users can create their custom data collector to tailor data collection for DORA metrics calculation for their specific setup.
Demo
✔️ Checklist