Add Markdown Report Generation to Civic Intelligence Engine#533
Add Markdown Report Generation to Civic Intelligence Engine#533RohanExploit wants to merge 1 commit intomainfrom
Conversation
…nement engine - Updated IntelligenceIndex to generate a human-readable markdown report (`REPORT_YYYY-MM-DD.md`) alongside the JSON snapshot. - Updated related unit tests in `dailyRefinement.test.ts` to assert markdown report generation and content structure.
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
✅ Deploy Preview for fixmybharat canceled.
|
🙏 Thank you for your contribution, @RohanExploit!PR Details:
Quality Checklist:
Review Process:
Note: The maintainers will monitor code quality and ensure the overall project flow isn't broken. |
📝 WalkthroughWalkthroughThe pull request adds markdown report generation to the intelligence index service. A new private method generates human-readable markdown reports for daily snapshots, which are written alongside existing JSON snapshots. Tests are updated to validate both file formats are created with appropriate content. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds a human-readable Markdown report output to the Daily Civic Intelligence Refinement Engine’s snapshot persistence, so each daily run produces both a JSON snapshot (machine-friendly) and a Markdown report (human-friendly).
Changes:
- Extend
IntelligenceIndex.saveSnapshot()to also generate aREPORT_<date>.mdMarkdown report alongside<date>.json. - Add Markdown report formatting for index score/delta, keywords, spikes, and severity region.
- Update the IntelligenceIndex integration test to validate both files are produced and the Markdown contains expected content.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| tests/dailyRefinement.test.ts | Updates the integration test to expect both JSON + Markdown outputs and validates basic report content. |
| services/intelligenceIndex.ts | Implements Markdown report generation and writes it during snapshot saving. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
| // Also save a human-readable markdown report | ||
| this.saveMarkdownReport(snapshot); |
There was a problem hiding this comment.
saveSnapshot now always writes a second file synchronously, so any filesystem error while creating the Markdown report will throw and make the daily refinement job look like it failed even though the JSON snapshot has already been persisted. Consider isolating Markdown generation failures (e.g., try/catch around saveMarkdownReport and log/continue) or writing both outputs in a way that keeps failure behavior consistent/intentional.
| // Also save a human-readable markdown report | |
| this.saveMarkdownReport(snapshot); | |
| // Also save a human-readable markdown report, but don't fail the snapshot | |
| // operation if markdown generation/writing encounters an error. | |
| try { | |
| this.saveMarkdownReport(snapshot); | |
| } catch (err) { | |
| // Log and continue; the primary JSON snapshot has already been persisted. | |
| console.error( | |
| `Failed to save markdown report for snapshot ${snapshot.date}:`, | |
| err, | |
| ); | |
| } |
| const jsonFile = files.find(f => f.endsWith('.json')); | ||
| const mdFile = files.find(f => f.endsWith('.md')); | ||
|
|
||
| expect(jsonFile).toBeDefined(); | ||
| expect(jsonFile).toContain(snapshot.date); | ||
|
|
||
| expect(mdFile).toBeDefined(); | ||
| expect(mdFile).toContain(`REPORT_${snapshot.date}`); | ||
|
|
||
| const mdContent = fs.readFileSync(path.join(testSnapshotsDir, mdFile!), 'utf-8'); |
There was a problem hiding this comment.
This test asserts mdFile/jsonFile are defined, but Jest assertions don’t short-circuit execution; if either is undefined, the following toContain/readFileSync will throw a secondary error and obscure the real failure. Prefer making the values non-optional before use (e.g., explicit if (!mdFile) throw … / expect(mdFile).toBeTruthy() plus narrowing) so failures stay clear and the test doesn’t crash mid-way.
| const jsonFile = files.find(f => f.endsWith('.json')); | |
| const mdFile = files.find(f => f.endsWith('.md')); | |
| expect(jsonFile).toBeDefined(); | |
| expect(jsonFile).toContain(snapshot.date); | |
| expect(mdFile).toBeDefined(); | |
| expect(mdFile).toContain(`REPORT_${snapshot.date}`); | |
| const mdContent = fs.readFileSync(path.join(testSnapshotsDir, mdFile!), 'utf-8'); | |
| const jsonFile = files.find(f => f.endsWith(".json")); | |
| if (!jsonFile) { | |
| throw new Error("Expected JSON snapshot file to be created, but none was found."); | |
| } | |
| const mdFile = files.find(f => f.endsWith(".md")); | |
| if (!mdFile) { | |
| throw new Error("Expected markdown snapshot file to be created, but none was found."); | |
| } | |
| expect(jsonFile).toContain(snapshot.date); | |
| expect(mdFile).toContain(`REPORT_${snapshot.date}`); | |
| const mdContent = fs.readFileSync(path.join(testSnapshotsDir, mdFile), "utf-8"); |
| expect(files.length).toBe(2); // Should have the JSON and the MD file | ||
|
|
||
| const jsonFile = files.find(f => f.endsWith('.json')); | ||
| const mdFile = files.find(f => f.endsWith('.md')); |
There was a problem hiding this comment.
expect(files.length).toBe(2) makes this test brittle if snapshot saving later adds any additional artifacts (e.g., future metadata files). Since you already locate files by extension below, consider asserting the presence of one .json and one .md instead of an exact directory size.
| expect(files.length).toBe(2); // Should have the JSON and the MD file | |
| const jsonFile = files.find(f => f.endsWith('.json')); | |
| const mdFile = files.find(f => f.endsWith('.md')); | |
| const jsonFile = files.find(f => f.endsWith('.json')); | |
| const mdFile = files.find(f => f.endswith('.md')); |
|
|
||
| markdown += `## Top 5 Emerging Keywords\n`; | ||
| if (snapshot.topKeywords.length > 0) { | ||
| snapshot.topKeywords.forEach((kw, i) => { |
There was a problem hiding this comment.
The report section header says "Top 5 Emerging Keywords", but the code iterates over all snapshot.topKeywords without limiting to 5. Either cap the list (e.g., slice to 5) or change the header so the generated report doesn’t claim a top-5 when more items may be printed (especially if saveSnapshot is called with a snapshot built outside TrendAnalyzer.getTopKeywords(…, 5)).
| snapshot.topKeywords.forEach((kw, i) => { | |
| snapshot.topKeywords.slice(0, 5).forEach((kw, i) => { |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@services/intelligenceIndex.ts`:
- Around line 77-81: The "Top 5 Emerging Keywords" heading is out of sync with
the loop that writes all entries from snapshot.topKeywords; update the logic in
services/intelligenceIndex.ts so the output matches the heading by iterating
only the first five items (e.g., use snapshot.topKeywords.slice(0, 5) or loop up
to Math.min(5, snapshot.topKeywords.length) when building markdown), or
alternatively change the heading text to reflect the actual count (e.g., "Top
Emerging Keywords" or include the count). Adjust the loop that references
snapshot.topKeywords to use the chosen approach so the section title and listed
items remain consistent.
- Around line 68-73: The code builds file paths using DailySnapshot.date without
validation (used in saveSnapshot and saveMarkdownReport), allowing path
traversal; validate snapshot.date against a strict YYYY-MM-DD regex before
constructing any path and reject (throw or return an error) on mismatch, and
then use path.join(this.snapshotsDir, `REPORT_${snapshot.date}.md`) only after
validation; ensure the same validation is applied where saveSnapshot is public
so no untrusted snapshot.date can escape snapshotsDir.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: b9144a7c-74e4-4689-8428-b9e2895f4ba3
📒 Files selected for processing (2)
services/intelligenceIndex.tstests/dailyRefinement.test.ts
| // Also save a human-readable markdown report | ||
| this.saveMarkdownReport(snapshot); | ||
| } | ||
|
|
||
| private saveMarkdownReport(snapshot: DailySnapshot): void { | ||
| const filePath = path.join(this.snapshotsDir, `REPORT_${snapshot.date}.md`); |
There was a problem hiding this comment.
Validate snapshot.date before building filenames.
saveSnapshot is public, and DailySnapshot.date is just a string. A value like ../../outside will make these writes escape snapshotsDir. Reject anything outside the expected YYYY-MM-DD format before constructing either path.
🔐 Suggested fix
public saveSnapshot(snapshot: DailySnapshot): void {
- const filePath = path.join(this.snapshotsDir, `${snapshot.date}.json`);
+ const safeDate = this.validateSnapshotDate(snapshot.date);
+ const filePath = path.join(this.snapshotsDir, `${safeDate}.json`);
fs.writeFileSync(filePath, JSON.stringify(snapshot, null, 2), "utf-8");
// Also save a human-readable markdown report
- this.saveMarkdownReport(snapshot);
+ this.saveMarkdownReport({ ...snapshot, date: safeDate });
}
+ private validateSnapshotDate(date: string): string {
+ if (!/^\d{4}-\d{2}-\d{2}$/.test(date)) {
+ throw new Error("snapshot.date must use YYYY-MM-DD format");
+ }
+ return date;
+ }
+
private saveMarkdownReport(snapshot: DailySnapshot): void {
const filePath = path.join(this.snapshotsDir, `REPORT_${snapshot.date}.md`);🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@services/intelligenceIndex.ts` around lines 68 - 73, The code builds file
paths using DailySnapshot.date without validation (used in saveSnapshot and
saveMarkdownReport), allowing path traversal; validate snapshot.date against a
strict YYYY-MM-DD regex before constructing any path and reject (throw or return
an error) on mismatch, and then use path.join(this.snapshotsDir,
`REPORT_${snapshot.date}.md`) only after validation; ensure the same validation
is applied where saveSnapshot is public so no untrusted snapshot.date can escape
snapshotsDir.
| markdown += `## Top 5 Emerging Keywords\n`; | ||
| if (snapshot.topKeywords.length > 0) { | ||
| snapshot.topKeywords.forEach((kw, i) => { | ||
| markdown += `${i + 1}. ${kw}\n`; | ||
| }); |
There was a problem hiding this comment.
Keep the section title and output in sync.
The heading says Top 5, but this loop writes every keyword in snapshot.topKeywords. Either cap the list here or rename the section so the report stays accurate when callers pass more than five keywords.
📝 Suggested fix
markdown += `## Top 5 Emerging Keywords\n`;
if (snapshot.topKeywords.length > 0) {
- snapshot.topKeywords.forEach((kw, i) => {
+ snapshot.topKeywords.slice(0, 5).forEach((kw, i) => {
markdown += `${i + 1}. ${kw}\n`;
});
} else {📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| markdown += `## Top 5 Emerging Keywords\n`; | |
| if (snapshot.topKeywords.length > 0) { | |
| snapshot.topKeywords.forEach((kw, i) => { | |
| markdown += `${i + 1}. ${kw}\n`; | |
| }); | |
| markdown += `## Top 5 Emerging Keywords\n`; | |
| if (snapshot.topKeywords.length > 0) { | |
| snapshot.topKeywords.slice(0, 5).forEach((kw, i) => { | |
| markdown += `${i + 1}. ${kw}\n`; | |
| }); |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@services/intelligenceIndex.ts` around lines 77 - 81, The "Top 5 Emerging
Keywords" heading is out of sync with the loop that writes all entries from
snapshot.topKeywords; update the logic in services/intelligenceIndex.ts so the
output matches the heading by iterating only the first five items (e.g., use
snapshot.topKeywords.slice(0, 5) or loop up to Math.min(5,
snapshot.topKeywords.length) when building markdown), or alternatively change
the heading text to reflect the actual count (e.g., "Top Emerging Keywords" or
include the count). Adjust the loop that references snapshot.topKeywords to use
the chosen approach so the section title and listed items remain consistent.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found across 2 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="services/intelligenceIndex.ts">
<violation number="1" location="services/intelligenceIndex.ts:79">
P3: Limit the rendered keywords to 5 to match the report section title.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
|
|
||
| markdown += `## Top 5 Emerging Keywords\n`; | ||
| if (snapshot.topKeywords.length > 0) { | ||
| snapshot.topKeywords.forEach((kw, i) => { |
There was a problem hiding this comment.
P3: Limit the rendered keywords to 5 to match the report section title.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At services/intelligenceIndex.ts, line 79:
<comment>Limit the rendered keywords to 5 to match the report section title.</comment>
<file context>
@@ -64,6 +64,43 @@ export class IntelligenceIndex {
+
+ markdown += `## Top 5 Emerging Keywords\n`;
+ if (snapshot.topKeywords.length > 0) {
+ snapshot.topKeywords.forEach((kw, i) => {
+ markdown += `${i + 1}. ${kw}\n`;
+ });
</file context>
| snapshot.topKeywords.forEach((kw, i) => { | |
| snapshot.topKeywords.slice(0, 5).forEach((kw, i) => { |
The Daily Civic Intelligence Refinement Engine now auto-generates a human-readable Markdown report alongside its standard JSON snapshot, providing easily accessible insights into the generated civic intelligence indices and trending keyword metrics. Included corresponding unit test updates to ensure code coverage and functional correctness.
PR created automatically by Jules for task 4688680061861365670 started by @RohanExploit
Summary by cubic
Adds Markdown report generation to the Daily Civic Intelligence Refinement Engine, producing
REPORT_YYYY-MM-DD.mdalongside the JSON snapshot for clearer, human-readable insights. Aligns with task 4688680061861365670.REPORT_<date>.mdwith index score and delta, top 5 keywords, emerging concern spikes, and highest-severity region.Written for commit 0a2d2e3. Summary will update on new commits.
Summary by CodeRabbit