Skip to content

fix: improve Chinese NLP filetype detection accuracy#362

Merged
deepin-bot[bot] merged 1 commit into
linuxdeepin:masterfrom
Johnson-zs:master
Jul 20, 2026
Merged

fix: improve Chinese NLP filetype detection accuracy#362
deepin-bot[bot] merged 1 commit into
linuxdeepin:masterfrom
Johnson-zs:master

Conversation

@Johnson-zs

@Johnson-zs Johnson-zs commented Jul 20, 2026

Copy link
Copy Markdown
Contributor
  1. Added new test case for design source files shadowing image
    substrings
  2. Included "包" back to application filetype rules with negative
    lookahead to prevent false matches
  3. Increased priority of design source files to shadow image matches
  4. Removed outdated comment about span consumption since it's now
    standard behavior

Influence:

  1. Test Chinese file search with "矢量图" to ensure design source files
    are prioritized over general image files
  2. Verify "包" matches application filetype while avoiding false
    positives like "表情包"
  3. Check that design source files (PSD, AI etc.) are correctly
    identified in complex queries

fix: 提升中文NLP文件类型检测准确性

  1. 新增设计源文件遮蔽图片子字符串的测试用例
  2. 重新包含"包"到应用文件类型规则,添加否定前瞻避免误判
  3. 提高设计源文件的优先级使其遮蔽图片匹配
  4. 移除关于span消费的过时注释,因这已是标准行为

Influence:

  1. 测试中文文件搜索"矢量图"确保设计源文件优先于普通图片文件
  2. 验证"包"能匹配应用文件类型同时避免"表情包"等误判
  3. 检查在复杂查询中设计源文件(PSD, AI等)能被正确识别

Fixes: #370659

Summary by Sourcery

Improve Chinese NLP filetype detection accuracy for design source and application files.

New Features:

  • Add a Chinese NLP test ensuring design source filetype matches shadow general image substring matches.

Bug Fixes:

  • Restore the "包" synonym to Chinese application filetype detection while avoiding false positives such as "表情包".
  • Ensure design source filetype rules take precedence over image rules in Chinese queries to prevent misclassification.

Enhancements:

  • Simplify span handling in filetype extraction by removing an outdated comment about span consumption behavior.

Tests:

  • Extend Chinese NLP filetype tests to cover design source precedence over image substrings and updated application synonyms.

1. Added new test case for design source files shadowing image
substrings
2. Included "包" back to application filetype rules with negative
lookahead to prevent false matches
3. Increased priority of design source files to shadow image matches
4. Removed outdated comment about span consumption since it's now
standard behavior

Influence:
1. Test Chinese file search with "矢量图" to ensure design source files
are prioritized over general image files
2. Verify "包" matches application filetype while avoiding false
positives like "表情包"
3. Check that design source files (PSD, AI etc.) are correctly
identified in complex queries

fix: 提升中文NLP文件类型检测准确性

1. 新增设计源文件遮蔽图片子字符串的测试用例
2. 重新包含"包"到应用文件类型规则,添加否定前瞻避免误判
3. 提高设计源文件的优先级使其遮蔽图片匹配
4. 移除关于span消费的过时注释,因这已是标准行为

Influence:
1. 测试中文文件搜索"矢量图"确保设计源文件优先于普通图片文件
2. 验证"包"能匹配应用文件类型同时避免"表情包"等误判
3. 检查在复杂查询中设计源文件(PSD, AI等)能被正确识别

Fixes: #370659
@deepin-ci-robot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Johnson-zs

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sourcery-ai

sourcery-ai Bot commented Jul 20, 2026

Copy link
Copy Markdown

Reviewer's Guide

This PR refines Chinese NLP filetype detection by reintroducing the “包” synonym for applications with stricter matching, adding a regression test to ensure design-source file rules override generic image matches, and cleaning up an outdated span-consumption comment in the extractor implementation, along with corresponding rule updates in the Chinese filetype rules JSON.

File-Level Changes

Change Details Files
Improve application filetype synonym handling to safely re-include “包”.
  • Extend the application filetype synonym test inputs to include the standalone term “包”.
  • Align tests with updated application filetype rules that now match “包” while avoiding false positives like “表情包” via regex rule changes in the zh_CN filetype JSON.
autotests/dfm-search-tests/tst_chinese_nlp.cpp
src/dfm-search/dfm-search-lib/semantic/rules/zh_CN/filetype_rules.json
Ensure design-source filetype rules shadow generic image matches for Chinese queries like “矢量图”.
  • Add a dedicated test that parses “昨天创建的矢量图” and asserts that only the design-source filetype rule is triggered and that the image filetype rule is not.
  • Verify in the new test that the resulting file extensions match the expected design-source rule set via expectedExtsForRule.
  • Raise the effective priority of design-source rules over image rules in the zh_CN filetype rules JSON so overlapping substrings prefer design-source matches.
autotests/dfm-search-tests/tst_chinese_nlp.cpp
src/dfm-search/dfm-search-lib/semantic/rules/zh_CN/filetype_rules.json
Clean up extractor implementation comments to reflect current span consumption behavior.
  • Remove an outdated inline comment about always consuming matched spans, since this behavior is now standard and enforced elsewhere.
  • Keep the MatchSpan construction and consumption logic unchanged to maintain current behavior while simplifying the code’s documentation.
src/dfm-search/dfm-search-lib/semantic/extractors/filetypeextractor.cpp

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • In fileType_designSource_shadowsImageSubstring, consider also asserting that the resulting fileExtensions() does not include the image rule’s extensions so the shadowing behavior is verified at both span and extension levels.
  • The test currently relies on hard-coded ruleId string literals like "filetype_design_source" and "filetype_image"; using shared constants or an enum for rule IDs would make the tests less brittle to future rule renames.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In `fileType_designSource_shadowsImageSubstring`, consider also asserting that the resulting `fileExtensions()` does not include the image rule’s extensions so the shadowing behavior is verified at both span and extension levels.
- The test currently relies on hard-coded `ruleId` string literals like `"filetype_design_source"` and `"filetype_image"`; using shared constants or an enum for rule IDs would make the tests less brittle to future rule renames.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@deepin-ci-robot

Copy link
Copy Markdown

deepin pr auto review

★ 总体评分:100分

■ 【总体评价】

代码通过正则优化和优先级调整有效解决了NLP语义解析的误匹配和类型冲突问题,并补充了完善的测试用例
逻辑正确且无安全漏洞,代码质量良好,修复方案精准有效

■ 【详细分析】

  • 1.语法逻辑(完全正确)✓

filetype_rules.json 中,filetype_application 的正则表达式修改为 包(?!含),正确使用了负向先行断言避免匹配"包含";filetype_design_source 的优先级提升至 160,确保高于图片规则,解决了"矢量图"的冲突。测试代码 fileType_designSource_shadowsImageSubstring 逻辑严密,正确验证了修复效果。
潜在问题:无
建议:无需修改

  • 2.代码质量(良好)✓

新增的测试用例覆盖了之前存在的边界条件,移除了不再适用的注释,代码可读性高。正则表达式的修改直接针对问题,且易于理解。
潜在问题:无
建议:可以考虑在 filetype_rules.json 中为 包(?!含) 添加简短注释说明为何排除"包含",以便后续维护

  • 3.代码性能(无性能问题)✓

正则表达式的负向先行断言开销极小,优先级数值的调整对匹配算法的复杂度没有负面影响。
潜在问题:无
建议:无需修改

  • 4.代码安全(存在0个安全漏洞)✓

漏洞对比统计:新增漏洞 0 个,减少漏洞 0 个,持平 0 个
本次修改仅涉及正则表达式规则调整和测试代码增加,不存在外部输入注入或内存操作风险,攻击面无变化。

  • 建议:无需安全修复

■ 【改进建议代码示例】

{
    "id": "filetype_application",
    "pattern": "安装包|软件|应用|脚本|程序|包(?!含)",
    "description": "Application packages",
    "enabled": true,
    "priority": 150,
    "metadata": {
        "extensions": ["deb", "rpm", "sh", "appimage", "uap"]
    }
}

@Johnson-zs

Copy link
Copy Markdown
Contributor Author

/forcemerge

@deepin-bot

deepin-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown

This pr force merged! (status: blocked)

@deepin-bot
deepin-bot Bot merged commit 533f5c0 into linuxdeepin:master Jul 20, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants