fix(assessors): support project-named directories and test-only repos in standard_layout check

[kami619]

Summary

Enhances StandardLayoutAssessor to recognize multiple valid Python project structures:

• Project-named directories (flat layout like pandas/pandas/) now pass
• Test-only repositories now receive NOT_APPLICABLE instead of failing

Related Issues

• Fixes [DISCUSSION] Reconsider src/ directory requirement? #246 — [DISCUSSION] Reconsider src/ directory requirement?
• Fixes [BUG] standard_layout check does not consider test repos #305 — [BUG] standard_layout check does not consider test repos

Root Cause

The original implementation hardcoded src/ as a required directory, following the research report's guidance for "src layout." However, this didn't account for:

1. The equally valid "flat layout" pattern used by major Python packages (pandas, numpy, scikit-learn, vllm)
2. Test-only repositories that contain no source code to organize

Changes

src/agentready/assessors/structure.py

• Added _NON_SOURCE_DIRS blocklist to exclude non-source directories from detection
• Added _find_source_directory() with three-strategy detection:
  1. Check for src/ (existing behavior)
  2. Parse pyproject.toml for project name, look for matching directory
  3. Fall back to any root directory with __init__.py not in blocklist
• Added _get_package_name_from_pyproject() supporting PEP 621 and Poetry formats
• Added _is_test_only_repository() to detect test repos by config files and naming
• Updated _create_remediation() to provide context-aware guidance

tests/unit/test_assessors_structure.py

Added 11 new tests covering:

• Project-named directory with pyproject.toml
• Hyphen-to-underscore name normalization
• Fallback detection without pyproject.toml
• Blocklist exclusion
• src/ precedence over project-named
• Test-only repo detection (by config, by name, by pytest.ini)
• Edge case: malformed pyproject.toml
• Negative case: generic repo without indicators should still fail

Testing

Scenario	Before	After
opendatahub-tests (test-only repo)	FAIL (50)	NOT_APPLICABLE
Project with mypackage/ dir	FAIL (50)	PASS (100)
Project with src/ dir	PASS (100)	PASS (100)

# All 16 tests pass
uv run pytest tests/unit/test_assessors_structure.py -v

Breaking Changes

None. Backward compatible:

• Repos with src/ layout continue to pass unchanged
• Repos that previously failed now pass or get NOT_APPLICABLE

Checklist

• Code follows project style guidelines
• Unit tests added for new functionality
• All existing tests pass
• Real-world verification on reported repos
• Documentation updated (docstrings, comments)
• No breaking changes

---

Co-Authored-By: Claude Opus 4.5 noreply@anthropic.com

[github-actions]

📈 Test Coverage Report

Branch	Coverage
This PR	66.7%
Main	66.4%
Diff	✅ +0.3%

---

Coverage calculated from unit tests only

[github-actions]

AgentReady Code Review — PR #322

Scope: StandardLayoutAssessor — support project-named directories and test-only repos
Files: src/agentready/assessors/structure.py (+275/-32), tests/unit/test_assessors_structure.py (+346/0)
Attribute Impact: standard_layout — Tier 1 Essential, 10% weight

---

Summary

This PR correctly identifies two real shortcomings in the original StandardLayoutAssessor and the general approach is sound. The three-strategy source detection and context-aware remediation are meaningful improvements. However, there are several correctness and reliability issues that need addressing before merge.

---

Issues

Critical

1. setup.cfg is not a test-only indicator (structure.py:262)

setup.cfg is used across nearly all Python projects for mypy, flake8, coverage.py, setuptools, and many other tools. Including it in test_indicators will incorrectly classify regular Python projects that lack a src/ layout as test-only repos — returning not_applicable when they should fail and receive remediation guidance.

tox.ini is similarly risky — it is used by many non-test-only projects for linting, building, and docs. At minimum remove setup.cfg; consider also removing tox.ini.

---

2. Non-deterministic directory ordering in Strategy 3 (structure.py:214)

Path.iterdir() returns entries in filesystem-dependent order (hash-randomized on some filesystems, inode order on others). The first qualifying directory found becomes the reported source directory, producing inconsistent behaviour across runs and platforms.

Fix: for item in sorted(repository.path.iterdir()):

---

Significant

3. Strategy 3 fallback is too permissive (structure.py:204-220)

Any root-level directory with __init__.py not in _NON_SOURCE_DIRS qualifies as a source directory. This will match migrations/, alembic/, config/, middleware/, settings/, celery/ etc. — none of which are package source directories. A project with migrations/__init__.py and tests/ but no src/ and no pyproject.toml would pass when it should fail.

Consider whether Strategy 3 should exist at all without a pyproject.toml.

---

4. Name-based test-only detection is over-broad (structure.py:272)

Using pattern in repository.name.lower() with "test" matches "testimonial-service", "contest-platform", "latest-features", "manifest-testing-tool". These are normal Python projects that would silently get not_applicable instead of a failing score and remediation guidance.

Use word-boundary matching instead:

import re
name_suggests_tests = bool(
    re.search(r"(^\|[-_.])(test\|tests\|testing\|spec\|specs)($\|[-_.])", repository.name.lower())
)

---

5. _NON_SOURCE_DIRS is missing common entries (structure.py:14)

The blocklist omits migrations, alembic, config, settings, middleware, static, assets, data, ci, resources, locale, i18n. These are common root-level directories with __init__.py that Strategy 3 would pick up as false-positive source directories.

---

Minor

6. Redundant has_tests re-check in _is_test_only_repository (structure.py:249)

The caller only invokes this method when not has_source and has_tests is already True. The guard at the top of the method is dead code.

7. Empty strings in commands list create blank entries (structure.py:320)

Empty strings in the commands list render as blank command lines in the HTML report. Use a comment line or handle spacing in the template instead.

---

Positives

• frozenset for _NON_SOURCE_DIRS: correct choice, O(1) lookup, immutable.
• tomllib is stdlib in Python 3.11+ (project requires 3.11+), no new dependency.
• OSError | tomllib.TOMLDecodeError handling is correct graceful degradation per CLAUDE.md principles.
• Finding.not_applicable(reason=...) usage is correct — reason lands in evidence[0], test assertions check this correctly.
• src/ precedence over project-named (Strategy 1 before 2) is the right call.
• Context-aware _create_remediation(has_source, has_tests) is a meaningful improvement.
• Updated PyPA citation URL (src-layout-vs-flat-layout) is more specific and relevant.
• 11 new tests with good edge case coverage. test_malformed_pyproject_toml_handled_gracefully is exactly the kind of defensive test this codebase needs.
• Backward compatible: src/ repos continue to pass unchanged.

---

AgentReady Attribute Compliance

Check	Result
Inherits BaseAssessor	pass
Implements attribute_id	pass
Implements assess(repository)	pass
Returns Finding (not raw data)	pass
Graceful degradation on error	pass
Proportional scoring via calculate_proportional_score()	pass
Actionable remediation steps	pass (improved)
Unit tests added	pass (11 tests)
Follows CLAUDEmdAssessor patterns	pass

---

Score Impact Assessment

Test-only repo case (not_applicable): Repos penalised for a non-applicable attribute will have it excluded from their denominator. This is the correct behaviour and improves score accuracy.

Project-named directory case: Repos using flat layout (numpy/pandas style) that were scoring 50/100 on this attribute will now score 100/100. This directly fixes the reported false negatives.

standard_layout carries 10% weight (Tier 1 Essential). A 50 to 100 correction on this attribute translates to a +5 point boost to the overall score for affected repos.

---

Recommendation

Request changes. The core approach is correct and the bugs being fixed are real. The critical issues (setup.cfg false positive, non-deterministic iteration, over-broad name matching, Strategy 3 false positives) need to be addressed before this is safe to merge. The risk is that the fix introduces a new class of false positives that silently suppress remediation guidance for repos that need it.

[github-actions]

AgentReady Code Review — PR #322

Scope: StandardLayoutAssessor — support project-named directories and test-only repos
Files: src/agentready/assessors/structure.py (+275/-32), tests/unit/test_assessors_structure.py (+346/0)
Attribute Impact: standard_layout — Tier 1 Essential, 10% weight

---

Summary

This PR correctly identifies two real shortcomings in the original StandardLayoutAssessor and the general approach is sound. The three-strategy source detection and context-aware remediation are meaningful improvements. However, there are several correctness and reliability issues that need addressing before merge.

---

Issues

Critical

1. setup.cfg is not a test-only indicator (structure.py:262)

setup.cfg is used across nearly all Python projects for mypy, flake8, coverage.py, setuptools, and many other tools. Including it in test_indicators will incorrectly classify regular Python projects that lack a src/ layout as test-only repos — returning not_applicable when they should fail and receive remediation guidance.

tox.ini is similarly risky — it is used by many non-test-only projects for linting, building, and docs. At minimum remove setup.cfg; consider also removing tox.ini.

---

2. Non-deterministic directory ordering in Strategy 3 (structure.py:214)

Path.iterdir() returns entries in filesystem-dependent order. The first qualifying directory found becomes the reported source directory, producing inconsistent behaviour across runs and platforms.

Fix: for item in sorted(repository.path.iterdir()):

---

Significant

3. Strategy 3 fallback is too permissive (structure.py:204-220)

Any root-level directory with __init__.py not in _NON_SOURCE_DIRS qualifies as a source directory. This will match migrations/, alembic/, config/, middleware/, settings/, celery/ etc. A project with migrations/__init__.py and tests/ but no src/ and no pyproject.toml would pass when it should fail.

Consider whether Strategy 3 should exist at all without a pyproject.toml.

---

4. Name-based test-only detection is over-broad (structure.py:272)

Using pattern in repository.name.lower() with "test" matches "testimonial-service", "contest-platform", "latest-features". These are normal Python projects that would silently get not_applicable instead of a failing score and remediation guidance.

Use word-boundary matching:

name_suggests_tests = bool(
    re.search(r"(^|[-_.])(test|tests|testing|spec|specs)($|[-_.])", repository.name.lower())
)

---

5. _NON_SOURCE_DIRS is missing common entries (structure.py:14)

The blocklist omits migrations, alembic, config, settings, middleware, static, assets, data, ci, resources, locale, i18n. These are common root-level directories with __init__.py that Strategy 3 would pick up as false-positive source directories.

---

Minor

6. Redundant has_tests re-check in _is_test_only_repository (structure.py:249)

The caller only invokes this method when not has_source and has_tests is already True. The guard at the top of the method is dead code.

7. Empty strings in commands list (structure.py:320)

Empty strings in the commands list render as blank lines in the HTML report. Use a comment line instead.

---

Positives

• frozenset for _NON_SOURCE_DIRS: correct choice, O(1) lookup, immutable.
• tomllib is stdlib in Python 3.11+ (project requires 3.11+), no new dependency introduced.
• OSError | tomllib.TOMLDecodeError handling is correct graceful degradation per CLAUDE.md principles.
• Finding.not_applicable(reason=...) usage is correct — reason lands in evidence[0], test assertions check this correctly.
• src/ precedence over project-named (Strategy 1 before 2) is the right call.
• Context-aware _create_remediation(has_source, has_tests) is a meaningful improvement.
• Updated PyPA citation URL (src-layout-vs-flat-layout) is more specific and relevant.
• 11 new tests with good edge case coverage; test_malformed_pyproject_toml_handled_gracefully is exactly the kind of defensive test this codebase needs.
• Backward compatible: src/ repos continue to pass unchanged.

---

AgentReady Attribute Compliance

Check	Result
Inherits BaseAssessor	pass
Implements attribute_id	pass
Implements assess(repository)	pass
Returns Finding (not raw data)	pass
Graceful degradation on error	pass
Proportional scoring via calculate_proportional_score()	pass
Actionable remediation steps	pass (improved)
Unit tests added	pass (11 tests)
Follows CLAUDEmdAssessor patterns	pass

---

Score Impact Assessment

Test-only repo case (not_applicable): Excluded from denominator — correct behaviour, improves score accuracy for repos like opendatahub-tests.

Project-named directory case: Repos using flat layout (numpy/pandas style) that were scoring 50/100 will now score 100/100. Directly fixes reported false negatives.

standard_layout carries 10% weight (Tier 1 Essential). A 50→100 correction translates to a +5 point boost to the overall score for affected repos.

---

Recommendation

Request changes. The core approach is correct and the bugs being fixed are real. The critical issues (setup.cfg false positive, non-deterministic iteration, over-broad name matching, Strategy 3 false positives) need to be addressed before this is safe to merge. The risk is that the fix introduces a new class of false positives that silently suppress remediation guidance for repos that genuinely need it.

[github-actions]

AgentReady Code Review — PR #322

Attribute: standard_layout (Tier 1, 10% weight)
Files changed: src/agentready/assessors/structure.py, tests/unit/test_assessors_structure.py

---

Summary

This PR addresses two real issues (#246, #305) with solid overall design: the three-strategy source detection and the not_applicable path for test-only repos are both correct approaches. The test coverage for new functionality is thorough. There are a few issues that need attention before merging.

---

Issues

[Bug] pytest.ini is not a reliable test-only indicator

pytest.ini at the project root is standard configuration for virtually any Python project using pytest — not a marker of a test-only repo. A project with tests/ + pytest.ini but whose source directory wasn't detected by Strategy 1–3 will incorrectly receive not_applicable instead of fail.

# _is_test_only_repository: current
test_indicators = [
    repository.path / "conftest.py",
    repository.path / "pytest.ini",
]

Same concern applies to conftest.py: it's routinely placed at the project root for shared fixtures in mixed projects.

Recommendation: Rely primarily on the name regex (which is well-designed) and require both config files plus the absence of pyproject.toml as a combined signal — test-only repos rarely have pyproject.toml with a [project] section.

---

[Bug] celery in _NON_SOURCE_DIRS would fail the Celery project itself

The actual Celery package has celery/ as its source directory. Blocking it by name means any assessment of the Celery repo will fail standard_layout even though it follows the flat layout pattern this PR intends to support.

The blocklist rationale is sound for generic terms, but framework/package proper names (celery, middleware, alembic) are risky inclusions. These should be removed from the blocklist, or the strategy should be inverted to an allowlist of known non-source patterns.

---

[Design concern] Strategy 3 has broad false-positive exposure

When pyproject.toml exists but the project name doesn't match a root directory, Strategy 3 scans all root directories with __init__.py. In a Django project with api/, users/, payments/ packages, this returns whichever comes first alphabetically — which may not be the primary source package.

# Inside `if package_name:` block — runs when pyproject.toml exists but name doesn't match
for item in sorted(repository.path.iterdir(), key=lambda p: p.name):
    if (item / "__init__.py").exists():
        return {"found": True, "type": "project-named", "directory": ...}

Recommendation: Surface this as a heuristic in the evidence (e.g., "source directory: heuristic match — verify") rather than silently returning a pass.

---

[Style] Self-referential PR comments should be cleaned up

Comments like # PR #322 feedback: ... appear throughout the diff. These look like iterative review notes that weren't removed. They add noise and are confusing as permanent comments.

# PR #322 feedback: Sort for deterministic behavior across platforms

Replace with direct rationale (e.g., # Sort for deterministic behavior across platforms).

---

Positives

• Three-strategy detection is well-structured with clear precedence: src/ → name-matched → fallback.
• Hyphen-to-underscore normalization (my-package → my_package/) correctly handles PEP 508 naming.
• src/ precedence over project-named is the right call.
• TOML exception handling covers both OSError and TOMLDecodeError.
• Name regex (^|[-_.])(?:test|tests|testing|spec|specs)($|[-_.]) correctly avoids false matches on testimonial-service and contest-platform.
• Test coverage is thorough: 11 new tests covering happy path, edge cases, and negative cases.
• Remediation guidance is now context-aware and the updated citation URL is more specific.
• Backward compatibility: repos that previously passed continue to pass.

---

Score Impact

Scenario	Before	After
Flat-layout project (pandas-style)	FAIL (50) — false negative	PASS (100) ✓
Test-only repo (opendatahub-tests)	FAIL (50) — false positive	NOT_APPLICABLE ✓
Project with pytest.ini, no detectable src	FAIL (50)	NOT_APPLICABLE — possible false skip ⚠️
Celery project assessment	PASS (100)	FAIL (50) — regression ⚠️

---

Verdict

Request changes on the pytest.ini/conftest.py test-only indicators and the celery blocklist entry before merging. The core design is sound and most changes are clear improvements.

---

Review generated by AgentReady code review skill

[github-actions]

AgentReady Code Review - PR #322

Summary

This PR addresses two real-world correctness bugs (#246, #305) in StandardLayoutAssessor by extending source directory detection beyond the hard-coded src/ assumption and adding graceful handling for test-only repositories. The implementation is well-structured and includes 11 new unit tests. There is one documentation inaccuracy in the Strategy 3 comment and a handful of minor items worth addressing before merge.

---

AgentReady Attribute Analysis

Attribute	Impact	Details
claude_md	Improves	Issue references (#246, #305) and fix markers in docstrings make the change auditable
readme	Neutral	No README changes needed for an internal assessor fix
type_annotations	Concerns	_find_source_directory returns dict without a typed return signature; a TypedDict or dataclass would align with the codebase's typing standards
test_coverage	Improves	11 new focused tests covering happy paths, edge cases, Poetry support, malformed TOML, and heuristic detection
error_handling	Improves	tomllib.TOMLDecodeError and OSError are caught, malformed TOML falls through safely
code_organization	Improves	Three private helper methods cleanly separate the three detection strategies
llm_friendly_naming	Improves	_find_source_directory, _get_package_name_from_pyproject, _is_test_only_repository are all self-documenting
input_validation	Concerns	Strategy 3 (heuristic) can produce a false positive when pyproject.toml declares name A but only package B (unrelated to A) exists at root; the assessor passes with a "heuristic" tag but the tag is easy to overlook in reports
security_scanning	Neutral	No new untrusted input surfaces introduced

---

Security Review

No security issues found. The PR reads only local filesystem paths derived from the already-trusted repository.path. TOML parsing uses the stdlib tomllib, which is memory-safe and does not evaluate code.

---

Code Quality

Strategy 3 gating comment is inaccurate.

Strategy 3 executes inside the if package_name: branch. This means it only runs when pyproject.toml exists and _get_package_name_from_pyproject returned a non-empty name. The inline comment says "Only do this when pyproject.toml exists to avoid false positives," but the actual guard is stricter: it also requires a parseable project name. A pyproject.toml containing only [build-system] with no [project].name or [tool.poetry].name would silently skip Strategy 3 and return not-found. The test test_heuristic_match_shows_verify_in_evidence passes precisely because its fixture pyproject.toml does contain [project].name = "myproject". This behavior may be intentional (requiring a name further reduces false positives), but the comment should document the actual condition accurately.

Untyped return value from _find_source_directory.

The method returns a plain dict. Callers access source_info["found"], source_info["type"], and source_info["directory"] with no IDE-checkable types. A TypedDict would align with the codebase's typing standards:

from typing import TypedDict

class SourceDirectoryResult(TypedDict):
    found: bool
    type: str
    directory: str

Latent fragility in _create_remediation.

Remediation.__post_init__ raises ValueError if steps is empty. If _create_remediation were ever called in a state where both has_source=True and has_tests=True, the steps list would be empty and cause a runtime error. The status == "fail" call-site guard prevents this today, but a defensive assertion inside _create_remediation would make the invariant explicit and safe under future refactoring.

Test assertion is missing an explanatory comment.

In test_test_only_repo_returns_not_applicable, the assertion checks finding.evidence for the reason string. Finding.not_applicable stores the reason string inside evidence (see finding.py line 107), not in error_message. A brief comment explaining this would help future contributors understand why the assertion targets evidence rather than error_message.

_NON_SOURCE_DIRS blocklist is reasonable but incomplete.

Common directories like lib, app, core, and pkg are used as source roots in some Python projects and are absent from the blocklist. This is an acceptable scope limitation for this PR -- worth a follow-up issue.

Python version compatibility -- confirmed safe.

tomllib is stdlib since Python 3.11. The project requires Python 3.11+, so the import is correct with no compatibility shim needed.

---

Recommendations

P0 -- Address before merge:

1. Clarify the Strategy 3 gating comment -- update the inline comment to accurately state that Strategy 3 requires both pyproject.toml to exist and a parseable project name, not just file existence. The current comment says "Only do this when pyproject.toml exists" but the actual condition is stricter.

2. Add a test for pyproject.toml with no parseable project name -- a fixture pyproject.toml containing only [build-system] should be tested to confirm whether Strategy 3 is reached and the resulting behavior is intentional and documented.

P1 -- Strong recommendations:

3. Return a TypedDict from _find_source_directory instead of a plain dict to improve type safety and IDE/linter support.

4. Add a defensive guard in _create_remediation to assert at least one of has_source or has_tests is False before building the steps list, preventing a future ValueError from Remediation.__post_init__.

P2 -- Nice to have:

5. Add a comment in test_test_only_repo_returns_not_applicable explaining why the reason appears in finding.evidence rather than finding.error_message.

6. Open a follow-up issue to track gaps in _NON_SOURCE_DIRS for lib/, app/, core/, pkg/ patterns.

---

Score Impact Estimate

The standard_layout attribute is Tier 1 Essential with default_weight = 0.10 (10% of total score). Repositories with flat layouts (pandas-style) that previously incorrectly scored 50/100 will now score 100/100 when a matching project-named directory is detected. Test-only repositories that previously failed (50/100) will be marked not_applicable and excluded from scoring, eliminating unfair penalty.

Estimated impact: affected repositories could see a +5 to +10 point improvement in overall AgentReady score, depending on which other Tier 1 attributes were previously passing. This directly improves assessor correctness and reduces false negatives for real-world Python project layouts.

---

Review generated by the AgentReady Development Agent.