Skip to content

feat: add assignment submission scoring - #235

Merged
Siddhant-K-code merged 1 commit into
mainfrom
feat/assignment-mode
Aug 16, 2026
Merged

feat: add assignment submission scoring#235
Siddhant-K-code merged 1 commit into
mainfrom
feat/assignment-mode

Conversation

@Siddhant-K-code

Copy link
Copy Markdown
Owner

Description

Closes #213.

Adds privacy-minimized assignment submissions and deterministic local scoring for agent-based technical hiring workflows:

  • agent-strace share --assignment [SESSION_ID] --output submission.zip
  • agent-strace score SUBMISSION.zip --rubric FILE
  • bounded directory comparison with --compare
  • versioned JSON output with stable ties and opaque submission labels
  • strict six-member assignment-v1 ZIP schema with relative timestamps, anonymous references, frozen pricing/lint/viewer semantics, and internal SHA-256 consistency checks
  • hostile-input validation without archive extraction
  • documented human-review and privacy limitations
  • checked-in rubric, synthetic generator, and reviewable golden submission members

Assignment bundles intentionally omit raw identity, prompts, results, commands, paths, file contents, environment values, URLs, and custom model identifiers. Scores are deterministic process-telemetry arithmetic for human review, not hiring decisions or task/candidate-quality judgments. Bundle hashes provide consistency, not signatures or provenance.

Validation

  • Ona Test task: 1,895 Python tests passed
  • VS Code extension: 3 tests passed
  • Ona Build task passed; wheel built as agent_strace-0.93.0-py3-none-any.whl
  • Focused assignment suite: 26 tests passed
  • Two independent reviews approved with no blockers
  • Compile, CLI help, example generation/scoring, golden compatibility, and git diff --check passed

Release

This feature bumps the package from 0.92.0 to 0.93.0 following the repository's minor-version policy.

Co-authored-by: Codex <noreply@openai.com>
@Siddhant-K-code Siddhant-K-code added the enhancement New feature or request label Aug 16, 2026
@Siddhant-K-code

Copy link
Copy Markdown
Owner Author

Independent review: approved with no blockers.

Reviewed the complete PR against #213, including CLI behavior, the assignment-v1 data contract, privacy minimization, strict archive/rubric validation, deterministic weighted scoring and ranking, compatibility-frozen assets/pricing/lint semantics, docs, examples, and the v0.93.0 bump.

Validation evidence:

  • Hosted CI is green on Python 3.10, 3.11, 3.12, and 3.13
  • Hosted VS Code extension check is green
  • Settled-tree Ona test task: 1,895 Python tests and 3 extension tests passed
  • Focused assignment suite: 26/26 passed
  • Ona build task passed and built the 0.93.0 wheel
  • Two independent reviewers found no remaining blockers

Security/privacy review confirmed that decompressed bundle members omit raw identity, prompts, results, commands, paths, file contents, environment values, URLs, credentials, and custom model identifiers. Reports clearly state that scoring is a human review aid, not an automated hiring decision or a task/candidate-quality measure.

Approved for squash merge.

@Siddhant-K-code
Siddhant-K-code merged commit c1d5083 into main Aug 16, 2026
5 checks passed
@Siddhant-K-code
Siddhant-K-code deleted the feat/assignment-mode branch August 16, 2026 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: assignment mode — trace submission and scoring for agent-based technical hiring

1 participant