Skip to content

Add objective integration tests for all example_ws configs#534

Draft
rlpratt12 wants to merge 1 commit intomainfrom
claude/add-objective-integration-tests
Draft

Add objective integration tests for all example_ws configs#534
rlpratt12 wants to merge 1 commit intomainfrom
claude/add-objective-integration-tests

Conversation

@rlpratt12
Copy link

Summary

  • Add objectives_integration_test.py to 11 config packages (kitchen_sim, april_tag_sim, grinding_sim, dual_arm_sim, factory_sim, hangar_sim, kinova_sim, space_satellite_sim, space_satellite_sim_camera_cal, mock_sim, multi_arm_sim)
  • Each test auto-discovers objectives via SystemConfigParser and runs them with per-config skip lists for user-input/ML/GPU objectives
  • New objectives are automatically tested — if one requires skipping, CI fails and the PR author adds it to the skip list (fail-safe design)
  • Add CI matrix job to run each config's tests in parallel via the existing workspace_integration_test reusable workflow
  • Add missing ament_cmake_pytest + moveit_pro_test_utils test deps across all configs (including existing lab_sim)

Known TODOs

  • Skip lists are best-guess based on objective names/XML analysis — need to validate with actual CI runs
  • The existing integration-test-in-studio-container job still runs all tests including these, causing double-runs. Consider adding --packages-skip for config packages to that job
  • 600s timeout may need tuning for configs with many objectives (e.g. lab_sim has 80)
  • Cancel objectives sets are empty for new configs — looping objectives may need to be identified and added

Test plan

  • All 12 test files parse correctly with valid Python syntax
  • Skip list objective IDs verified against actual main_tree_to_execute XML attributes — all match
  • CMakeLists.txt and package.xml changes follow existing lab_sim pattern
  • Run integration tests for mock_sim (simplest, no skips) to validate end-to-end
  • Run integration tests for a config with skips (e.g. kitchen_sim) to validate skip behavior

🤖 Generated with Claude Code

@rlpratt12 rlpratt12 requested review from JWhitleyWork and shaur-k and removed request for JWhitleyWork and shaur-k March 19, 2026 01:51
Copy link
Member

@JWhitleyWork JWhitleyWork left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is a good start but needs some tweaking.

moveit_license_key: ${{ secrets.STUDIO_CI_LICENSE_KEY }}

# Per-config objective integration tests (run in parallel via matrix)
objective-integration-test:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The downside to doing this with a matrix is that it launches a separate Github Actions runner for each invocation of the matrix. If we do these all as a single colcon test --packages-select <list_of_packages>, many of them can still be run in parallel but we only spin up an instance, install dependencies, clone stuff, etc. once. Also, we should be doing this test for at least Humble and Jazzy (in a matrix because they are completely different environments).

ament_lint_auto_find_test_dependencies()
ament_add_pytest_test(
objectives_integration_test test/objectives_integration_test.py
TIMEOUT 600
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

10 minutes? We usually give 10 minutes for all of colcon test. Does it actually take anywhere near this long?

Replicate the lab_sim objectives_integration_test.py pattern across all 11
remaining sim config packages: kitchen_sim, april_tag_sim, grinding_sim,
dual_arm_sim, factory_sim, hangar_sim, kinova_sim, space_satellite_sim,
space_satellite_sim_camera_cal, mock_sim, and multi_arm_sim.

Each test auto-discovers objectives via SystemConfigParser and runs them,
with per-config skip lists for objectives requiring user input or GPU/ML
inference. New objectives are automatically tested — if one requires
skipping, CI fails and the PR author adds it to the skip list.

Also adds a CI matrix job to run each config's tests in parallel, and
adds missing ament_cmake_pytest + moveit_pro_test_utils test deps
(including for the existing lab_sim test).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@JWhitleyWork JWhitleyWork force-pushed the claude/add-objective-integration-tests branch from 863b4ca to a853397 Compare March 23, 2026 17:06
@JWhitleyWork JWhitleyWork added this to the 9.2.0 milestone Mar 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants