test_forwarddiff.jl's POLAR_MATRICES check flakes at ~4%, and the oper

test_forwarddiff.jl's POLAR_MATRICES check flakes at ~4%, and the operating point 320e756 chose to prevent it does not move

test/solver/test_forwarddiff.jl:88 asserts rel_err < 1e-3 on max|jac_fwd - jac_fd| / max|jac_fwd|. It failed on #285's CI in Julia 1.12 - ubuntu - x64 - with code coverage at 0.0400, with the other five matrix cells green on the same commit. #285's diff is entirely inside the NONLIN branch of gamma_loop! (src/solver.jl:913-1010, which returns before the LOOP branch at :1012) and this test builds a solver_type=LOOP solver, so no changed line is reachable from it.

This is the third time the check has been in the frame. 320e756 recorded "one Windows run crossed 1e-3 at 0.044 while a re-run passed", moved y_op[1:4] off zero (defl_rad = deg2rad(1.0)) and loosened the bound to 0.1 in the same commit. ae11d48 then pinned NeuralFoil table generation to one BLAS thread, tightened the bound back to 1e-3, and put y_op[1:4] back to zeros(4). 0.0400 is the same jump as 0.044.

Measured here (VSM at the test's operating point, 8 panels, tables regenerated on this box):

rel_err = 3.31e-6, repeatable to four significant figures — 300x inside the 1e-3 bound. CI's 0.0400 is 12000x that, so it is a discrete derivative jump, not tolerance drift.
Flat in freestream alpha: 3.23e-6 at 5.0 deg, 3.23e-6 at 6.0 deg, 3.31e-6 at 7.5 deg, 3.23e-6 at 10.0 deg.
Flat in theta: 3.314e-6 at theta = 0, and the same 3.314e-6 at 1e-12, -1e-12, 1e-9, -1e-9, 1e-6 and -1e-6; 3.316e-6 at 0.25 deg, 3.273e-6 at 1.0 deg, 3.307e-6 at -1.0 deg. There is no knife edge in theta anywhere between 0 and 1 deg.

That last line is the point: theta_idxs=1:4 feeds twist, and linearize passes delta_angles=nothing, so 320e756's defl_rad never moved the operating point off a delta knot. Its green run came from the 0.1 bound it set in the same commit, which ae11d48 was right to tighten. Restoring that operating point would buy nothing — worth writing down before someone reaches for it again.

Where the knife edge actually is, still open. Two candidates measured at the operating point:

Every panel's delta is exactly 0.0, which is exactly a knot of the table's delta_range = -3:3:3 deg. Whichever delta strip's slopes the interpolant selects at an exact knot decides the alpha-direction slopes, and the worst entry is always a theta column (2 or 3).
The tip panels solve to alpha 19.04 and 18.93 deg, outside the table's alpha_range = -5:5:15 deg, so they are extrapolated.

Deciding between them needs the environment that produces the 4% jump; it does not reproduce here. A fix that would follow from (1) is shifting delta_range so 0 is interior to a cell (e.g. deg2rad.(-4.5:3:4.5), putting the operating point 1.5 deg from the nearest knot, 2600x the 1e-5 finite-difference step) rather than moving an input that does not reach delta. Not done here: it changes what the test covers and I cannot show it red before and green after.

Answering the review comment on #292: measuring what FD step linearize's own default (fd_absstep/fd_relstep, the backend=nothing path) should carry, against a tight-solve AD reference rather than against AD at the same rtol. Moving that default if the numbers say so, plus the docstring line that misstates when it applies. Not touching the test's 1e-8/rtol=1e-11 pair, which is already green.

Reopening: the check flaked again, twice more, after #292 closed this.

main @ 6af7183, job 103414910220 (1.12 ubuntu, coverage): 0.04001.
Fit the contour generate_airfoils is handed, not a second wrap of it #294 @ 4b261f0, job 103534947632 (1.12 ubuntu, pull_request): 0.04207, with the other five test jobs on that same commit green — 1.11 ubuntu pull_request, both coverage cells, macOS aarch64 and windows x64.

Re-running that job on 4b261f0 with not one line changed turned it green, and with it all seven checks on the commit — so the state is in the runner, not the commit.

Same bimodal signature this issue already records: ~0.040 or ~1e-8, never anything between. #292 moved linearize's finite-difference default from 1e-3 to 1e-8 and tightened the bound to 1e-4; neither touches the jump.

Two candidate causes ruled out here, and candidate (2) is one of them.

Solver convergence noise amplified by the small step. rel_err stays in 3e-8…5e-8 across solver rtol 1e-11, 1e-10, 1e-9, 1e-8, 1e-7 crossed with finite-difference steps 1e-8, 1e-7, 1e-6, 1e-5, 1e-4. Flat — no U-curve, so the step size is not trading truncation against noise.

Candidate (2), the tip panels crossing the table's top knot. #294 fits the contour generate_airfoils is handed instead of wrapping it a second time, and that alone moves the tips off the extrapolated region:

`generate_airfoils` fitting	tip alphas [deg]	`norm(jac_fwd)`	rel_err here
`main` (contour wrapped twice)	19.0386 … 18.9322	2.2059	4.80e-8
#294 (wrapped once)	14.9906 … 14.9428	3.2971	4.84e-8

On main the tips sit 4 deg past the knot at 15 deg, in extrap_flat where dcl/dalpha is 0. With #294 they sit 0.009 deg below it. If the crossing were the knife edge, walking the operating point across it would show the jump — it does not. Freestream alpha 7.0 -> 8.2 deg in 0.1 deg steps takes the tips from 14.63 through 15.0 to 15.87 and rel_err never leaves 1.9e-8…5.6e-8. Wider: alpha 0:0.25:20 deg at zero sideslip, then alpha 5:0.5:10 crossed with beta -4:0.5:4 — 268 operating points, worst rel_err 2.3e-7, at alpha 19.5 deg. There is no knife edge anywhere on the operating-point axis of this box's fixture.

Which leaves the fixture itself, i.e. #291. The failing runner reports norm(jac_fwd) = 3.28551 where this box gets 3.29711 on the same commit, so its NeuralFoil Float32 tables are not the same tables. A discrete difference in them — remove_nan dropping a different set of alpha rows is the obvious one, the 1.12 log carries Removed 3 indices containing NaNs from aero_data — would change which knots exist, and that is the axis I have no way to walk from here. Reproducing the table, not the operating point, is what this needs next.

Not fixed in #294: that diff is three lines in src/airfoil_aero/geometry_gen.jl and a regression test, and this is a different concern.

A third occurrence, and the first outside ubuntu: Julia 1.12 - windows-latest - x64 - pull_request on #301 @ 5e78ab6, job 103560385346 — test/solver/test_forwarddiff.jl:87, relative_error(jac_fd, jac_fwd) = 0.04206403446037785, against the 1e-4 bound.

Two things this adds to what is already here.

It lands on the same two values as before. 0.04206 is the figure this issue already records for #294 @ 4b261f0 (job 103534947632), to four figures, and 0.04001 is the one from main @ 6af7183 and #285. Still ~0.040 or ~1e-8, never between, and now the ~0.042 branch has been hit twice on two different operating systems.

It is not ubuntu-specific, so it is not a property of one runner image. On that same commit the other four test jobs were green — 1.11 ubuntu, 1.12 ubuntu, 1.12 macOS aarch64 and the setup test — and the two 1.12 cells that differ are ubuntu (green, this run) and windows (0.04206). The previous run on the same branch @ 3c4fd7e had it the other way round: both 1.12 ubuntu cells at 0.03999947 and 0.04001102, windows green.

Consistent with this issue's own conclusion that the state is in the runner rather than in the commit, and it rules out "the ubuntu image's NeuralFoil Float32 tables" as the axis — whatever is discrete here reproduces on a different OS with a different BLAS.

A fourth occurrence, this one on main itself: the #308 merge, run 34696079829, Julia 1.12 - ubuntu-latest - x64 - push — 0.0420681631196805, twenty minutes after #301's windows job hit 0.04206403446037785. The ~0.042 branch, on two operating systems within half an hour, with no pull request in the frame.

More useful than the instance: a trap for anyone trying the "reproduce the table, not the operating point" lead above. ram_air_matrix_dir keys its generated directory on hash((n_sections, alpha_range, delta_range)) and on nothing about the code that fills it (test/test_data_utils.jl:46-49), and test/generated/ is gitignored. So a box that generated the fixture before a contour-generation change keeps testing the old contour indefinitely, while CI regenerates on every run.

It caught me today. My first test/solver/test_forwarddiff.jl run on a tree that already had #294 merged reported norm(jac_fwd) = 2.2058859447213175 — the doubly-wrapped, pre-#294 value in this issue's own table — from a directory written on 2026-09-11. After rm -rf test/generated/ram_matrix_*:

fixture	`norm(jac_fwd)`	`rel_err`
cached from 2026-09-11 (pre-#294 contour)	2.2058859447213175	green
regenerated on a post-#294 tree	3.2971127281317214	5.215614800118794e-8

So the 3.29711 in my previous comment was right, and the box's bimodal low branch is now 5.22e-8 rather than the 3.31e-6 recorded before #292 tightened the finite-difference step. Both are far inside 1e-4; the jump is still the only thing that crosses it.

The implication for this issue is that a local "cannot reproduce" is worth nothing until the fixture has been deleted first — and that the cache key omitting the generator's version is itself worth fixing, since it silently decouples what a box tests from the code it has checked out. I have not touched it: it is shared test infrastructure that nothing in #301 needs. Noted on #301 as a find.

Another instance, on #273 — but the run it came in gives a cleaner separation than an instance usually does, so it is worth the numbers rather than the count.

Run 34702041263 on 3dd5d948, one commit, one workflow run, attempt 1. norm_fd is the finite-difference reference and so tracks the table rather than the AD path:

job	`norm_fd`	`norm_fwd`	result
1.12 ubuntu x64	3.3230544105654944	3.3237655578200043	fail, 0.03933245
1.12 windows x64	3.3230544105751267	3.3245424692065706	fail, 0.03932894
1.12 macOS aarch64	3.323054410575126	3.3230545245914778	pass
1.11 ubuntu x64	3.335248455701877	3.3352485908653873	pass

norm_fd falls into exactly two clusters — 3.32305441 and 3.33524846 — inside a single run of a single commit, which is the table varying per runner and not the operating point, which is identical everywhere.

The jump is confined to one of them. On the 3.33524846 table norm_fwd is 3.3352485908653873 and the check passes every time: 1.11 ubuntu above, and both 1.12 jobs on attempt 2 after a plain re-run with no change (they go green, norm_fwd = 3.3352485908653873 on each). On the 3.32305441 table, aarch64's AD lands on 3.3230545245914778 and agrees, while both x86-64 jobs jump — to different values, 3.3237655… and 3.3245424…, yet to the same rel_err to four figures, which is the one-entry jump this issue describes.

That also answers "it does not reproduce here", beyond the fixture-cache trap above. With test/generated/ deleted and regenerated, this box produces norm_fd = 3.335248455636236, norm_fwd = 3.3352485908653873 — the benign table, bit-identical in norm_fwd to every green runner. The box does not fail to reproduce the jump because of a stale fixture; it fails because its freshly generated table is the one the jump does not live on. Getting the 3.32305441 table onto a box, rather than regenerating a table there, is what would make this walkable.

I have not touched the test — #273 is a panel_axes fix and only shifts the numbers slightly (its 0.0393 against the 0.0400/0.0420 seen elsewhere), it does not create the jump.

Across the last 30 CI.yml runs, whether the check fails follows the table the runner builds, not the commit. The table shows up as norm_fd in the log: A ≈ 3.2851809771, B ≈ 3.2971126036.

Table B: never fails. norm_fwd is always 3.2971127281317214 (~15 jobs: ubuntu 1.11/1.12/1.13, windows).
Table A, Julia 1.12 on x86-64: 9 of 12 jobs failed. Each OS lands on its own fixed value: ubuntu 0.0420681631196805 (norm_fwd 3.285478443737291), windows 0.04206403446037785 (3.2861963508418337).
Table A elsewhere: passes. That covers macOS aarch64 in 8 runs, ubuntu 1.11 in 2 jobs and ubuntu 1.13 in 2 jobs.
Re-runs that went green built table B: 35135066117 attempt 2 and 35124191885 attempt 3.
Region doesn't decide it. Both tables were built in westus3. PR Build a Solver from VSMSettings or from panel and section counts, deprecate the body_aero constructors, and check the sizes in solve! #340 got table A twice, in westus3 and then northcentralus, and printed the same norms both times.

No log records the CPU (show-versioninfo: false, no lscpu), so I can't confirm the hardware is what splits the tables. A versioninfo() or lscpu step in CI would.

Two more data points from 2026-09-16, both against the "table A on Julia 1.12 x86-64 fails" pattern above.

A new bad value. #352 @ 4965a5f, job 105008016132 (1.12 ubuntu, pull_request, Julia 1.12.7): relative_error = 0.0460232000169766, norm_fd = 3.285180977112122 (table A), norm_fwd = 3.2832728432488474. Every earlier ubuntu failure on table A landed on 3.285478443737291 (0.04207). Re-running the job with nothing changed turned it green on table B (norm_fwd 3.2971127281317214).

Table A passing on ubuntu 1.12. Same norm_fd 3.285180977112122, norm_fwd 3.285181098628289, check green: run 35158113156 (whose windows cell failed at 0.04206403446037785 in the same run), 35157225287 and 35155526534. So on one table and one OS the check sometimes passes and sometimes fails, and it has failed at two different values.

That branch's diff doesn't reach this path with its default settings: the three cells that passed on #352 print norm_fwd and norm_fd identical to all 16 digits to main @ 246237a (run 35156111861) on both tables.

Another instance, and the first time a test with no polar table failed in the same run.

#355 @ 1d76212, job 105013970853 (1.12 windows x64, Azure westus3): this check failed on table A at the known Windows value, 0.04206403446037785 (norm_fd 3.2851809771070055, norm_fwd 3.2861963508418337). In the same process, #355's new linearize: AutoForwardDiff matches AutoFiniteDiff (test/body_aerodynamics/test_body_aerodynamics.jl:594) also failed its rtol=1e-4 check. That test uses a two-wing INVISCID body, which reads no table at all.

Re-running the job with nothing changed (westus2) built table B (norm_fd 3.2971126035795564, norm_fwd 3.2971127281317214), and both tests passed.

What the INVISCID failure shows:

Only four corners of each Jacobian are in the log: rows 1, 2, 11, 12 × columns 1, 2, 17, 18. All 16 printed entries of both jac_fwd and jac_fd are bit-identical to the same test on an i5-13500 here, e.g. jac_fwd[1,1] = -6.476121593442287 and jac_fd[1,1] = -6.476064724481034. So one or more hidden entries moved, not the whole solve.
Here the relative norm error is 2.689e-7, 370× inside the bound.
It stays flat when every twist and deflection input shifts by ±1e-12, ±1e-9 or ±1e-7.
It is bit-identical with BLAS on 1, 2, 4 or 10 threads, and after filling the heap with 0, 1e30 or NaN before each solve.
Tightening the solver's rtol from 1e-8 down to 1e-14 moves the FD Jacobian by at most 2e-9.
A read of the LOOP/INVISCID solve path found no buffer read before it is written, and no state carried between solve! calls.

So whatever makes a runner "table A" also reaches a solve that never touches a table. That points past NeuralFoil and the table's contents, towards the runner itself (CPU and code generation). The only runner detail the two logs differ in is the Azure region. The next step I'd take is to print Sys.cpu_info()[1].model in the CI log, and the worst (row, column) entry when either check fails, so the next instance says which one it was.

The offset grid from f8761aa (#360, alpha_range=deg2rad.(-2:5:23)) did not end the per-runner table split. There are still two tables. Call them A′ (norm_fd ≈ 2.6186324991) and B′ (≈ 2.6237859245). A job built on B′ has failed on Windows again.

Fail, B′: set_va! turns the body about a stored reference_point, and set_va!(body_aero, settings) applies yaw_rate #353 @ 8a2b61d (merges f8761aa), 1.12 windows x64, run 35223368928. relative_error = 0.046430340012943265, norm_fwd = 2.622107445593296, norm_fd = 2.6237859245239954. Attempts 1 and 2 gave the same bits. Attempt 3 then passed on the same table (norm_fwd = 2.6237860087686893), so B′ does not decide the failure by itself.
Pass, A′, same commit, same OS: workflow_dispatch run 35229327523, 1.12 windows. norm_fwd = 2.618632581116621, norm_fd = 2.6186324991070755. These are the bits main produces on Windows.
main @ f8761aa moves between tables too: push run 35223316047 ubuntu 1.12 got A′. Dispatch run 35229324579 ubuntu 1.13 got B′ (fwd 2.6237860087686893, fd 2.6237859245239954) and passed. macOS got B′ in every run.
Locally (Linux, Julia 1.12.7): on the same generated table, set_va! turns the body about a stored reference_point, and set_va!(body_aero, settings) applies yaw_rate #353's code and main's give identical bits (A′). A sweep of 144 one-cell ±1e-4 edits to the polar CSVs near the operating point never moved the result to B′ and never failed. So B′ is not rounding noise in the CSV values.

So f8761aa moved the kink off the tip cluster for table A′, but B′ still reaches a point where ForwardDiff and finite differences disagree by 4.6% on Windows.

@1-Bort-1

@1-Bort-1 open a PR fixing the test such that it is not on the knife edge.

Navigation Menu

Repository navigation

test_forwarddiff.jl's POLAR_MATRICES check flakes at ~4%, and the oper #287

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

test_forwarddiff.jl's POLAR_MATRICES check flakes at ~4%, and the oper #287

Description

Activity

1-Bort-1 commented on Sep 9, 2026

1-Bort-1 commented on Sep 12, 2026

1-Bort-1 commented on Sep 12, 2026

1-Bort-1 commented on Sep 12, 2026

1-Bort-1 commented on Sep 12, 2026

1-Bort-1 commented on Sep 16, 2026

1-Bort-1 commented on Sep 16, 2026

1-Bort-1 commented on Sep 16, 2026

1-Bort-1 commented on Sep 17, 2026

1-Bart-1 commented on Sep 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions