Record audited nonlinear obstruction findings and next experiment

experiments/NONLINEAR_STEP_INVESTIGATION_2026-09-18.md

@@ -1,5 +1,23 @@
 # Nonlinear step obstruction after gravity preconditioning
 
+The completed frozen-state investigation supports a consistent directional
+Jacobian and a narrow range of useful trial steps at the n = 1.5, J = 0
+endpoint. Tightening the full-J forcing tolerance tenfold costs about 70%
+more linear work without improving the useful sampled step. Compared with
+the seed, the late direction is 17.35x larger despite a 19.59x smaller
+residual; strongly cancelling shape and field responses amplify otherwise
+moderate corrected-path curvature.
+
+Gravity correction cuts the remainder norm but can increase its adverse
+merit projection through the retained equations. A bounded fully coupled
+trial correction is the next recommended experiment. This branch implements
+diagnostics and records evidence; it makes no production numerical-policy,
+physics API, normalization or acceptance-threshold changes.
+
+Three immutable probes cover 144 trial-path observations and 60 successful
+auxiliary gravity solves. Their independent final audits pass 20,899 checks;
+the new checker passes fourteen synthetic tests. No new model is accepted.
+
 ## Recorded decision from the completed preconditioner investigation
 
 The completed report is `PRECONDITIONER_IMPROVEMENT_2026-09-18.md`, committed
@@ -133,5 +151,334 @@ The diagnostic saves the frozen vectors, layout and normalization, checks
 restoration and accepted-buffer invariance, and reports failed preparations
 and inner solves without inventing values for missing trials.
 
-Results are pending. No numerical remedy or new convergence claim follows
-from the investigation design alone.
+## Late-state results at forcing tolerance 0.03
+
+The first campaign, `late_v1`, completed in 696.63 s using executable source
+`ba057b3983ff6006e460a1fa80e3e6523ceeb4a5`. Its independent review
+`late_v1_review.json` passes 162 provenance checks and 6,238 saved-vector,
+arithmetic, layout, failure-semantics, and restoration checks. The checker
+and its fourteen focused synthetic tests are committed at `b8d1c6a`.
+All saved base residual and Jacobian-action restoration differences are zero;
+accepted physical/normalized/residual buffers and normalization factors remain
+bitwise identical. There are no accepted model commits or positivity repairs.
+
+The cold gravity-Schur solve takes 84 iterations and 47.9374 s. A separate full
+Jacobian application verifies relative defect 0.0295592 against the requested
+0.03 gate. The base norm is 1.3966991044e-5, the direction norm is 0.0794897,
+and the merit slope is -1.94906e-10. Thus the computed direction is a strong
+local descent direction for this fixed normalized residual.
+
+| Alpha | Raw residual / base | Gravity-corrected residual / base |
+| --- | ---: | ---: |
+| 1, 1/4, 1/16 | pressure preparation fails | blocked by raw preparation |
+| 1/64 | 1.807268 | 1.080323 |
+| 1/256 | 1.002510 | 1.002025 |
+| 1/1024 | 0.999170 | 0.999394 |
+| 1/4096 | 0.999764 | 0.999779 |
+| 1/16384 | 0.999940 | 0.999940 |
+
+These are sampled trial residuals, not a production line search or a claim
+that one of these alphas is optimal. The three smaller decreasing steps
+also satisfy the ordinary sufficient-decrease metric. Full and fields-only
+large steps fail with `thermodynamic_domain` at `pressure_force`; shape-only
+steps prepare across the grid. Eighteen auxiliary gravity solves pass, and
+all 36 prepared raw/corrected paths pass their applicable gates. The absence
+of pressure rejections in the preceding trajectory is consistent with these
+new observations at a different endpoint and direction.
+
+### The measured tangent is consistent
+
+For both full paths, independently recomputed vector remainders have resolved
+order approximately two from alpha 1/64 through 1/16384. Shrinking alpha by
+four shrinks ||R||/alpha approximately fourfold. Raw ||R||/alpha^2 stabilizes
+near 0.085426; corrected stabilizes near 0.005807. No persistent first-order
+directional discrepancy is detected over this range. This verifies the
+sampled direction and scales, not every Jacobian column or arbitrary states.
+
+Despite this consistency, corrected ||R||/(alpha ||Jp||) is still 1.63 at
+alpha 1/256, 0.406 at 1/1024, and 0.0254 at 1/16384. The first-order residual
+response is so small that formally second-order terms can dominate it well
+before pressure preparation fails.
+
+### The direction couples large cancelling responses
+
+||J p_shape|| = 0.7730368 and ||J p_fields|| = 0.7730377, while
+||J p|| = 1.396089e-5. The cancellation factor
+(||J p_shape||+||J p_fields||)/||J p|| is 110,743, and the image cosine is
+-0.999999999838. All are measured in the unchanged diagonal normalization.
+
+The full directional gain ||Jp||/||p|| is 1.7563e-4; the shape-only gain
+is 15.933. Their ratio supplies a lower bound of approximately 9.07e4 on
+the normalized two-norm condition number if this finite-dimensional Jacobian
+is nonsingular. It is not an estimate of that condition number, a discovered
+nullspace, or evidence identifying a particular physical symmetry. In
+particular, it does not justify imposing an additional central-density
+constraint on this n = 1.5 model.
+
+### Smaller remainder does not guarantee better merit
+
+At alpha 1/1024, gravity correction reduces ||R|| by about 14.7x, but raises
+the adverse projection <F+alpha Jp,R> from 2.49849e-14 to 7.20763e-14, a
+factor of 2.88. This explains why its residual is slightly worse than the
+raw residual at this small step, despite its much smaller remainder.
+
+The corrected adverse projection splits approximately as follows:
+
+| Retained residual block | Share of signed total projection |
+| --- | ---: |
+| Hydrostatic balance | 43.52% |
+| Barotropic closure | 32.33% |
+| Surface balance | 19.53% |
+| Total mass | 4.63% |
+
+Gravity contributes only arithmetic-scale terms. The raw hydrostatic
+projection is small and favorable; correction makes it the largest adverse
+contribution. The same signed full-predictor projection decomposes into
+1.18% corrected shape-direction remainder, 47.75% corrected fields-direction
+remainder, and 51.07% mixed remainder. Even the corrected shape-direction
+trial changes gravity-field values. These terms sum by vector identity;
+the mixed term includes interactions introduced by the correction. They
+must not be identified with the pure shape/field Hessian of the raw operator.
+
+## Matched direction-accuracy control
+
+The tighter late-state solve passes its requested 0.003 true-defect gate:
+143 iterations, 81.6901 s, verified relative defect 0.00294840. The physical
+base, normalized base, F, RHS, both normalization-factor files and layout match
+the looser experiment byte-for-byte. Its individual audit passes 146 producer
+and hash checks and 6,238 vector/invariant checks, with zero restoration drift
+and bitwise unchanged accepted buffers/factors. All eighteen auxiliary solves
+pass. The final controls campaign audit also passes: 349 provenance checks,
+6,238 tight-case vector/invariant checks and 7,912 seed-case checks.
+
+| Quantity | Eta = 0.03 | Eta = 0.003 |
+| --- | ---: | ---: |
+| Verified relative full-J defect | 0.0295592 | 0.00294840 |
+| Cold linear iterations | 84 | 143 |
+| Cold linear seconds | 47.9374 | 81.6901 |
+| Normalized direction norm | 0.0794897 | 0.0796012 |
+| Corrected residual ratio at alpha 1/64 | 1.080323 | 1.080567 |
+| Corrected residual ratio at alpha 1/256 | 1.002025 | 1.002037 |
+| Corrected residual ratio at alpha 1/1024 | 0.999394139 | 0.999394272 |
+
+For w = p_tight-p_loose, ||w|| = 7.06262e-4, or 0.8885% of ||p_loose||.
+The direction cosine is 0.99996157. ||Jw|| = 4.10794e-7 and
+||Jw||/||w|| = 5.81645e-4. Thus improving linear accuracy tenfold has not
+substantially changed the large cancelling direction or its useful sampled
+step range. Both full paths again have resolved approximately quadratic
+remainders; the corrected remainder actually grows about 0.272% at alpha
+1/1024. The tighter solve costs about 70% more iterations and diagnostic
+solve time. These are single diagnostic timings, not repeated performance
+benchmarks; the accuracy and iteration comparison is the principal control.
+
+### A smaller direction in this two-vector span offers little
+
+A supplementary algebraic experiment solves
+
+    minimize ||s||,  s in span(p_loose,p_tight),
+    subject to ||F+Js|| <= 0.03 ||F||.
+
+It uses an orthonormal direction basis, QR/SVD of its two-column action, a
+scalar secular solve and a convex KKT check. Complete saved-vector arithmetic
+independently checks the candidate and its action; no new FEM operation is
+performed. The two-dimensional reduced action condition is 3.3594 and the
+KKT relative residual is 1.97e-14, so the small result is resolved within this
+algebraic problem.
+
+| Direction | Norm |
+| --- | ---: |
+| Original loose direction | 0.0794897230590 |
+| Original tight direction | 0.0796012421522 |
+| Best feasible scalar scaling of tight | 0.0772247555310 |
+| Minimum norm in their span | 0.0772243033891 |
+
+The additional reduction beyond ordinary scalar damping is only 0.00058549%.
+The candidate's defect using the complete saved actions is 0.0299999999970.
+That tiny inward margin is not a certificate that a fresh FEM application
+would pass exactly at the boundary. No candidate trial is evaluated, and no
+nonlinear improvement is claimed. This deprioritizes selection in this
+particular two-vector span; it does not rule out a richer Krylov subspace or
+a trust-region method with an explicit non-Newton acceptance contract.
+
+The clean algebraic evidence is `minimum_norm_pair_v4/assessment.json`, with
+input/output hashes and reproduction instructions. The preliminary v1
+preflight correctly refused the tight case's unflushed layout. A v2 run
+produced unexpected NumPy floating-status warnings in two-column BLAS
+products despite finite results; v3 replaces those products with explicit
+two-column arithmetic and enables floating exceptions. The clean rerun gives
+the same result. `analyze_nonlinear_direction_pair.py` preserves that
+reproduction logic in this branch at `db93bf7`; v4 reruns that version with
+additional RHS-hash and immutable-directory checks.
+
+`minimum_norm_pair_v4_binding_review.json` passes 61 additional checks
+binding this assessment and its reproduced vectors to the final audited
+producers. That closes the assessment's pending provenance qualification;
+its explicit lack of a fresh FEM action or nonlinear trial remains.
+
+## Initial-state control
+
+The initial n = 1.5, J = 0 model has base residual 2.7365921762e-4.
+Its cold solve passes in 26 iterations and 15.0205 s, with verified relative
+defect 0.0281993. The raw full step prepares and reduces the residual to
+1.7298999698e-4. Gravity correction reduces it to 1.61484154945e-5: a 94.10%
+decrease, reproducing to roundoff the recorded residual norm of the preceding
+production campaign's first accepted step. The late model's best sampled
+corrected decrease is only 0.06059%, at alpha 1/1024.
+
+Both normalization-factor vectors are byte-identical across these states.
+The following comparison therefore uses the same normalized coordinates,
+although it samples different states and directions:
+
+| Quantity | Initial model | Late model, eta = 0.03 |
+| --- | ---: | ---: |
+| ||F|| | 2.736592e-4 | 1.396699e-5 |
+| ||p|| | 0.00458195 | 0.0794897 |
+| ||p||/||F|| | 16.7433 | 5691.26 |
+| Shape/fields action cancellation factor | about 332.1 | about 110,743 |
+| Corrected ||R||/(alpha^2 ||p||^2), alpha = 1/64 | 0.656881 | 0.926040 |
+| Corrected ||R||/(alpha^2 ||F||), alpha = 1/64 | 0.0503939 | 418.937 |
+
+At the common alpha 1/64 both full corrected remainders are resolved above
+the heuristic floor. The late direction is 17.35x larger despite a 19.59x
+smaller residual; ||p||/||F|| increases about 340x. By contrast, sampled
+curvature per unit direction increases only 1.41x. The relative quadratic
+remainder coefficient ||R||/(alpha^2 ||F||) grows approximately 8,313x.
+This describes the observed amplification; it does not estimate a full
+Hessian norm or identify a physical nullspace.
+
+For these two sampled corrected paths at the common alpha, the main change
+is not orders-of-magnitude growth in remainder per squared unit-direction
+step. This does not isolate intrinsic raw-operator or shape-Hessian curvature.
+A much larger, strongly cancelling Newton direction is
+being asked to eliminate a much smaller residual. Small second-order errors
+relative to the individual shape/field responses become large relative to
+the desired combined response. All 48 seed paths prepare, all 24 auxiliary
+gravity solves pass, and restoration differences are zero. Corrected full
+vector remainders show resolved order approximately two from alpha 1 through
+1/256. Smaller corrected seed remainders are excluded from order claims by
+the heuristic screen including the gravity arithmetic allowance.
+
+## Implications and next intervention
+
+The measured evidence favors a bounded, fully coupled second-order trial
+correction as the next isolated intervention. Keep the verified Newton
+direction and capture the entire affine target L = F0+alpha J0p. For a
+prepared trial y, solve J(y)c = -(F(y)-L), prepare y+c, and evaluate the
+complete unchanged residual. This targets the closure, hydrostatic, surface
+and mass terms that a gravity-only correction leaves or introduces.
+
+For a locally bounded inverse and a smooth residual, c is second order in
+alpha and preserves the original tangent. The observed weak response makes
+that boundedness assumption uncertain here. The experiment should therefore
+limit correction norm and work, try at most one correction at a selected
+prepared trial, and keep the uncorrected candidate available as a fallback.
+Recheck geometry and the selected enthalpy-admissibility policy after the
+correction, since it can change both shape and enthalpy. Keep that policy
+identical for the corrected candidate, fallback and control: this campaign
+uses sampled enthalpy without polynomial repair. Adding a polynomial
+certificate requires a separate, matched admissibility control. This does
+not bypass the pressure preparation failures at large alpha; it tests whether the prepared range can
+deliver substantially more useful progress.
+
+Architecturally, implement this as a distinct, defaulted compile-time trial
+path policy composed by `PreparedNewtonStrategy`, not as another gravity
+specialization. Capture F0 and J0p in `CaptureDirection` before trial mutation;
+compose correction in `PrepareTrial`; explicitly refresh the full inverse
+after changing the prepared state. The ordinary full-residual acceptance
+test and accepted-state commit remain in `stellar_context.cppm`. The public
+physics model and existing equations/normalization need no new arguments.
+
+Independently verify the auxiliary full defect ||J(y)c+F(y)-L|| against its
+own declared solve tolerance. Preserve and report the base Newton direction's
+original full-J gate separately; an auxiliary success must not replace it.
+
+Compare full cost per useful accepted step, including extra solves,
+preparations and rejected attempts. Removing a correction residual at high
+cost is not itself a solver improvement. Use the same late snapshot, inverse,
+forcing tolerance and virial/field diagnostics as the control; a subsequent
+short nonlinear run is necessary before claiming convergence improvement.
+
+Tenfold tighter forcing and minimum-norm selection in the two measured
+directions are lower priorities after their negative controls. Broader
+Krylov-space regularization remains untested. A trust step that intentionally
+misses the current Newton gate would need an explicit distinct contract;
+solving a shifted system accurately cannot be reported as passing the
+original full-J gate.
+
+Continuation in n and then angular momentum remains a separate path for
+testing basin dependence, not a demonstrated cure for this local weak
+response. A production continuation driver needs a typed checkpoint/state
+transfer initializer: the current ordinary initializer requires a radial
+seed, whereas the experimental physical snapshot loader is not a general
+rotating-model restart API. Record the EOS K(n) path, mass, angular momentum,
+mesh and normalization. Nothing here establishes a need for an extra n = 1.5
+central-density constraint, a translation gauge, or pseudo-arclength.
+
+No new equilibrium has been accepted in this investigation. The input late
+state still has the previously measured negative sampled density and
+undefined density-pressure virials, even though its enthalpy-pressure
+virials pass 1e-5. Both internal admissibility and the virial milestone remain
+required. These n = 1.5, J = 0 controls make no new claim about higher n,
+rotating convergence, or any percentage of critical rotation.
+
+## Reproduction and artifact scope
+
+The campaign manifests are `nonlinear_step_late_case_2026-09-18.json` and
+`nonlinear_step_controls_2026-09-18.json`. Each records every probe, linear,
+and gravity-solve option. Run them serially through `run_plateau_campaign.py`
+with the configured Release build, the original `sandbox.smesh`, and a new
+output directory. The existing evidence root is the task workspace's
+`diagnostics/nonlinear_step_2026-09-18/` directory. Its campaign directories
+are immutable; reviews, analysis and figures are kept alongside them.
+
+`late_v1` was built at `ba057b3`; `controls_v1` at `b8d1c6a`, which only adds
+the independent checker and tests. Both copied executables have SHA-256
+`5f5c4ed0dae351097ec1f632fa4132e490b35c6b8ec3aab689c5a6535b77c634`.
+The mesh SHA-256 is
+`ce11ae99e21e6c3bbfbee402acd2e191c1da0d8261d2227b4203f73f2f337a74`.
+The recorded physical snapshot, frozen normalized state, residual, factors,
+and layout provide an exact binding for the paired tolerance control.
+
+The checker reads every saved trial state/residual and recomputes the vector
+remainders, signed projections, mixed interactions and norm identities.
+It also checks the complete path grid, failed-preparation semantics, budgets,
+hashes, input binding, gravity affine-target gates, counters, frozen-state
+restoration and accepted-buffer invariants. A failed linear gate remains a
+valid finite directional Taylor experiment, but cannot support a claimed
+Newton-forcing comparison. Its fourteen synthetic tests include intentional
+vector corruption, a hidden first-order error, noninteger superlinear order,
+unresolved arithmetic-scale differences, missing hashes/invariants and
+failed preparation/inner/prepared-gravity gates.
+
+Example independent review commands, replacing E with the evidence-directory
+path and keeping reviews outside the immutable campaigns:
+
+```text
+python3 experiments/check_nonlinear_step.py E/late_v1 --output E/late_v1_review.json
+python3 experiments/check_nonlinear_step.py E/controls_v1 --output E/controls_v1_review.json
+python3 -m unittest discover -s experiments -p test_nonlinear_step_checker.py -v
+```
+
+The final comparison is saved as
+`figures_v1/nonlinear_step_comparison.png`, `.pdf` and `.json`. Its JSON
+records source reviews/hashes and numerical comparisons. Reproduce it with
+`plot_nonlinear_step.py --audited-campaign E/late_v1 E/late_v1_review.json
+--audited-campaign E/controls_v1 E/controls_v1_review.json --output E/figures_new`,
+using a Python installation with NumPy and Matplotlib. This run used the
+existing `/opt/homebrew/anaconda3/envs/general/bin/python`; no dependencies
+were installed. The final PNG was visually inspected.
+
+Final numerical audits comprise 162 + 349 provenance checks and 6,238 +
+6,238 + 7,912 case checks, totaling 20,899. These are arithmetic/provenance
+checks, not independent physical validation tests. The supplementary
+two-direction assessment has 61 additional binding checks. All three probes
+restore the original residual and actions exactly, preserve accepted buffers
+and factors bitwise, and perform no accepted-state commits.
+
+Input-snapshot equality is additionally recorded in `late_v1_input_binding.json`:
+all 178,073 supplied physical coefficients match exactly. The generic checker
+verifies the supplied file's provenance hash and physical/normalized factors;
+that separate coefficient check binds the actual imported values themselves.
+Reported operator norms use the configured diagonal normalization, not exact
+physical L2 norms. The subtraction/gravity allowance is a conservative
+heuristic resolution screen, not a certified bound on FEM assembly error.