MeanField/experiments/NONLINEAR_STEP_INVESTIGATION_2026-09-18.md

Nonlinear step obstruction after gravity preconditioning

The completed frozen-state investigation supports a consistent directional Jacobian and a narrow range of useful trial steps at the n = 1.5, J = 0 endpoint. Tightening the full-J forcing tolerance tenfold costs about 70% more linear work without improving the useful sampled step. Compared with the seed, the late direction is 17.35x larger despite a 19.59x smaller residual; strongly cancelling shape and field responses amplify otherwise moderate corrected-path curvature.

Gravity correction cuts the remainder norm but can increase its adverse merit projection through the retained equations. A bounded fully coupled trial correction is the next recommended experiment. This branch implements diagnostics and records evidence; it makes no production numerical-policy, physics API, normalization or acceptance-threshold changes.

Three immutable probes cover 144 trial-path observations and 60 successful auxiliary gravity solves. Their independent final audits pass 20,899 checks; the new checker passes fourteen synthetic tests. No new model is accepted.

Recorded decision from the completed preconditioner investigation

The completed report is PRECONDITIONER_IMPROVEMENT_2026-09-18.md, committed at dc1da62 on research/preconditioner-benchmarks-20260918. Its immutable campaigns live under the task workspace's diagnostics/preconditioner_improvement_2026-09-18/ directory.

• Consistent gravity Schur refinement solves the saved n = 1.5, J = 0 linear system in 132 iterations and 72.3727 s, versus 231 iterations and 100.9943 s for the independent control. Both satisfy the same independently verified relative full-J defect of 0.03: a measured 1.39548x speedup.
• Combining this refinement with stellar LDU solves the saved rotating n = 1, J = 0.1 system in 142 iterations and 96.8236 s. The independent and gravity-only controls still fail after 600 iterations. Their finite time-to-convergence is unknown, so no rotating speedup ratio is reported.
• Prepared geometry reuse makes the existing stellar LDU application 8.0425x cheaper without improving its convergence by itself. Retain this implementation optimization independently of policy selection.
• The tested nine-mode shape correction does not improve the n = 1.5 case. It remains an experimental negative result, not a rejection of all coarse spaces.
• The easy n = 1, J = 0 model preserves all eight virials below 3.74e-7, while the stronger inverse costs about 4% more. Defaults remain unchanged.
• Neither hard nonlinear model is physically acceptable. Negative sampled density makes density-pressure virials undefined. In rotation, a smaller nonlinear residual accompanied a larger enthalpy-pressure virial error. Residual decrease is not a substitute for virial and field-admissibility validation. No angular momentum has been calibrated to critical rotation.

New investigation and controls

Branch: research/nonlinear-step-obstruction-20260918, based on dc1da62. The configured worktree is reused to avoid an unrelated full rebuild; the preceding branch and copied campaign executables retain their exact source and binary identities. This branch adds experimental diagnostics only.

Start with the n = 1.5, J = 0 endpoint of newton_v1/n1p5_J0_curvature_gravity_schur. All twelve outer linear solves and all 31 auxiliary gravity solves passed, but the final nonlinear residual was 1.3966991e-5 against a target of 2.7365922e-12. The final accepted alpha was 1/512. Every trial prepared successfully; all nineteen rejections were residual-decrease failures, and geometry never limited the step.

That campaign did not retain every raw Newton direction. The new late-state probe computes a fresh direction at its saved final endpoint; it does not claim to replay the preceding iteration's rejected trials. A plateau in a bounded twelve-step run is also not proof that all subsequent steps must fail. The fresh trial measurements can test that possibility.

The final squared-residual shares are approximately 42.42% hydrostatic, 26.23% closure, 15.87% surface, 10.43% gravity, and 5.05% mass. This motivates examining retained-equation curvature after the gravity correction, rather than assuming the earlier gravity or pressure-preparation failure persists. The last auxiliary prepared defect, approximately 3.33e-14, measures agreement with the affine gravity target, not satisfaction of the gravity equations. The final gravity-block residual norm is approximately 4.51e-6.

Keep the complete physical equations, fixed normalization, optional central-density specification (absent here), mesh, and acceptance thresholds unchanged. Use one gravity Schur refinement at relaxation 0.5 and the same restart length 160. Fresh cold solves and independently applied full-J true defects determine direction quality. Trial probes never commit a model.

Questions and discriminating evidence

1. Does the prepared trial path have the expected tangent? For frozen residual F and computed direction p, measure Jp and actual prepared differences. The curvature-corrected path should preserve that tangent because its gravity correction is second order in alpha. A discrepancy that remains first order above arithmetic noise would implicate the linearization or preparation path.
2. Does the inexact linear direction cause the obstruction? Compare directions at the same saved state with forcing tolerances 0.03 and 0.003, subject to a recorded finite iteration budget. A tighter requested tolerance counts only if its independent true gate passes. Failed solves are recorded and do not substantiate this comparison.
3. Which nonlinear terms oppose the linear prediction? Split p into its actual surface coordinates and all remaining coordinates. Compare raw and gravity-corrected trials for full, shape-only and fields-only directions. Form mixed remainders by vector subtraction. Signed projections against the full linear prediction distinguish adverse curvature from large but cancelling terms.
4. Is the behavior specific to the late state? Use the same-polytrope Lane–Emden seed as an early-state control, where the previous production run accepted its first full Newton step. Record preparation and enthalpy feasibility separately from the residual metric.

For phi = ||F||^2/2 and ||Jp + F|| <= eta ||F||,

phi'(0) = <F,Jp> <= -(1-eta) ||F||^2.

At eta = 0.03 the slope must be strongly negative if J differentiates the prepared path. The production sufficient-decrease test therefore ought to accept sufficiently small resolved steps. The investigation measures whether large adverse higher-order terms explain how small those steps become.

Each block's vector remainder R = F_trial - F - alpha Jp must satisfy

||F_trial||^2 = ||F + alpha Jp||^2
               + 2 <F + alpha Jp, R> + ||R||^2.

For the corrected trial map, its mixed remainder includes interactions introduced by gravity correction; it is not identified solely with the raw physical shape Hessian. Below-floor differences are unresolved, not evidence of exact linearity.

The EOS provides an additional interpretation limit: rho(h) is proportional to max(h,0)^1.5 here. At an exact zero with a positive enthalpy direction, the closure remainder can scale as alpha^1.5 while its first derivative is correct. A superlinear slope different from two does not alone establish a Jacobian defect; the direct tangent check is whether R/alpha decreases in a resolved regime. The strong zero-pressure boundary row is already expressed as h-h(P_surface), with derivative delta h, so it does not inherit a vanishing dP/dh boundary row.

The paired directions also permit a conditioning diagnostic without another operator solve: w = p_tight - p_loose and Jw = Jp_tight - Jp_loose. A large change in direction with a small action demonstrates a weak response in that direction, not a proved nullspace or a particular symmetry. Measure the shape/fields action cancellation as well. Fixed-K, fixed-M n = 1.5 does not have the familiar exact n = 3 mass-preserving homology; no additional gauge or central-density constraint is justified by that analogy.

Execution and validation

Run a late-state probe first, then use its evidence to choose the matched tighter-direction and early-state controls. All model jobs and builds run serially. The campaign runner records clean source commits, exact commands, input/output and executable hashes, and refuses existing output directories. The diagnostic saves the frozen vectors, layout and normalization, checks restoration and accepted-buffer invariance, and reports failed preparations and inner solves without inventing values for missing trials.

Late-state results at forcing tolerance 0.03

The first campaign, late_v1, completed in 696.63 s using executable source ba057b3983ff6006e460a1fa80e3e6523ceeb4a5. Its independent review late_v1_review.json passes 162 provenance checks and 6,238 saved-vector, arithmetic, layout, failure-semantics, and restoration checks. The checker and its fourteen focused synthetic tests are committed at b8d1c6a. All saved base residual and Jacobian-action restoration differences are zero; accepted physical/normalized/residual buffers and normalization factors remain bitwise identical. There are no accepted model commits or positivity repairs.

The cold gravity-Schur solve takes 84 iterations and 47.9374 s. A separate full Jacobian application verifies relative defect 0.0295592 against the requested 0.03 gate. The base norm is 1.3966991044e-5, the direction norm is 0.0794897, and the merit slope is -1.94906e-10. Thus the computed direction is a strong local descent direction for this fixed normalized residual.

Alpha	Raw residual / base	Gravity-corrected residual / base
1, 1/4, 1/16	pressure preparation fails	blocked by raw preparation
1/64	1.807268	1.080323
1/256	1.002510	1.002025
1/1024	0.999170	0.999394
1/4096	0.999764	0.999779
1/16384	0.999940	0.999940

These are sampled trial residuals, not a production line search or a claim that one of these alphas is optimal. The three smaller decreasing steps also satisfy the ordinary sufficient-decrease metric. Full and fields-only large steps fail with thermodynamic_domain at pressure_force; shape-only steps prepare across the grid. Eighteen auxiliary gravity solves pass, and all 36 prepared raw/corrected paths pass their applicable gates. The absence of pressure rejections in the preceding trajectory is consistent with these new observations at a different endpoint and direction.

The measured tangent is consistent

For both full paths, independently recomputed vector remainders have resolved order approximately two from alpha 1/64 through 1/16384. Shrinking alpha by four shrinks ||R||/alpha approximately fourfold. Raw ||R||/alpha^2 stabilizes near 0.085426; corrected stabilizes near 0.005807. No persistent first-order directional discrepancy is detected over this range. This verifies the sampled direction and scales, not every Jacobian column or arbitrary states.

Despite this consistency, corrected ||R||/(alpha ||Jp||) is still 1.63 at alpha 1/256, 0.406 at 1/1024, and 0.0254 at 1/16384. The first-order residual response is so small that formally second-order terms can dominate it well before pressure preparation fails.

The direction couples large cancelling responses

||J p_shape|| = 0.7730368 and ||J p_fields|| = 0.7730377, while ||J p|| = 1.396089e-5. The cancellation factor (||J p_shape||+||J p_fields||)/||J p|| is 110,743, and the image cosine is -0.999999999838. All are measured in the unchanged diagonal normalization.

The full directional gain ||Jp||/||p|| is 1.7563e-4; the shape-only gain is 15.933. Their ratio supplies a lower bound of approximately 9.07e4 on the normalized two-norm condition number if this finite-dimensional Jacobian is nonsingular. It is not an estimate of that condition number, a discovered nullspace, or evidence identifying a particular physical symmetry. In particular, it does not justify imposing an additional central-density constraint on this n = 1.5 model.

Smaller remainder does not guarantee better merit

At alpha 1/1024, gravity correction reduces ||R|| by about 14.7x, but raises the adverse projection <F+alpha Jp,R> from 2.49849e-14 to 7.20763e-14, a factor of 2.88. This explains why its residual is slightly worse than the raw residual at this small step, despite its much smaller remainder.

The corrected adverse projection splits approximately as follows:

Retained residual block	Share of signed total projection
Hydrostatic balance	43.52%
Barotropic closure	32.33%
Surface balance	19.53%
Total mass	4.63%

Gravity contributes only arithmetic-scale terms. The raw hydrostatic projection is small and favorable; correction makes it the largest adverse contribution. The same signed full-predictor projection decomposes into 1.18% corrected shape-direction remainder, 47.75% corrected fields-direction remainder, and 51.07% mixed remainder. Even the corrected shape-direction trial changes gravity-field values. These terms sum by vector identity; the mixed term includes interactions introduced by the correction. They must not be identified with the pure shape/field Hessian of the raw operator.

Matched direction-accuracy control

The tighter late-state solve passes its requested 0.003 true-defect gate: 143 iterations, 81.6901 s, verified relative defect 0.00294840. The physical base, normalized base, F, RHS, both normalization-factor files and layout match the looser experiment byte-for-byte. Its individual audit passes 146 producer and hash checks and 6,238 vector/invariant checks, with zero restoration drift and bitwise unchanged accepted buffers/factors. All eighteen auxiliary solves pass. The final controls campaign audit also passes: 349 provenance checks, 6,238 tight-case vector/invariant checks and 7,912 seed-case checks.

Quantity	Eta = 0.03	Eta = 0.003
Verified relative full-J defect	0.0295592	0.00294840
Cold linear iterations	84	143
Cold linear seconds	47.9374	81.6901
Normalized direction norm	0.0794897	0.0796012
Corrected residual ratio at alpha 1/64	1.080323	1.080567
Corrected residual ratio at alpha 1/256	1.002025	1.002037
Corrected residual ratio at alpha 1/1024	0.999394139	0.999394272

For w = p_tight-p_loose, ||w|| = 7.06262e-4, or 0.8885% of ||p_loose||. The direction cosine is 0.99996157. ||Jw|| = 4.10794e-7 and ||Jw||/||w|| = 5.81645e-4. Thus improving linear accuracy tenfold has not substantially changed the large cancelling direction or its useful sampled step range. Both full paths again have resolved approximately quadratic remainders; the corrected remainder actually grows about 0.272% at alpha 1/1024. The tighter solve costs about 70% more iterations and diagnostic solve time. These are single diagnostic timings, not repeated performance benchmarks; the accuracy and iteration comparison is the principal control.

A smaller direction in this two-vector span offers little

A supplementary algebraic experiment solves

minimize ||s||,  s in span(p_loose,p_tight),
subject to ||F+Js|| <= 0.03 ||F||.

It uses an orthonormal direction basis, QR/SVD of its two-column action, a scalar secular solve and a convex KKT check. Complete saved-vector arithmetic independently checks the candidate and its action; no new FEM operation is performed. The two-dimensional reduced action condition is 3.3594 and the KKT relative residual is 1.97e-14, so the small result is resolved within this algebraic problem.

Direction	Norm
Original loose direction	0.0794897230590
Original tight direction	0.0796012421522
Best feasible scalar scaling of tight	0.0772247555310
Minimum norm in their span	0.0772243033891

The additional reduction beyond ordinary scalar damping is only 0.00058549%. The candidate's defect using the complete saved actions is 0.0299999999970. That tiny inward margin is not a certificate that a fresh FEM application would pass exactly at the boundary. No candidate trial is evaluated, and no nonlinear improvement is claimed. This deprioritizes selection in this particular two-vector span; it does not rule out a richer Krylov subspace or a trust-region method with an explicit non-Newton acceptance contract.

The clean algebraic evidence is minimum_norm_pair_v4/assessment.json, with input/output hashes and reproduction instructions. The preliminary v1 preflight correctly refused the tight case's unflushed layout. A v2 run produced unexpected NumPy floating-status warnings in two-column BLAS products despite finite results; v3 replaces those products with explicit two-column arithmetic and enables floating exceptions. The clean rerun gives the same result. analyze_nonlinear_direction_pair.py preserves that reproduction logic in this branch at db93bf7; v4 reruns that version with additional RHS-hash and immutable-directory checks.

minimum_norm_pair_v4_binding_review.json passes 61 additional checks binding this assessment and its reproduced vectors to the final audited producers. That closes the assessment's pending provenance qualification; its explicit lack of a fresh FEM action or nonlinear trial remains.

Initial-state control

The initial n = 1.5, J = 0 model has base residual 2.7365921762e-4. Its cold solve passes in 26 iterations and 15.0205 s, with verified relative defect 0.0281993. The raw full step prepares and reduces the residual to 1.7298999698e-4. Gravity correction reduces it to 1.61484154945e-5: a 94.10% decrease, reproducing to roundoff the recorded residual norm of the preceding production campaign's first accepted step. The late model's best sampled corrected decrease is only 0.06059%, at alpha 1/1024.

Both normalization-factor vectors are byte-identical across these states. The following comparison therefore uses the same normalized coordinates, although it samples different states and directions:

Quantity	Initial model	Late model, eta = 0.03
		F
		p
		p
Shape/fields action cancellation factor	about 332.1	about 110,743
Corrected		R
Corrected		R

At the common alpha 1/64 both full corrected remainders are resolved above the heuristic floor. The late direction is 17.35x larger despite a 19.59x smaller residual; ||p||/||F|| increases about 340x. By contrast, sampled curvature per unit direction increases only 1.41x. The relative quadratic remainder coefficient ||R||/(alpha^2 ||F||) grows approximately 8,313x. This describes the observed amplification; it does not estimate a full Hessian norm or identify a physical nullspace.

For these two sampled corrected paths at the common alpha, the main change is not orders-of-magnitude growth in remainder per squared unit-direction step. This does not isolate intrinsic raw-operator or shape-Hessian curvature. A much larger, strongly cancelling Newton direction is being asked to eliminate a much smaller residual. Small second-order errors relative to the individual shape/field responses become large relative to the desired combined response. All 48 seed paths prepare, all 24 auxiliary gravity solves pass, and restoration differences are zero. Corrected full vector remainders show resolved order approximately two from alpha 1 through 1/256. Smaller corrected seed remainders are excluded from order claims by the heuristic screen including the gravity arithmetic allowance.

Implications and next intervention

The measured evidence favors a bounded, fully coupled second-order trial correction as the next isolated intervention. Keep the verified Newton direction and capture the entire affine target L = F0+alpha J0p. For a prepared trial y, solve J(y)c = -(F(y)-L), prepare y+c, and evaluate the complete unchanged residual. This targets the closure, hydrostatic, surface and mass terms that a gravity-only correction leaves or introduces.

For a locally bounded inverse and a smooth residual, c is second order in alpha and preserves the original tangent. The observed weak response makes that boundedness assumption uncertain here. The experiment should therefore limit correction norm and work, try at most one correction at a selected prepared trial, and keep the uncorrected candidate available as a fallback. Recheck geometry and the selected enthalpy-admissibility policy after the correction, since it can change both shape and enthalpy. Keep that policy identical for the corrected candidate, fallback and control: this campaign uses sampled enthalpy without polynomial repair. Adding a polynomial certificate requires a separate, matched admissibility control. This does not bypass the pressure preparation failures at large alpha; it tests whether the prepared range can deliver substantially more useful progress.

Architecturally, implement this as a distinct, defaulted compile-time trial path policy composed by PreparedNewtonStrategy, not as another gravity specialization. Capture F0 and J0p in CaptureDirection before trial mutation; compose correction in PrepareTrial; explicitly refresh the full inverse after changing the prepared state. The ordinary full-residual acceptance test and accepted-state commit remain in stellar_context.cppm. The public physics model and existing equations/normalization need no new arguments.

Independently verify the auxiliary full defect ||J(y)c+F(y)-L|| against its own declared solve tolerance. Preserve and report the base Newton direction's original full-J gate separately; an auxiliary success must not replace it.

Compare full cost per useful accepted step, including extra solves, preparations and rejected attempts. Removing a correction residual at high cost is not itself a solver improvement. Use the same late snapshot, inverse, forcing tolerance and virial/field diagnostics as the control; a subsequent short nonlinear run is necessary before claiming convergence improvement.

Tenfold tighter forcing and minimum-norm selection in the two measured directions are lower priorities after their negative controls. Broader Krylov-space regularization remains untested. A trust step that intentionally misses the current Newton gate would need an explicit distinct contract; solving a shifted system accurately cannot be reported as passing the original full-J gate.

Continuation in n and then angular momentum remains a separate path for testing basin dependence, not a demonstrated cure for this local weak response. A production continuation driver needs a typed checkpoint/state transfer initializer: the current ordinary initializer requires a radial seed, whereas the experimental physical snapshot loader is not a general rotating-model restart API. Record the EOS K(n) path, mass, angular momentum, mesh and normalization. Nothing here establishes a need for an extra n = 1.5 central-density constraint, a translation gauge, or pseudo-arclength.

No new equilibrium has been accepted in this investigation. The input late state still has the previously measured negative sampled density and undefined density-pressure virials, even though its enthalpy-pressure virials pass 1e-5. Both internal admissibility and the virial milestone remain required. These n = 1.5, J = 0 controls make no new claim about higher n, rotating convergence, or any percentage of critical rotation.

Reproduction and artifact scope

The campaign manifests are nonlinear_step_late_case_2026-09-18.json and nonlinear_step_controls_2026-09-18.json. Each records every probe, linear, and gravity-solve option. Run them serially through run_plateau_campaign.py with the configured Release build, the original sandbox.smesh, and a new output directory. The existing evidence root is the task workspace's diagnostics/nonlinear_step_2026-09-18/ directory. Its campaign directories are immutable; reviews, analysis and figures are kept alongside them.

late_v1 was built at ba057b3; controls_v1 at b8d1c6a, which only adds the independent checker and tests. Both copied executables have SHA-256 5f5c4ed0dae351097ec1f632fa4132e490b35c6b8ec3aab689c5a6535b77c634. The mesh SHA-256 is ce11ae99e21e6c3bbfbee402acd2e191c1da0d8261d2227b4203f73f2f337a74. The recorded physical snapshot, frozen normalized state, residual, factors, and layout provide an exact binding for the paired tolerance control.

The checker reads every saved trial state/residual and recomputes the vector remainders, signed projections, mixed interactions and norm identities. It also checks the complete path grid, failed-preparation semantics, budgets, hashes, input binding, gravity affine-target gates, counters, frozen-state restoration and accepted-buffer invariants. A failed linear gate remains a valid finite directional Taylor experiment, but cannot support a claimed Newton-forcing comparison. Its fourteen synthetic tests include intentional vector corruption, a hidden first-order error, noninteger superlinear order, unresolved arithmetic-scale differences, missing hashes/invariants and failed preparation/inner/prepared-gravity gates.

Example independent review commands, replacing E with the evidence-directory path and keeping reviews outside the immutable campaigns:

python3 experiments/check_nonlinear_step.py E/late_v1 --output E/late_v1_review.json
python3 experiments/check_nonlinear_step.py E/controls_v1 --output E/controls_v1_review.json
python3 -m unittest discover -s experiments -p test_nonlinear_step_checker.py -v

The final comparison is saved as figures_v1/nonlinear_step_comparison.png, .pdf and .json. Its JSON records source reviews/hashes and numerical comparisons. Reproduce it with plot_nonlinear_step.py --audited-campaign E/late_v1 E/late_v1_review.json --audited-campaign E/controls_v1 E/controls_v1_review.json --output E/figures_new, using a Python installation with NumPy and Matplotlib. This run used the existing /opt/homebrew/anaconda3/envs/general/bin/python; no dependencies were installed. The final PNG was visually inspected.

Final numerical audits comprise 162 + 349 provenance checks and 6,238 + 6,238 + 7,912 case checks, totaling 20,899. These are arithmetic/provenance checks, not independent physical validation tests. The supplementary two-direction assessment has 61 additional binding checks. All three probes restore the original residual and actions exactly, preserve accepted buffers and factors bitwise, and perform no accepted-state commits.

Input-snapshot equality is additionally recorded in late_v1_input_binding.json: all 178,073 supplied physical coefficients match exactly. The generic checker verifies the supplied file's provenance hash and physical/normalized factors; that separate coefficient check binds the actual imported values themselves. Reported operator norms use the configured diagonal normalization, not exact physical L2 norms. The subtraction/gravity allowance is a conservative heuristic resolution screen, not a certified bound on FEM assembly error.