MeanField/experiments/NONLINEAR_STEP_INVESTIGATION_2026-09-18.md

# Nonlinear step obstruction after gravity preconditioning

The completed frozen-state investigation supports a consistent directional
Jacobian and a narrow range of useful trial steps at the n = 1.5, J = 0
endpoint. Tightening the full-J forcing tolerance tenfold costs about 70%
more linear work without improving the useful sampled step. Compared with
the seed, the late direction is 17.35x larger despite a 19.59x smaller
residual; strongly cancelling shape and field responses amplify otherwise
moderate corrected-path curvature.

Gravity correction cuts the remainder norm but can increase its adverse
merit projection through the retained equations. A bounded fully coupled
trial correction is the next recommended experiment. This branch implements
diagnostics and records evidence; it makes no production numerical-policy,
physics API, normalization or acceptance-threshold changes.

Three immutable probes cover 144 trial-path observations and 60 successful
auxiliary gravity solves. Their independent final audits pass 20,899 checks;
the new checker passes fourteen synthetic tests. No new model is accepted.

## Recorded decision from the completed preconditioner investigation

The completed report is `PRECONDITIONER_IMPROVEMENT_2026-09-18.md`, committed
at `dc1da62` on `research/preconditioner-benchmarks-20260918`. Its immutable
campaigns live under the task workspace's
`diagnostics/preconditioner_improvement_2026-09-18/` directory.

- Consistent gravity Schur refinement solves the saved n = 1.5, J = 0 linear
  system in 132 iterations and 72.3727 s, versus 231 iterations and 100.9943 s
  for the independent control. Both satisfy the same independently verified
  relative full-J defect of 0.03: a measured 1.39548x speedup.
- Combining this refinement with stellar LDU solves the saved rotating
  n = 1, J = 0.1 system in 142 iterations and 96.8236 s. The independent and
  gravity-only controls still fail after 600 iterations. Their finite
  time-to-convergence is unknown, so no rotating speedup ratio is reported.
- Prepared geometry reuse makes the existing stellar LDU application 8.0425x
  cheaper without improving its convergence by itself. Retain this
  implementation optimization independently of policy selection.
- The tested nine-mode shape correction does not improve the n = 1.5 case.
  It remains an experimental negative result, not a rejection of all coarse
  spaces.
- The easy n = 1, J = 0 model preserves all eight virials below 3.74e-7,
  while the stronger inverse costs about 4% more. Defaults remain unchanged.
- Neither hard nonlinear model is physically acceptable. Negative sampled
  density makes density-pressure virials undefined. In rotation, a smaller
  nonlinear residual accompanied a larger enthalpy-pressure virial error.
  Residual decrease is not a substitute for virial and field-admissibility
  validation. No angular momentum has been calibrated to critical rotation.

## New investigation and controls

Branch: `research/nonlinear-step-obstruction-20260918`, based on `dc1da62`.
The configured worktree is reused to avoid an unrelated full rebuild; the
preceding branch and copied campaign executables retain their exact source
and binary identities. This branch adds experimental diagnostics only.

Start with the n = 1.5, J = 0 endpoint of
`newton_v1/n1p5_J0_curvature_gravity_schur`. All twelve outer linear solves and
all 31 auxiliary gravity solves passed, but the final nonlinear residual was
1.3966991e-5 against a target of 2.7365922e-12. The final accepted alpha was
1/512. Every trial prepared successfully; all nineteen rejections were
residual-decrease failures, and geometry never limited the step.

That campaign did not retain every raw Newton direction. The new late-state
probe computes a fresh direction at its saved final endpoint; it does not
claim to replay the preceding iteration's rejected trials. A plateau in a
bounded twelve-step run is also not proof that all subsequent steps must
fail. The fresh trial measurements can test that possibility.

The final squared-residual shares are approximately 42.42% hydrostatic,
26.23% closure, 15.87% surface, 10.43% gravity, and 5.05% mass. This motivates
examining retained-equation curvature after the gravity correction, rather
than assuming the earlier gravity or pressure-preparation failure persists.
The last auxiliary prepared defect, approximately 3.33e-14, measures agreement
with the affine gravity target, not satisfaction of the gravity equations.
The final gravity-block residual norm is approximately 4.51e-6.

Keep the complete physical equations, fixed normalization, optional
central-density specification (absent here), mesh, and acceptance thresholds
unchanged. Use one gravity Schur refinement at relaxation 0.5 and the same
restart length 160. Fresh cold solves and independently applied full-J true
defects determine direction quality. Trial probes never commit a model.

## Questions and discriminating evidence

1. **Does the prepared trial path have the expected tangent?** For frozen
   residual F and computed direction p, measure Jp and actual prepared
   differences. The curvature-corrected path should preserve that tangent
   because its gravity correction is second order in alpha. A discrepancy
   that remains first order above arithmetic noise would implicate the
   linearization or preparation path.
2. **Does the inexact linear direction cause the obstruction?** Compare
   directions at the same saved state with forcing tolerances 0.03 and 0.003,
   subject to a recorded finite iteration budget. A tighter requested
   tolerance counts only if its independent true gate passes. Failed solves
   are recorded and do not substantiate this comparison.
3. **Which nonlinear terms oppose the linear prediction?** Split p into its
   actual surface coordinates and all remaining coordinates. Compare raw
   and gravity-corrected trials for full, shape-only and fields-only
   directions. Form mixed remainders by vector subtraction. Signed
   projections against the full linear prediction distinguish adverse
   curvature from large but cancelling terms.
4. **Is the behavior specific to the late state?** Use the same-polytrope
   Lane–Emden seed as an early-state control, where the previous production
   run accepted its first full Newton step. Record preparation and enthalpy
   feasibility separately from the residual metric.

For phi = ||F||^2/2 and ||Jp + F|| <= eta ||F||,

    phi'(0) = <F,Jp> <= -(1-eta) ||F||^2.

At eta = 0.03 the slope must be strongly negative if J differentiates the
prepared path. The production sufficient-decrease test therefore ought to
accept sufficiently small resolved steps. The investigation measures whether
large adverse higher-order terms explain how small those steps become.

Each block's vector remainder R = F_trial - F - alpha Jp must satisfy

    ||F_trial||^2 = ||F + alpha Jp||^2
                   + 2 <F + alpha Jp, R> + ||R||^2.

For the corrected trial map, its mixed remainder includes interactions
introduced by gravity correction; it is not identified solely with the raw
physical shape Hessian. Below-floor differences are unresolved, not evidence
of exact linearity.

The EOS provides an additional interpretation limit: rho(h) is proportional
to max(h,0)^1.5 here. At an exact zero with a positive enthalpy direction,
the closure remainder can scale as alpha^1.5 while its first derivative is
correct. A superlinear slope different from two does not alone establish a
Jacobian defect; the direct tangent check is whether R/alpha decreases in a
resolved regime. The strong zero-pressure boundary row is already expressed
as h-h(P_surface), with derivative delta h, so it does not inherit a vanishing
dP/dh boundary row.

The paired directions also permit a conditioning diagnostic without another
operator solve: w = p_tight - p_loose and Jw = Jp_tight - Jp_loose. A large
change in direction with a small action demonstrates a weak response in that
direction, not a proved nullspace or a particular symmetry. Measure the
shape/fields action cancellation as well. Fixed-K, fixed-M n = 1.5 does not
have the familiar exact n = 3 mass-preserving homology; no additional gauge
or central-density constraint is justified by that analogy.

## Execution and validation

Run a late-state probe first, then use its evidence to choose the matched
tighter-direction and early-state controls. All model jobs and builds run
serially. The campaign runner records clean source commits, exact commands,
input/output and executable hashes, and refuses existing output directories.
The diagnostic saves the frozen vectors, layout and normalization, checks
restoration and accepted-buffer invariance, and reports failed preparations
and inner solves without inventing values for missing trials.

## Late-state results at forcing tolerance 0.03

The first campaign, `late_v1`, completed in 696.63 s using executable source
`ba057b3983ff6006e460a1fa80e3e6523ceeb4a5`. Its independent review
`late_v1_review.json` passes 162 provenance checks and 6,238 saved-vector,
arithmetic, layout, failure-semantics, and restoration checks. The checker
and its fourteen focused synthetic tests are committed at `b8d1c6a`.
All saved base residual and Jacobian-action restoration differences are zero;
accepted physical/normalized/residual buffers and normalization factors remain
bitwise identical. There are no accepted model commits or positivity repairs.

The cold gravity-Schur solve takes 84 iterations and 47.9374 s. A separate full
Jacobian application verifies relative defect 0.0295592 against the requested
0.03 gate. The base norm is 1.3966991044e-5, the direction norm is 0.0794897,
and the merit slope is -1.94906e-10. Thus the computed direction is a strong
local descent direction for this fixed normalized residual.

| Alpha | Raw residual / base | Gravity-corrected residual / base |
| --- | ---: | ---: |
| 1, 1/4, 1/16 | pressure preparation fails | blocked by raw preparation |
| 1/64 | 1.807268 | 1.080323 |
| 1/256 | 1.002510 | 1.002025 |
| 1/1024 | 0.999170 | 0.999394 |
| 1/4096 | 0.999764 | 0.999779 |
| 1/16384 | 0.999940 | 0.999940 |

These are sampled trial residuals, not a production line search or a claim
that one of these alphas is optimal. The three smaller decreasing steps
also satisfy the ordinary sufficient-decrease metric. Full and fields-only
large steps fail with `thermodynamic_domain` at `pressure_force`; shape-only
steps prepare across the grid. Eighteen auxiliary gravity solves pass, and
all 36 prepared raw/corrected paths pass their applicable gates. The absence
of pressure rejections in the preceding trajectory is consistent with these
new observations at a different endpoint and direction.

### The measured tangent is consistent

For both full paths, independently recomputed vector remainders have resolved
order approximately two from alpha 1/64 through 1/16384. Shrinking alpha by
four shrinks ||R||/alpha approximately fourfold. Raw ||R||/alpha^2 stabilizes
near 0.085426; corrected stabilizes near 0.005807. No persistent first-order
directional discrepancy is detected over this range. This verifies the
sampled direction and scales, not every Jacobian column or arbitrary states.

Despite this consistency, corrected ||R||/(alpha ||Jp||) is still 1.63 at
alpha 1/256, 0.406 at 1/1024, and 0.0254 at 1/16384. The first-order residual
response is so small that formally second-order terms can dominate it well
before pressure preparation fails.

### The direction couples large cancelling responses

||J p_shape|| = 0.7730368 and ||J p_fields|| = 0.7730377, while
||J p|| = 1.396089e-5. The cancellation factor
(||J p_shape||+||J p_fields||)/||J p|| is 110,743, and the image cosine is
-0.999999999838. All are measured in the unchanged diagonal normalization.

The full directional gain ||Jp||/||p|| is 1.7563e-4; the shape-only gain
is 15.933. Their ratio supplies a lower bound of approximately 9.07e4 on
the normalized two-norm condition number if this finite-dimensional Jacobian
is nonsingular. It is not an estimate of that condition number, a discovered
nullspace, or evidence identifying a particular physical symmetry. In
particular, it does not justify imposing an additional central-density
constraint on this n = 1.5 model.

### Smaller remainder does not guarantee better merit

At alpha 1/1024, gravity correction reduces ||R|| by about 14.7x, but raises
the adverse projection <F+alpha Jp,R> from 2.49849e-14 to 7.20763e-14, a
factor of 2.88. This explains why its residual is slightly worse than the
raw residual at this small step, despite its much smaller remainder.

The corrected adverse projection splits approximately as follows:

| Retained residual block | Share of signed total projection |
| --- | ---: |
| Hydrostatic balance | 43.52% |
| Barotropic closure | 32.33% |
| Surface balance | 19.53% |
| Total mass | 4.63% |

Gravity contributes only arithmetic-scale terms. The raw hydrostatic
projection is small and favorable; correction makes it the largest adverse
contribution. The same signed full-predictor projection decomposes into
1.18% corrected shape-direction remainder, 47.75% corrected fields-direction
remainder, and 51.07% mixed remainder. Even the corrected shape-direction
trial changes gravity-field values. These terms sum by vector identity;
the mixed term includes interactions introduced by the correction. They
must not be identified with the pure shape/field Hessian of the raw operator.

## Matched direction-accuracy control

The tighter late-state solve passes its requested 0.003 true-defect gate:
143 iterations, 81.6901 s, verified relative defect 0.00294840. The physical
base, normalized base, F, RHS, both normalization-factor files and layout match
the looser experiment byte-for-byte. Its individual audit passes 146 producer
and hash checks and 6,238 vector/invariant checks, with zero restoration drift
and bitwise unchanged accepted buffers/factors. All eighteen auxiliary solves
pass. The final controls campaign audit also passes: 349 provenance checks,
6,238 tight-case vector/invariant checks and 7,912 seed-case checks.

| Quantity | Eta = 0.03 | Eta = 0.003 |
| --- | ---: | ---: |
| Verified relative full-J defect | 0.0295592 | 0.00294840 |
| Cold linear iterations | 84 | 143 |
| Cold linear seconds | 47.9374 | 81.6901 |
| Normalized direction norm | 0.0794897 | 0.0796012 |
| Corrected residual ratio at alpha 1/64 | 1.080323 | 1.080567 |
| Corrected residual ratio at alpha 1/256 | 1.002025 | 1.002037 |
| Corrected residual ratio at alpha 1/1024 | 0.999394139 | 0.999394272 |

For w = p_tight-p_loose, ||w|| = 7.06262e-4, or 0.8885% of ||p_loose||.
The direction cosine is 0.99996157. ||Jw|| = 4.10794e-7 and
||Jw||/||w|| = 5.81645e-4. Thus improving linear accuracy tenfold has not
substantially changed the large cancelling direction or its useful sampled
step range. Both full paths again have resolved approximately quadratic
remainders; the corrected remainder actually grows about 0.272% at alpha
1/1024. The tighter solve costs about 70% more iterations and diagnostic
solve time. These are single diagnostic timings, not repeated performance
benchmarks; the accuracy and iteration comparison is the principal control.

### A smaller direction in this two-vector span offers little

A supplementary algebraic experiment solves

    minimize ||s||,  s in span(p_loose,p_tight),
    subject to ||F+Js|| <= 0.03 ||F||.

It uses an orthonormal direction basis, QR/SVD of its two-column action, a
scalar secular solve and a convex KKT check. Complete saved-vector arithmetic
independently checks the candidate and its action; no new FEM operation is
performed. The two-dimensional reduced action condition is 3.3594 and the
KKT relative residual is 1.97e-14, so the small result is resolved within this
algebraic problem.

| Direction | Norm |
| --- | ---: |
| Original loose direction | 0.0794897230590 |
| Original tight direction | 0.0796012421522 |
| Best feasible scalar scaling of tight | 0.0772247555310 |
| Minimum norm in their span | 0.0772243033891 |

The additional reduction beyond ordinary scalar damping is only 0.00058549%.
The candidate's defect using the complete saved actions is 0.0299999999970.
That tiny inward margin is not a certificate that a fresh FEM application
would pass exactly at the boundary. No candidate trial is evaluated, and no
nonlinear improvement is claimed. This deprioritizes selection in this
particular two-vector span; it does not rule out a richer Krylov subspace or
a trust-region method with an explicit non-Newton acceptance contract.

The clean algebraic evidence is `minimum_norm_pair_v4/assessment.json`, with
input/output hashes and reproduction instructions. The preliminary v1
preflight correctly refused the tight case's unflushed layout. A v2 run
produced unexpected NumPy floating-status warnings in two-column BLAS
products despite finite results; v3 replaces those products with explicit
two-column arithmetic and enables floating exceptions. The clean rerun gives
the same result. `analyze_nonlinear_direction_pair.py` preserves that
reproduction logic in this branch at `db93bf7`; v4 reruns that version with
additional RHS-hash and immutable-directory checks.

`minimum_norm_pair_v4_binding_review.json` passes 61 additional checks
binding this assessment and its reproduced vectors to the final audited
producers. That closes the assessment's pending provenance qualification;
its explicit lack of a fresh FEM action or nonlinear trial remains.

## Initial-state control

The initial n = 1.5, J = 0 model has base residual 2.7365921762e-4.
Its cold solve passes in 26 iterations and 15.0205 s, with verified relative
defect 0.0281993. The raw full step prepares and reduces the residual to
1.7298999698e-4. Gravity correction reduces it to 1.61484154945e-5: a 94.10%
decrease, reproducing to roundoff the recorded residual norm of the preceding
production campaign's first accepted step. The late model's best sampled
corrected decrease is only 0.06059%, at alpha 1/1024.

Both normalization-factor vectors are byte-identical across these states.
The following comparison therefore uses the same normalized coordinates,
although it samples different states and directions:

| Quantity | Initial model | Late model, eta = 0.03 |
| --- | ---: | ---: |
| ||F|| | 2.736592e-4 | 1.396699e-5 |
| ||p|| | 0.00458195 | 0.0794897 |
| ||p||/||F|| | 16.7433 | 5691.26 |
| Shape/fields action cancellation factor | about 332.1 | about 110,743 |
| Corrected ||R||/(alpha^2 ||p||^2), alpha = 1/64 | 0.656881 | 0.926040 |
| Corrected ||R||/(alpha^2 ||F||), alpha = 1/64 | 0.0503939 | 418.937 |

At the common alpha 1/64 both full corrected remainders are resolved above
the heuristic floor. The late direction is 17.35x larger despite a 19.59x
smaller residual; ||p||/||F|| increases about 340x. By contrast, sampled
curvature per unit direction increases only 1.41x. The relative quadratic
remainder coefficient ||R||/(alpha^2 ||F||) grows approximately 8,313x.
This describes the observed amplification; it does not estimate a full
Hessian norm or identify a physical nullspace.

For these two sampled corrected paths at the common alpha, the main change
is not orders-of-magnitude growth in remainder per squared unit-direction
step. This does not isolate intrinsic raw-operator or shape-Hessian curvature.
A much larger, strongly cancelling Newton direction is
being asked to eliminate a much smaller residual. Small second-order errors
relative to the individual shape/field responses become large relative to
the desired combined response. All 48 seed paths prepare, all 24 auxiliary
gravity solves pass, and restoration differences are zero. Corrected full
vector remainders show resolved order approximately two from alpha 1 through
1/256. Smaller corrected seed remainders are excluded from order claims by
the heuristic screen including the gravity arithmetic allowance.

## Implications and next intervention

The measured evidence favors a bounded, fully coupled second-order trial
correction as the next isolated intervention. Keep the verified Newton
direction and capture the entire affine target L = F0+alpha J0p. For a
prepared trial y, solve J(y)c = -(F(y)-L), prepare y+c, and evaluate the
complete unchanged residual. This targets the closure, hydrostatic, surface
and mass terms that a gravity-only correction leaves or introduces.

For a locally bounded inverse and a smooth residual, c is second order in
alpha and preserves the original tangent. The observed weak response makes
that boundedness assumption uncertain here. The experiment should therefore
limit correction norm and work, try at most one correction at a selected
prepared trial, and keep the uncorrected candidate available as a fallback.
Recheck geometry and the selected enthalpy-admissibility policy after the
correction, since it can change both shape and enthalpy. Keep that policy
identical for the corrected candidate, fallback and control: this campaign
uses sampled enthalpy without polynomial repair. Adding a polynomial
certificate requires a separate, matched admissibility control. This does
not bypass the pressure preparation failures at large alpha; it tests whether the prepared range can
deliver substantially more useful progress.

Architecturally, implement this as a distinct, defaulted compile-time trial
path policy composed by `PreparedNewtonStrategy`, not as another gravity
specialization. Capture F0 and J0p in `CaptureDirection` before trial mutation;
compose correction in `PrepareTrial`; explicitly refresh the full inverse
after changing the prepared state. The ordinary full-residual acceptance
test and accepted-state commit remain in `stellar_context.cppm`. The public
physics model and existing equations/normalization need no new arguments.

Independently verify the auxiliary full defect ||J(y)c+F(y)-L|| against its
own declared solve tolerance. Preserve and report the base Newton direction's
original full-J gate separately; an auxiliary success must not replace it.

Compare full cost per useful accepted step, including extra solves,
preparations and rejected attempts. Removing a correction residual at high
cost is not itself a solver improvement. Use the same late snapshot, inverse,
forcing tolerance and virial/field diagnostics as the control; a subsequent
short nonlinear run is necessary before claiming convergence improvement.

Tenfold tighter forcing and minimum-norm selection in the two measured
directions are lower priorities after their negative controls. Broader
Krylov-space regularization remains untested. A trust step that intentionally
misses the current Newton gate would need an explicit distinct contract;
solving a shifted system accurately cannot be reported as passing the
original full-J gate.

Continuation in n and then angular momentum remains a separate path for
testing basin dependence, not a demonstrated cure for this local weak
response. A production continuation driver needs a typed checkpoint/state
transfer initializer: the current ordinary initializer requires a radial
seed, whereas the experimental physical snapshot loader is not a general
rotating-model restart API. Record the EOS K(n) path, mass, angular momentum,
mesh and normalization. Nothing here establishes a need for an extra n = 1.5
central-density constraint, a translation gauge, or pseudo-arclength.

No new equilibrium has been accepted in this investigation. The input late
state still has the previously measured negative sampled density and
undefined density-pressure virials, even though its enthalpy-pressure
virials pass 1e-5. Both internal admissibility and the virial milestone remain
required. These n = 1.5, J = 0 controls make no new claim about higher n,
rotating convergence, or any percentage of critical rotation.

## Reproduction and artifact scope

The campaign manifests are `nonlinear_step_late_case_2026-09-18.json` and
`nonlinear_step_controls_2026-09-18.json`. Each records every probe, linear,
and gravity-solve option. Run them serially through `run_plateau_campaign.py`
with the configured Release build, the original `sandbox.smesh`, and a new
output directory. The existing evidence root is the task workspace's
`diagnostics/nonlinear_step_2026-09-18/` directory. Its campaign directories
are immutable; reviews, analysis and figures are kept alongside them.

`late_v1` was built at `ba057b3`; `controls_v1` at `b8d1c6a`, which only adds
the independent checker and tests. Both copied executables have SHA-256
`5f5c4ed0dae351097ec1f632fa4132e490b35c6b8ec3aab689c5a6535b77c634`.
The mesh SHA-256 is
`ce11ae99e21e6c3bbfbee402acd2e191c1da0d8261d2227b4203f73f2f337a74`.
The recorded physical snapshot, frozen normalized state, residual, factors,
and layout provide an exact binding for the paired tolerance control.

The checker reads every saved trial state/residual and recomputes the vector
remainders, signed projections, mixed interactions and norm identities.
It also checks the complete path grid, failed-preparation semantics, budgets,
hashes, input binding, gravity affine-target gates, counters, frozen-state
restoration and accepted-buffer invariants. A failed linear gate remains a
valid finite directional Taylor experiment, but cannot support a claimed
Newton-forcing comparison. Its fourteen synthetic tests include intentional
vector corruption, a hidden first-order error, noninteger superlinear order,
unresolved arithmetic-scale differences, missing hashes/invariants and
failed preparation/inner/prepared-gravity gates.

Example independent review commands, replacing E with the evidence-directory
path and keeping reviews outside the immutable campaigns:

```text
python3 experiments/check_nonlinear_step.py E/late_v1 --output E/late_v1_review.json
python3 experiments/check_nonlinear_step.py E/controls_v1 --output E/controls_v1_review.json
python3 -m unittest discover -s experiments -p test_nonlinear_step_checker.py -v
```

The final comparison is saved as
`figures_v1/nonlinear_step_comparison.png`, `.pdf` and `.json`. Its JSON
records source reviews/hashes and numerical comparisons. Reproduce it with
`plot_nonlinear_step.py --audited-campaign E/late_v1 E/late_v1_review.json
--audited-campaign E/controls_v1 E/controls_v1_review.json --output E/figures_new`,
using a Python installation with NumPy and Matplotlib. This run used the
existing `/opt/homebrew/anaconda3/envs/general/bin/python`; no dependencies
were installed. The final PNG was visually inspected.

Final numerical audits comprise 162 + 349 provenance checks and 6,238 +
6,238 + 7,912 case checks, totaling 20,899. These are arithmetic/provenance
checks, not independent physical validation tests. The supplementary
two-direction assessment has 61 additional binding checks. All three probes
restore the original residual and actions exactly, preserve accepted buffers
and factors bitwise, and perform no accepted-state commits.

Input-snapshot equality is additionally recorded in `late_v1_input_binding.json`:
all 178,073 supplied physical coefficients match exactly. The generic checker
verifies the supplied file's provenance hash and physical/normalized factors;
that separate coefficient check binds the actual imported values themselves.
Reported operator norms use the configured diagonal normalization, not exact
physical L2 norms. The subtraction/gravity allowance is a conservative
heuristic resolution screen, not a certified bound on FEM assembly error.