MeanField/experiments/PRECONDITIONER_IMPROVEMENT_2026-09-18.md

# Preconditioner improvement: frozen systems and nonlinear checks

This experiment changes preconditioning only. The physical equations,
complete Newton Jacobian construction, normalization rules, mesh, and linear
acceptance gate are unchanged. Within each frozen comparison, the numerical
Jacobian, right-hand side, and normalization factors are held fixed; nonlinear
runs rebuild them along their respective trajectories. The date in branch
and artifact names is an experiment identifier; provenance files contain the
actual UTC execution times.

The strongest equal-accuracy timing result is a 1.39548x frozen linear solve
speedup for n = 1.5, J = 0. The combined material/gravity inverse also passes
a saved rotating system that the default cannot solve within 600 iterations.
Neither result establishes convergence of the hard nonlinear models. The
easy n = 1, J = 0 physical regression retains all virial errors below 3.74e-7,
but takes approximately 4% longer with the stronger inverse.

## Reproduction and scope

Integration branch: `research/preconditioner-benchmarks-20260918`, based on
`10adf7e`. Hypotheses are retained separately on
`research/gravity-consistent-schur-20260918`,
`research/preconditioner-cached-cross-actions-20260918`, and
`research/coupled-shape-coarse-20260918`.

The experiment workspace is
`/Users/tboudreaux/.codex/.chatgpt-projects/g-p-6a6b3f2194ec8191bebcd59075897f1d`.
Evidence is stored under `diagnostics/preconditioner_improvement_2026-09-18`
in that workspace. Each campaign retains its clean source commit, build
configuration, copied executable and hash, mesh hash, exact commands,
input/output hashes, and logs. Runs are serial, without overlapping builds or
FEM jobs during timing.

Frozen comparisons start FGMRES from zero on the same saved state and
right-hand side. Each role constructs a fresh inverse, including its own
specification border responses. The restart length is 160, the initial cap is
200 iterations, and the required relative full-J true defect is 0.03. The
independently applied full Jacobian decides acceptance; the Krylov recurrence
flag alone does not. All accepted/trial state buffers, normalization factors,
preparation generation, and a deterministic full-J probe must remain
unchanged.

Costs include inverse construction, normalized binding, Krylov setup, Krylov
solve, and full-J verification. Extra diagnostic applications and artifact I/O
are recorded separately. A speedup to convergence is reported only if both
methods meet the same true gate. Less work spent on a failed solve is not a
time-to-convergence improvement. A frozen linear pass is not nonlinear or
physical convergence.

## Existing two-way coupling: negative controls

`screen_v1` uses source `018fc1c`, before the cached cross-action
optimization.

| Saved state | Inverse | Iterations | Full-J relative defect | Complete verified cost (s) |
|---|---|---:|---:|---:|
| n = 1.5, J = 0 plateau | independent | 200 | 0.0590170471860 | 87.9302 |
| same | stellar LDU | 200 | 0.0600668598825 | 599.5473 |
| n = 1, J = 0.1 plateau | independent | 200 | 0.0826534298794 | 86.9409 |
| same | stellar LDU | 200 | 0.0720550396371 | 594.2053 |

All four solves fail the 0.03 gate. Stellar LDU incurs 6.82–6.83 times the
total cost; it slightly worsens the n = 1.5 result and reduces the rotating
defect by 12.82%. Mean inverse application time rises from approximately 0.19
s to 2.7 s, while full-J actions remain approximately 0.23 s. The uncached
gravity-to-material force column explains the application-cost anomaly.

Poisson contributes 95.53% of the squared n = 1.5 baseline defect and 98.28%
with stellar LDU. For the rotating state those fractions are 85.00% and
98.10%; LDU reduces the gradient-row norm by 94.16% but the Poisson norm by
only 6.35%. These are global normalized block diagnostics, not proof of an
exclusively exterior error. No exterior weighting is changed.

The independent artifact audit passes 65 provenance checks and 93 checks per
case. Both saved states and the normalization data exactly match their
producer artifacts. The two independent baselines reproduce the earlier failed
solves.

## Candidate mechanisms

1. **Prepared cross action.** Reuse the complete prepared gravity force with
   zero density and displacement directions to apply its gravity-gradient
   column. This is the same mathematical coupling with cached geometry; it does
   not strengthen the inverse. See `PRECONDITIONER_CACHED_CROSS_ACTION_2026-09-18.md`.
2. **Consistent gravity Schur refinement.** Keep the existing AMG action, but
   apply a fixed number of corrections against `B R B^T`, where `R` is the same
   mass inverse used in flux recovery. Fixed work preserves linear stationarity
   for stationary child inverses. Damping does not guarantee contraction; it
   must earn its additional cost. See `GRAVITY_CONSISTENT_SCHUR_2026-09-18.md`.
3. **Experimental coupled shape coarse inverse.** Use nine fixed geometric
   modes, complete their material/gravity responses, and construct a checked
   low-rank correction to the normalized inverse. Modes do not depend on the
   Newton right-hand side or previous Krylov results. A rank/overlap guard
   explicitly falls back to the underlying inverse when the correction cannot
   be trusted. Coarse setup and storage are counted, and actual FEM subspace
   identities are independently checked before solving.

## First candidate screen

`candidates_v1` uses source `dc8d4b0` and the same n = 1.5, J = 0 plateau
state. Every fresh independent control reproduces relative defect
0.059017047186 at 200 iterations. Complete control costs remain between 87.53
and 88.25 s.

| Candidate | Iterations | Full-J relative defect | Gate | Complete verified cost (s) |
|---|---:|---:|---|---:|
| Cached stellar LDU | 200 | 0.0600668598779 | fail | 117.4845 |
| Consistent gravity Schur, k = 1, omega = 0.5 | 132 | 0.0292091549685 | pass | 72.3981 |
| Cached stellar LDU + consistent gravity Schur | 121 | 0.0285619160766 | pass | 84.7104 |
| Nine-mode coupled shape coarse correction | 200 | 0.0668657989082 | fail | 97.0564 |

At the same iteration cap, the cached LDU reduces total execution time by a
factor of 5.1032 relative to the old path, while mean inverse application time
drops by a factor of 8.0425. This is an execution improvement, not a
convergence improvement. Saved inputs and the full-J probe agree exactly; the
final normalized correction changes by only 2.7051e-11 relatively. Both LDU
solves fail, and the cached version remains 34.2% more expensive than the
independent control on this state.

Gravity refinement is the first candidate to cross the original gate. It
lowers the Poisson-row defect by 55.4% and the gradient-row defect by 15.0%.
The squared Poisson share drops from 95.53% to 77.59%; surface and hydrostatic
defects grow, so the improvement is judged by the verified global norm. Work
changes from 400 to 396 mass-inverse applications, 200 to 264 AMG Schur
applications, and 200 to 132 outer inverse applications. At this iteration
cap, the default does not pass the same gate, so dividing its failed 88 s
attempt by 72.4 s is not a convergence speedup. The combined LDU saves another
eleven iterations relative to gravity refinement alone but takes 17% more time
overall.

The coarse candidate has all nine modes active, minimum overlap lower bound
0.79465, and no fallback. Its extra construction costs 8.6012 s, including
mode completion and eighteen setup Jacobian applications; it stores 28,491,680
bytes in persistent vectors. The actual FEM subspace check passes.
Nevertheless, its 200-iteration defect worsens by 13.3%. This is a negative
numerical result for these modes on this state, not evidence about every
possible coarse space or rapidly rotating model. The prototype remains
confined to experiments.

## Validation status

The integrated library and experiment driver compile with the repository's
Release/ThinLTO configuration. Source `dc8d4b0` passes 20 focused serial test
cases (441,978 assertions), including FEM equivalence, geometry/discretization
refresh, fixed-work algebra, and coarse inverse tests. Distributed unit and
coarse tests pass on two ranks (13 cases and 213 assertions per rank).

The broad real-FEM MPI run aborts because the existing METIS partition gives
one rank no compactified vacuum element, which the mapped mass constructor
requires. The failing log is retained; this campaign does not claim
distributed FEM Schur validation. See the gravity refinement note for the
exact scope and evidence.

The evidence checker passes 22 lightweight tests. Independent reviews of the
cached LDU and gravity-only cases validate provenance, state/operator
invariants, true gates, and child-work accounting. Reviews are stored outside
immutable campaign directories.

Source `fcb7833` adds owning experiment-only Newton recipes through the
existing ADL preparation interface. They use the same concrete prepared
inverses as the frozen benchmark and the existing context refresh/border
lifecycle. The driver supports default, gravity-Schur, stellar-LDU, and
combined stellar-LDU/gravity-Schur recipes for ordinary Newton, positivity,
curvature, and positivity-curvature runs. The combined Newton recipe was
added at `7b4f674`; the driver builds successfully at `5298d88`. This adds no
public physics-facing solver API.

## Equal-accuracy confirmation

`confirmation_v1` uses source `87569c0`, a restart length of 160, and a
maximum of 600 iterations for both roles. The target remains a freshly
verified relative defect of 0.03.

| State | Inverse | Iterations | Full-J relative defect | Gate | Complete verified cost (s) |
|---|---|---:|---:|---|---:|
| n = 1.5, J = 0 plateau | independent | 231 | 0.0290033752814 | pass | 100.9943 |
| same | gravity Schur k = 1, omega = 0.5 | 132 | 0.0292091549685 | pass | 72.3727 |
| n = 1, J = 0.1 plateau | independent | 600 | 0.0739197073237 | fail | 257.9399 |
| same | gravity Schur k = 1, omega = 0.5 | 600 | 0.0408505998829 | fail | 321.5918 |

For n = 1.5, the frozen linear solve speedup to the same requested accuracy is
**1.39548x**, with 28.34% less complete cost and 42.86% fewer iterations. The
candidate repeats its earlier 132 iterations, with timing close to the earlier
72.3981 s. Both endpoints finish slightly below the same stopping threshold;
neither tolerance nor the full-J check was relaxed.

Rotation behaves differently. Refinement reduces the 600-step defect by
44.74%, but costs 24.68% more and still fails the gate. The remaining gradient
and Poisson defects contribute 0.02664 and 0.03018 to the global relative
norm. The older rotating LDU control reduced the gradient contribution to
0.001813 while leaving a large Poisson contribution. This supports one
complementary LDU+Schur follow-up; it neither proves a gradient floor nor
permits multiplying the separate gains.

All 73 provenance checks and 186 case checks pass. The reproducible figure and
plotting provenance are in `confirmation_v1_figures`; reported Krylov
histories are shown separately from independently verified final points.
Plotting used an existing compatible Python environment and installed no
dependencies.

## Easy nonlinear and physical regression

The first two `newton_v1` cases use source `bed1893`, the same n = 1, J = 0
model with no central-density constraint, positivity strategy, and identical
tolerances. Both converge in six accepted full steps, with no repairs or
rejections. Initial physical diagnostics and layout are byte-identical; all
linear true gates pass.

| Measure | Independent | Gravity Schur k = 1, omega = 0.5 |
|---|---:|---:|
| Total linear iterations | 307 | 258 |
| Summed Newton iteration time (s) | 191.068 | 199.375 |
| Summed linear solve time (s) | 125.035 | 132.028 |
| Complete diagnostic-run wall time (s) | 251.720 | 261.285 |
| Final nonlinear residual | 1.65157e-13 | 1.05839e-13 |

Both final residuals are below the shared 1.81187e-12 nonlinear target. Fewer
iterations do not produce an easy-case speedup: Newton time rises by 4.35% and
full run time by 3.80%. The full run includes physical validation and artifact
I/O; it is not presented as pure solver cost.

All eight virial errors (four definitions at quadrature orders 14 and 18) lie
between 2.61811e-7 and 3.73092e-7 for both models, comfortably below 1e-5. The
maximum endpoint coefficient difference is 2.679e-11; the radius difference is
3.48e-14. The independent paired audit also checks mass, sampled
density/enthalpy admissibility and local EOS metrics. At quadrature order 18,
the density EOS mismatch, scaled by the sampled maximum density, has RMS
8.5654e-5 and maximum 4.2355e-4. The small virial error therefore does not
establish pointwise accuracy to 1e-5 or spatial mesh convergence. This
preserves the existing physical regression; it does not add independent
ESTER validation.

## Higher-index nonlinear follow-through

The n = 1.5, J = 0 curvature case uses the same initial physical state, layout,
mesh, normalization, and stopping settings as the historical independent
preconditioner run. Gravity refinement completes all twelve permitted Newton
steps, with all twelve independently checked linear defects below 0.03. The
historical run stopped after six accepted steps because its next linear solve
failed. The new run therefore removes that immediate linear obstruction.

It does not establish nonlinear convergence. The final residual is
1.3966991e-5 against a target of 2.7365922e-12, only 6.13% below the historical
endpoint. After the first full step, accepted step lengths are mostly 1/16 or
1/32; the final two are 1/64 and 1/512. All nineteen rejected trials fail the
residual-decrease test. All 31 auxiliary gravity solves pass. This evidence
points to the remaining nonlinear step obstruction, rather than another
failed outer or auxiliary gravity solve.

The run spends 415.038 s in outer linear solves and 590.003 s in preparation,
including 178.937 s in auxiliary gravity solves. Summed Newton time is
1075.368 s and complete diagnostic-run wall time is 1138.002 s. These are
costs of an unconverged trajectory, not a nonlinear speedup.

The enthalpy-pressure energy and force virials pass 1e-5, with quadrature-18
errors 4.06909e-6 and 2.57118e-6. Density-pressure virials remain undefined:
the sampled density minimum is -8.14767e-4, with 9600 negative samples and
negative mass fraction 2.89597e-5. Negative density also occurred in the
historical case. Sampled enthalpy stays positive, but this curvature-only run
does not provide a global polynomial positivity certificate. Thus the virial
numbers alone do not make this an acceptable physical model. The independent
audit records these failures separately from successful linear solves and
artifact verification.

## Rotating nonlinear follow-through

The n = 1, J = 0.1 positivity-curvature run accepts one full trial step after
a positivity repair modifies its search direction. Its first raw linear
solve passes in thirteen iterations with relative true defect 0.023583;
that defect does not certify the subsequently repaired direction. The second
reaches 200 iterations with defect 0.0315391 and fails the unchanged 0.03 gate,
before a second trial can be tested. There are no line-search rejections. This case
therefore retains a linear obstruction as well as an unconverged nonlinear
state.

The final nonlinear residual is 0.00896796 against a target of 8.58812e-10.
Summed Newton time is 139.198 s, including 113.338 s in outer linear solves
and 18.544 s in preparation; full diagnostic-run wall time is 201.245 s.
Enthalpy-pressure virials are approximately 0.0108 and fail the physical
milestone. Density-pressure virials are undefined, with sampled density
minimum -0.00692627 and 2015 negative samples at quadrature order 18. The
enthalpy positivity policy passes its checks, but does not establish density
admissibility.

The historical rotating run starts from matching physical diagnostics and
uses matching physics and linear controls, but permits eight Newton steps
instead of twelve. Its endpoint is not used for a controlled timing claim.
All 67 full-campaign provenance checks pass; physical and solver failures are
retained as experimental outcomes, not treated as successful convergence.

## Complementary rotating coupling

`rotation_coupling_v1`, built from `5298d88`, combines cached stellar LDU with
one consistent gravity Schur correction at relaxation 0.5. On the saved
rotating plateau state it passes the unchanged 0.03 gate in 142 iterations,
with relative true defect 0.0299068971 and complete verified cost 96.8236 s.
Its fresh independent control reproduces the original 200-iteration defect
0.0826534298794 and costs 85.8991 s. The earlier independent control still
failed at 600 iterations, so there is no measured finite baseline
time-to-convergence with which to form a speedup ratio.

The independent audit verifies byte-identical saved states, right-hand side,
normalization factors, layout, and deterministic Jacobian probe against the
earlier rotating campaigns. Its fresh baseline correction and exact residual
also match the original 200-iteration control byte for byte. The combination
reduces the Poisson defect norm by 63.33% and the gradient defect norm by
94.47%. Surface and hydrostatic defects increase; acceptance still uses the
complete norm. Mean inverse application cost rises from 0.18409 to 0.43046 s,
so iteration reduction alone must not be presented as an application speedup.

This combination succeeds where either isolated change failed on this frozen
rotating problem. The follow-through in `rotation_newton_v1`, source
`83afd82`, demonstrates the limit of that result. With the historical
eight-step budget and otherwise matching controls, its first raw linear
solve passes in twelve iterations at relative true defect 0.0171651. A
positivity repair changes the search direction; the accepted full trial
reduces the nonlinear residual from 0.0858812 to 0.00685562. At this new state,
the next linear solve reaches 200 iterations with defect 0.0339685 and fails.
There is no second trial and no nonlinear convergence.

Summed Newton time is 172.396 s, including 141.631 s in outer linear solves;
full diagnostic-run wall time is 234.738 s. The quadrature-18 enthalpy-pressure
energy and force virial errors are 0.00712711 and 0.00709672. Both fail 1e-5.
Density-pressure virials are undefined, with 1975 negative density samples
and negative mass fraction 4.04801e-5. The successful frozen linear solve
therefore does not certify a physically acceptable rotating solution, or
even passage through every linear system along a new Newton trajectory.

The historical independent endpoint has a larger nonlinear residual,
0.00840387, but a smaller enthalpy energy virial error, 0.000544539. Both
physical endpoints fail. This is direct evidence that improvement in the
normalized nonlinear residual need not improve virial balance along distinct
trajectories. The combined run passes all 38 current-case artifact checks and
32 historical-control checks, including matching initial diagnostics,
layout, physics and solver settings.

Production defaults remain unchanged: the stronger inverse is an optional
typed policy, and the coarse candidate remains an experiment.

Here J = 0.1 denotes the angular-momentum input, not a measured fraction of
critical rotation. This campaign establishes neither a 50% nor a 99% critical
rotating equilibrium.

## Integration decision and reproducibility

Keep the prepared cross-action optimization: it preserves the mathematical
action and removes the demonstrated cost of rebuilding geometry inside the
two-way stellar inverse. Keep consistent gravity Schur refinement as an
optional typed policy. The measured setting is one sweep at relaxation 0.5;
use it alone for the tested n = 1.5 state and combine it with stellar LDU for
the tested rotating state. These are measured choices for these states, not
universal parameter recommendations. The nine-mode coarse prototype has not
earned promotion, and no production default is changed.

The production policy and cross-action files, with their tests, match their
isolated hypothesis branches. Changes are committed on the integration
branch; the original working checkout and its unrelated quadrature formatting
edit are preserved. No central-density constraint or physics-facing API
requirement was added.

The retained campaign source revisions and tracked recipes are:

| Evidence directory | Source | Recipe in `experiments/` |
|---|---|---|
| `screen_v1` | `018fc1c` | `preconditioner_screen_cases_2026-09-18.json` |
| `candidates_v1` | `dc8d4b0` | `preconditioner_candidates_2026-09-18.json` |
| `confirmation_v1` | `87569c0` | `preconditioner_confirmation_2026-09-18.json` |
| `newton_v1` | `bed1893` | `preconditioner_newton_cases_2026-09-18.json` |
| `rotation_coupling_v1` | `5298d88` | `preconditioner_rotation_coupling_2026-09-18.json` |
| `rotation_newton_v1` | `83afd82` | `preconditioner_rotation_newton_2026-09-18.json` |

Use `run_plateau_campaign.py` with the selected recipe, a build configured
from the recorded `provenance.json`, and a fresh output directory. It requires
a clean committed source tree, checks the build's source directory, copies
the executable before running cases, records hashes, and refuses to overwrite
an earlier campaign. The exact build and execution commands are retained in
each campaign. The mesh is versioned as `sandbox.smesh`; saved-state input
paths and hashes identify the earlier producer artifacts.

For frozen campaigns, `check_preconditioner_benchmark.py` independently checks
the artifacts. Place its `--output` outside the campaign directory to retain
the original evidence unchanged. The Newton audits and their source scripts
are retained alongside the campaigns, with solver and physical failures
reported separately from artifact-integrity checks. Six lightweight audit
tests cover failed solves, raw versus repaired directions, incorrect policy
labels, hash failures, incomplete producers, and unmatched Newton budgets.

The final comparison figure, script, and input hashes are in
`final_comparison_figures/` under the evidence directory. It shows reported
Krylov histories and independently verified endpoints separately, includes
setup and verification cost, and labels the unequal rotating iteration caps
and failed controls explicitly.