diff --git a/experiments/NONLINEAR_STEP_INVESTIGATION_2026-09-18.md b/experiments/NONLINEAR_STEP_INVESTIGATION_2026-09-18.md index 0988afb..c583fb6 100644 --- a/experiments/NONLINEAR_STEP_INVESTIGATION_2026-09-18.md +++ b/experiments/NONLINEAR_STEP_INVESTIGATION_2026-09-18.md @@ -1,5 +1,23 @@ # Nonlinear step obstruction after gravity preconditioning +The completed frozen-state investigation supports a consistent directional +Jacobian and a narrow range of useful trial steps at the n = 1.5, J = 0 +endpoint. Tightening the full-J forcing tolerance tenfold costs about 70% +more linear work without improving the useful sampled step. Compared with +the seed, the late direction is 17.35x larger despite a 19.59x smaller +residual; strongly cancelling shape and field responses amplify otherwise +moderate corrected-path curvature. + +Gravity correction cuts the remainder norm but can increase its adverse +merit projection through the retained equations. A bounded fully coupled +trial correction is the next recommended experiment. This branch implements +diagnostics and records evidence; it makes no production numerical-policy, +physics API, normalization or acceptance-threshold changes. + +Three immutable probes cover 144 trial-path observations and 60 successful +auxiliary gravity solves. Their independent final audits pass 20,899 checks; +the new checker passes fourteen synthetic tests. No new model is accepted. + ## Recorded decision from the completed preconditioner investigation The completed report is `PRECONDITIONER_IMPROVEMENT_2026-09-18.md`, committed @@ -133,5 +151,334 @@ The diagnostic saves the frozen vectors, layout and normalization, checks restoration and accepted-buffer invariance, and reports failed preparations and inner solves without inventing values for missing trials. -Results are pending. No numerical remedy or new convergence claim follows -from the investigation design alone. +## Late-state results at forcing tolerance 0.03 + +The first campaign, `late_v1`, completed in 696.63 s using executable source +`ba057b3983ff6006e460a1fa80e3e6523ceeb4a5`. Its independent review +`late_v1_review.json` passes 162 provenance checks and 6,238 saved-vector, +arithmetic, layout, failure-semantics, and restoration checks. The checker +and its fourteen focused synthetic tests are committed at `b8d1c6a`. +All saved base residual and Jacobian-action restoration differences are zero; +accepted physical/normalized/residual buffers and normalization factors remain +bitwise identical. There are no accepted model commits or positivity repairs. + +The cold gravity-Schur solve takes 84 iterations and 47.9374 s. A separate full +Jacobian application verifies relative defect 0.0295592 against the requested +0.03 gate. The base norm is 1.3966991044e-5, the direction norm is 0.0794897, +and the merit slope is -1.94906e-10. Thus the computed direction is a strong +local descent direction for this fixed normalized residual. + +| Alpha | Raw residual / base | Gravity-corrected residual / base | +| --- | ---: | ---: | +| 1, 1/4, 1/16 | pressure preparation fails | blocked by raw preparation | +| 1/64 | 1.807268 | 1.080323 | +| 1/256 | 1.002510 | 1.002025 | +| 1/1024 | 0.999170 | 0.999394 | +| 1/4096 | 0.999764 | 0.999779 | +| 1/16384 | 0.999940 | 0.999940 | + +These are sampled trial residuals, not a production line search or a claim +that one of these alphas is optimal. The three smaller decreasing steps +also satisfy the ordinary sufficient-decrease metric. Full and fields-only +large steps fail with `thermodynamic_domain` at `pressure_force`; shape-only +steps prepare across the grid. Eighteen auxiliary gravity solves pass, and +all 36 prepared raw/corrected paths pass their applicable gates. The absence +of pressure rejections in the preceding trajectory is consistent with these +new observations at a different endpoint and direction. + +### The measured tangent is consistent + +For both full paths, independently recomputed vector remainders have resolved +order approximately two from alpha 1/64 through 1/16384. Shrinking alpha by +four shrinks ||R||/alpha approximately fourfold. Raw ||R||/alpha^2 stabilizes +near 0.085426; corrected stabilizes near 0.005807. No persistent first-order +directional discrepancy is detected over this range. This verifies the +sampled direction and scales, not every Jacobian column or arbitrary states. + +Despite this consistency, corrected ||R||/(alpha ||Jp||) is still 1.63 at +alpha 1/256, 0.406 at 1/1024, and 0.0254 at 1/16384. The first-order residual +response is so small that formally second-order terms can dominate it well +before pressure preparation fails. + +### The direction couples large cancelling responses + +||J p_shape|| = 0.7730368 and ||J p_fields|| = 0.7730377, while +||J p|| = 1.396089e-5. The cancellation factor +(||J p_shape||+||J p_fields||)/||J p|| is 110,743, and the image cosine is +-0.999999999838. All are measured in the unchanged diagonal normalization. + +The full directional gain ||Jp||/||p|| is 1.7563e-4; the shape-only gain +is 15.933. Their ratio supplies a lower bound of approximately 9.07e4 on +the normalized two-norm condition number if this finite-dimensional Jacobian +is nonsingular. It is not an estimate of that condition number, a discovered +nullspace, or evidence identifying a particular physical symmetry. In +particular, it does not justify imposing an additional central-density +constraint on this n = 1.5 model. + +### Smaller remainder does not guarantee better merit + +At alpha 1/1024, gravity correction reduces ||R|| by about 14.7x, but raises +the adverse projection from 2.49849e-14 to 7.20763e-14, a +factor of 2.88. This explains why its residual is slightly worse than the +raw residual at this small step, despite its much smaller remainder. + +The corrected adverse projection splits approximately as follows: + +| Retained residual block | Share of signed total projection | +| --- | ---: | +| Hydrostatic balance | 43.52% | +| Barotropic closure | 32.33% | +| Surface balance | 19.53% | +| Total mass | 4.63% | + +Gravity contributes only arithmetic-scale terms. The raw hydrostatic +projection is small and favorable; correction makes it the largest adverse +contribution. The same signed full-predictor projection decomposes into +1.18% corrected shape-direction remainder, 47.75% corrected fields-direction +remainder, and 51.07% mixed remainder. Even the corrected shape-direction +trial changes gravity-field values. These terms sum by vector identity; +the mixed term includes interactions introduced by the correction. They +must not be identified with the pure shape/field Hessian of the raw operator. + +## Matched direction-accuracy control + +The tighter late-state solve passes its requested 0.003 true-defect gate: +143 iterations, 81.6901 s, verified relative defect 0.00294840. The physical +base, normalized base, F, RHS, both normalization-factor files and layout match +the looser experiment byte-for-byte. Its individual audit passes 146 producer +and hash checks and 6,238 vector/invariant checks, with zero restoration drift +and bitwise unchanged accepted buffers/factors. All eighteen auxiliary solves +pass. The final controls campaign audit also passes: 349 provenance checks, +6,238 tight-case vector/invariant checks and 7,912 seed-case checks. + +| Quantity | Eta = 0.03 | Eta = 0.003 | +| --- | ---: | ---: | +| Verified relative full-J defect | 0.0295592 | 0.00294840 | +| Cold linear iterations | 84 | 143 | +| Cold linear seconds | 47.9374 | 81.6901 | +| Normalized direction norm | 0.0794897 | 0.0796012 | +| Corrected residual ratio at alpha 1/64 | 1.080323 | 1.080567 | +| Corrected residual ratio at alpha 1/256 | 1.002025 | 1.002037 | +| Corrected residual ratio at alpha 1/1024 | 0.999394139 | 0.999394272 | + +For w = p_tight-p_loose, ||w|| = 7.06262e-4, or 0.8885% of ||p_loose||. +The direction cosine is 0.99996157. ||Jw|| = 4.10794e-7 and +||Jw||/||w|| = 5.81645e-4. Thus improving linear accuracy tenfold has not +substantially changed the large cancelling direction or its useful sampled +step range. Both full paths again have resolved approximately quadratic +remainders; the corrected remainder actually grows about 0.272% at alpha +1/1024. The tighter solve costs about 70% more iterations and diagnostic +solve time. These are single diagnostic timings, not repeated performance +benchmarks; the accuracy and iteration comparison is the principal control. + +### A smaller direction in this two-vector span offers little + +A supplementary algebraic experiment solves + + minimize ||s||, s in span(p_loose,p_tight), + subject to ||F+Js|| <= 0.03 ||F||. + +It uses an orthonormal direction basis, QR/SVD of its two-column action, a +scalar secular solve and a convex KKT check. Complete saved-vector arithmetic +independently checks the candidate and its action; no new FEM operation is +performed. The two-dimensional reduced action condition is 3.3594 and the +KKT relative residual is 1.97e-14, so the small result is resolved within this +algebraic problem. + +| Direction | Norm | +| --- | ---: | +| Original loose direction | 0.0794897230590 | +| Original tight direction | 0.0796012421522 | +| Best feasible scalar scaling of tight | 0.0772247555310 | +| Minimum norm in their span | 0.0772243033891 | + +The additional reduction beyond ordinary scalar damping is only 0.00058549%. +The candidate's defect using the complete saved actions is 0.0299999999970. +That tiny inward margin is not a certificate that a fresh FEM application +would pass exactly at the boundary. No candidate trial is evaluated, and no +nonlinear improvement is claimed. This deprioritizes selection in this +particular two-vector span; it does not rule out a richer Krylov subspace or +a trust-region method with an explicit non-Newton acceptance contract. + +The clean algebraic evidence is `minimum_norm_pair_v4/assessment.json`, with +input/output hashes and reproduction instructions. The preliminary v1 +preflight correctly refused the tight case's unflushed layout. A v2 run +produced unexpected NumPy floating-status warnings in two-column BLAS +products despite finite results; v3 replaces those products with explicit +two-column arithmetic and enables floating exceptions. The clean rerun gives +the same result. `analyze_nonlinear_direction_pair.py` preserves that +reproduction logic in this branch at `db93bf7`; v4 reruns that version with +additional RHS-hash and immutable-directory checks. + +`minimum_norm_pair_v4_binding_review.json` passes 61 additional checks +binding this assessment and its reproduced vectors to the final audited +producers. That closes the assessment's pending provenance qualification; +its explicit lack of a fresh FEM action or nonlinear trial remains. + +## Initial-state control + +The initial n = 1.5, J = 0 model has base residual 2.7365921762e-4. +Its cold solve passes in 26 iterations and 15.0205 s, with verified relative +defect 0.0281993. The raw full step prepares and reduces the residual to +1.7298999698e-4. Gravity correction reduces it to 1.61484154945e-5: a 94.10% +decrease, reproducing to roundoff the recorded residual norm of the preceding +production campaign's first accepted step. The late model's best sampled +corrected decrease is only 0.06059%, at alpha 1/1024. + +Both normalization-factor vectors are byte-identical across these states. +The following comparison therefore uses the same normalized coordinates, +although it samples different states and directions: + +| Quantity | Initial model | Late model, eta = 0.03 | +| --- | ---: | ---: | +| ||F|| | 2.736592e-4 | 1.396699e-5 | +| ||p|| | 0.00458195 | 0.0794897 | +| ||p||/||F|| | 16.7433 | 5691.26 | +| Shape/fields action cancellation factor | about 332.1 | about 110,743 | +| Corrected ||R||/(alpha^2 ||p||^2), alpha = 1/64 | 0.656881 | 0.926040 | +| Corrected ||R||/(alpha^2 ||F||), alpha = 1/64 | 0.0503939 | 418.937 | + +At the common alpha 1/64 both full corrected remainders are resolved above +the heuristic floor. The late direction is 17.35x larger despite a 19.59x +smaller residual; ||p||/||F|| increases about 340x. By contrast, sampled +curvature per unit direction increases only 1.41x. The relative quadratic +remainder coefficient ||R||/(alpha^2 ||F||) grows approximately 8,313x. +This describes the observed amplification; it does not estimate a full +Hessian norm or identify a physical nullspace. + +For these two sampled corrected paths at the common alpha, the main change +is not orders-of-magnitude growth in remainder per squared unit-direction +step. This does not isolate intrinsic raw-operator or shape-Hessian curvature. +A much larger, strongly cancelling Newton direction is +being asked to eliminate a much smaller residual. Small second-order errors +relative to the individual shape/field responses become large relative to +the desired combined response. All 48 seed paths prepare, all 24 auxiliary +gravity solves pass, and restoration differences are zero. Corrected full +vector remainders show resolved order approximately two from alpha 1 through +1/256. Smaller corrected seed remainders are excluded from order claims by +the heuristic screen including the gravity arithmetic allowance. + +## Implications and next intervention + +The measured evidence favors a bounded, fully coupled second-order trial +correction as the next isolated intervention. Keep the verified Newton +direction and capture the entire affine target L = F0+alpha J0p. For a +prepared trial y, solve J(y)c = -(F(y)-L), prepare y+c, and evaluate the +complete unchanged residual. This targets the closure, hydrostatic, surface +and mass terms that a gravity-only correction leaves or introduces. + +For a locally bounded inverse and a smooth residual, c is second order in +alpha and preserves the original tangent. The observed weak response makes +that boundedness assumption uncertain here. The experiment should therefore +limit correction norm and work, try at most one correction at a selected +prepared trial, and keep the uncorrected candidate available as a fallback. +Recheck geometry and the selected enthalpy-admissibility policy after the +correction, since it can change both shape and enthalpy. Keep that policy +identical for the corrected candidate, fallback and control: this campaign +uses sampled enthalpy without polynomial repair. Adding a polynomial +certificate requires a separate, matched admissibility control. This does +not bypass the pressure preparation failures at large alpha; it tests whether the prepared range can +deliver substantially more useful progress. + +Architecturally, implement this as a distinct, defaulted compile-time trial +path policy composed by `PreparedNewtonStrategy`, not as another gravity +specialization. Capture F0 and J0p in `CaptureDirection` before trial mutation; +compose correction in `PrepareTrial`; explicitly refresh the full inverse +after changing the prepared state. The ordinary full-residual acceptance +test and accepted-state commit remain in `stellar_context.cppm`. The public +physics model and existing equations/normalization need no new arguments. + +Independently verify the auxiliary full defect ||J(y)c+F(y)-L|| against its +own declared solve tolerance. Preserve and report the base Newton direction's +original full-J gate separately; an auxiliary success must not replace it. + +Compare full cost per useful accepted step, including extra solves, +preparations and rejected attempts. Removing a correction residual at high +cost is not itself a solver improvement. Use the same late snapshot, inverse, +forcing tolerance and virial/field diagnostics as the control; a subsequent +short nonlinear run is necessary before claiming convergence improvement. + +Tenfold tighter forcing and minimum-norm selection in the two measured +directions are lower priorities after their negative controls. Broader +Krylov-space regularization remains untested. A trust step that intentionally +misses the current Newton gate would need an explicit distinct contract; +solving a shifted system accurately cannot be reported as passing the +original full-J gate. + +Continuation in n and then angular momentum remains a separate path for +testing basin dependence, not a demonstrated cure for this local weak +response. A production continuation driver needs a typed checkpoint/state +transfer initializer: the current ordinary initializer requires a radial +seed, whereas the experimental physical snapshot loader is not a general +rotating-model restart API. Record the EOS K(n) path, mass, angular momentum, +mesh and normalization. Nothing here establishes a need for an extra n = 1.5 +central-density constraint, a translation gauge, or pseudo-arclength. + +No new equilibrium has been accepted in this investigation. The input late +state still has the previously measured negative sampled density and +undefined density-pressure virials, even though its enthalpy-pressure +virials pass 1e-5. Both internal admissibility and the virial milestone remain +required. These n = 1.5, J = 0 controls make no new claim about higher n, +rotating convergence, or any percentage of critical rotation. + +## Reproduction and artifact scope + +The campaign manifests are `nonlinear_step_late_case_2026-09-18.json` and +`nonlinear_step_controls_2026-09-18.json`. Each records every probe, linear, +and gravity-solve option. Run them serially through `run_plateau_campaign.py` +with the configured Release build, the original `sandbox.smesh`, and a new +output directory. The existing evidence root is the task workspace's +`diagnostics/nonlinear_step_2026-09-18/` directory. Its campaign directories +are immutable; reviews, analysis and figures are kept alongside them. + +`late_v1` was built at `ba057b3`; `controls_v1` at `b8d1c6a`, which only adds +the independent checker and tests. Both copied executables have SHA-256 +`5f5c4ed0dae351097ec1f632fa4132e490b35c6b8ec3aab689c5a6535b77c634`. +The mesh SHA-256 is +`ce11ae99e21e6c3bbfbee402acd2e191c1da0d8261d2227b4203f73f2f337a74`. +The recorded physical snapshot, frozen normalized state, residual, factors, +and layout provide an exact binding for the paired tolerance control. + +The checker reads every saved trial state/residual and recomputes the vector +remainders, signed projections, mixed interactions and norm identities. +It also checks the complete path grid, failed-preparation semantics, budgets, +hashes, input binding, gravity affine-target gates, counters, frozen-state +restoration and accepted-buffer invariants. A failed linear gate remains a +valid finite directional Taylor experiment, but cannot support a claimed +Newton-forcing comparison. Its fourteen synthetic tests include intentional +vector corruption, a hidden first-order error, noninteger superlinear order, +unresolved arithmetic-scale differences, missing hashes/invariants and +failed preparation/inner/prepared-gravity gates. + +Example independent review commands, replacing E with the evidence-directory +path and keeping reviews outside the immutable campaigns: + +```text +python3 experiments/check_nonlinear_step.py E/late_v1 --output E/late_v1_review.json +python3 experiments/check_nonlinear_step.py E/controls_v1 --output E/controls_v1_review.json +python3 -m unittest discover -s experiments -p test_nonlinear_step_checker.py -v +``` + +The final comparison is saved as +`figures_v1/nonlinear_step_comparison.png`, `.pdf` and `.json`. Its JSON +records source reviews/hashes and numerical comparisons. Reproduce it with +`plot_nonlinear_step.py --audited-campaign E/late_v1 E/late_v1_review.json +--audited-campaign E/controls_v1 E/controls_v1_review.json --output E/figures_new`, +using a Python installation with NumPy and Matplotlib. This run used the +existing `/opt/homebrew/anaconda3/envs/general/bin/python`; no dependencies +were installed. The final PNG was visually inspected. + +Final numerical audits comprise 162 + 349 provenance checks and 6,238 + +6,238 + 7,912 case checks, totaling 20,899. These are arithmetic/provenance +checks, not independent physical validation tests. The supplementary +two-direction assessment has 61 additional binding checks. All three probes +restore the original residual and actions exactly, preserve accepted buffers +and factors bitwise, and perform no accepted-state commits. + +Input-snapshot equality is additionally recorded in `late_v1_input_binding.json`: +all 178,073 supplied physical coefficients match exactly. The generic checker +verifies the supplied file's provenance hash and physical/normalized factors; +that separate coefficient check binds the actual imported values themselves. +Reported operator norms use the configured diagonal normalization, not exact +physical L2 norms. The subtraction/gravity allowance is a conservative +heuristic resolution screen, not a certified bound on FEM assembly error.