Snapshot optimized solver and audit diagnostics before numerical experiments

docs/profiling_and_mpi.md

@@ -0,0 +1,126 @@
+# Profiling and MPI validation
+
+This guide describes the profiling and distributed-test tools added after
+checkpoint `25510008ddf9810eaf9821149734d526c76b087a`.
+
+## Build modes
+
+The profiling macros compile out by default. For a profiling build, enable them
+and leave IPO off so profile attribution is easier to interpret:
+
+```bash
+./format
+cmake -S . -B cmake-build-profile-homebrew-llvm -G Ninja \
+  -DCMAKE_BUILD_TYPE=Release \
+  -DMEAN_FIELD_ENABLE_PROFILING=ON \
+  -DMEAN_FIELD_ENABLE_IPO=OFF
+./format
+cmake --build cmake-build-profile-homebrew-llvm --target tests mpi_tests -j 6
+```
+
+For production timings, use profiling off and IPO on:
+
+```bash
+./format
+cmake -S . -B cmake-build-release-homebrew -G Ninja \
+  -DCMAKE_BUILD_TYPE=Release \
+  -DMEAN_FIELD_ENABLE_PROFILING=OFF \
+  -DMEAN_FIELD_ENABLE_IPO=ON
+./format
+cmake --build cmake-build-release-homebrew --target tests mpi_tests experiments -j 6
+```
+
+Always run `./format` immediately before a command that can compile.
+
+## Bounded test runner
+
+`tools/profile_tests.py` accepts a Catch2 filter, rank counts, repetitions, and
+a per-run timeout. It writes one log per run and updates the summary CSV after
+every repetition.
+
+```bash
+python3 tools/profile_tests.py \
+  --executable cmake-build-profile-homebrew-llvm/tests \
+  --ranks 1 --repeat 1 --timeout 360 \
+  --output-directory /private/tmp/meanfield-profile \
+  "Gravity Field Virial Consistency Across Volume Preserving Deformation" \
+  -- --reporter console
+```
+
+MPI sweep:
+
+```bash
+python3 tools/profile_tests.py \
+  --executable cmake-build-release-homebrew/tests \
+  --ranks 1,2,4 --repeat 3 --timeout 360 \
+  --output-directory /private/tmp/meanfield-rank-sweep \
+  "Gravity Field Virial Consistency Across Volume Preserving Deformation"
+```
+
+Arguments after `--` go to Catch2. Runner options may appear before or after the
+filter. On timeout, the runner sends `SIGTERM` to the entire process group,
+waits for `--kill-grace`, and then sends `SIGKILL` if necessary. This prevents
+MPI workers from surviving a timed-out launcher.
+
+The CSV records rank count, repetition, status, exit code, wall time, timeout,
+log path, and exact command. Profile-region output is in the corresponding log.
+
+## Profile columns
+
+- `Samples`: recorded observations after warmups.
+- `Warmups`: discarded warmup observations.
+- `Work/rank`: minimum-to-maximum work-unit count across ranks.
+- `Avg max ms`: maximum per-rank average duration.
+- `Min ms` and `Max ms`: global extrema across all samples and ranks.
+- `Total max s`: maximum accumulated duration on any rank.
+
+MPI aggregation first unions label sets across ranks. A region may therefore be
+visited on only one rank without causing collective-order divergence.
+
+## MPI regression suite
+
+The dedicated `mpi_tests` executable covers distributed semantics without
+replicating the complete serial suite on every process. Registered CTest jobs
+use two and four ranks:
+
+```bash
+ctest --test-dir cmake-build-release-homebrew \
+  --label-regex mpi --output-on-failure --timeout 180
+```
+
+Or build and run through the convenience target (remember that the target can
+compile, so format first):
+
+```bash
+./format
+cmake --build cmake-build-release-homebrew --target check_mpi
+```
+
+The suite checks communicator splitting, global element coverage, mapped H(div)
+mass symmetry and linearity, divergence adjointness, a complete distributed
+gravity analysis/solve, and profiling with rank-local labels. Catch2 uses a
+fixed synchronized seed so all ranks enter tests and collectives in the same
+order. Only rank zero emits the report.
+
+The current measurements use multiple MPI ranks on one host. The suite is
+appropriate for a multi-node launcher, but inter-host fabric behavior still
+needs validation on actual multi-node hardware.
+
+## Known high-level limitation
+
+The reduced gravity and virial paths run on multiple ranks. The complete
+spectral equilibrium experiment does not yet do so because logical radial
+surface interpolation explicitly requires one rank. Supporting that path needs
+distributed ownership and transpose-consistent communication for interpolation
+rows whose surface element may be remote. This is a separate architectural
+feature, not a failure of the corrected gravity kernels.
+
+## Current reference results
+
+- Prepared spectral Jacobian, one rank: 3.964707 s.
+- Gravity virial: 205.843 s (one rank), 118.926 s (two), 65.210 s (four).
+- Centrifugal h refinement: 254.984 s, historically 525.601 s.
+- Release MPI CTest jobs: 18.12 s (two ranks), 10.11 s (four ranks).
+- Full release suite: 410 cases and 5,086,850 assertions passed in 2,807.831 s.
+
+No tolerance or resolution was changed to obtain these results.