# Profiling and MPI validation

This guide describes the profiling and distributed-test tools added after
checkpoint `25510008ddf9810eaf9821149734d526c76b087a`.

## Build modes

The profiling macros compile out by default. For a profiling build, enable them
and leave IPO off so profile attribution is easier to interpret:

```bash
./format
cmake -S . -B cmake-build-profile-homebrew-llvm -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DMEAN_FIELD_ENABLE_PROFILING=ON \
  -DMEAN_FIELD_ENABLE_IPO=OFF
./format
cmake --build cmake-build-profile-homebrew-llvm --target tests mpi_tests -j 6
```

For production timings, use profiling off and IPO on:

```bash
./format
cmake -S . -B cmake-build-release-homebrew -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DMEAN_FIELD_ENABLE_PROFILING=OFF \
  -DMEAN_FIELD_ENABLE_IPO=ON
./format
cmake --build cmake-build-release-homebrew --target tests mpi_tests experiments -j 6
```

Always run `./format` immediately before a command that can compile.

## Bounded test runner

`tools/profile_tests.py` accepts a Catch2 filter, rank counts, repetitions, and
a per-run timeout. It writes one log per run and updates the summary CSV after
every repetition.

```bash
python3 tools/profile_tests.py \
  --executable cmake-build-profile-homebrew-llvm/tests \
  --ranks 1 --repeat 1 --timeout 360 \
  --output-directory /private/tmp/meanfield-profile \
  "Gravity Field Virial Consistency Across Volume Preserving Deformation" \
  -- --reporter console
```

MPI sweep:

```bash
python3 tools/profile_tests.py \
  --executable cmake-build-release-homebrew/tests \
  --ranks 1,2,4 --repeat 3 --timeout 360 \
  --output-directory /private/tmp/meanfield-rank-sweep \
  "Gravity Field Virial Consistency Across Volume Preserving Deformation"
```

Arguments after `--` go to Catch2. Runner options may appear before or after the
filter. On timeout, the runner sends `SIGTERM` to the entire process group,
waits for `--kill-grace`, and then sends `SIGKILL` if necessary. This prevents
MPI workers from surviving a timed-out launcher.

The CSV records rank count, repetition, status, exit code, wall time, timeout,
log path, and exact command. Profile-region output is in the corresponding log.

## Profile columns

- `Samples`: recorded observations after warmups.
- `Warmups`: discarded warmup observations.
- `Work/rank`: minimum-to-maximum work-unit count across ranks.
- `Avg max ms`: maximum per-rank average duration.
- `Min ms` and `Max ms`: global extrema across all samples and ranks.
- `Total max s`: maximum accumulated duration on any rank.

MPI aggregation first unions label sets across ranks. A region may therefore be
visited on only one rank without causing collective-order divergence.

## MPI regression suite

The dedicated `mpi_tests` executable covers distributed semantics without
replicating the complete serial suite on every process. Registered CTest jobs
use two and four ranks:

```bash
ctest --test-dir cmake-build-release-homebrew \
  --label-regex mpi --output-on-failure --timeout 180
```

Or build and run through the convenience target (remember that the target can
compile, so format first):

```bash
./format
cmake --build cmake-build-release-homebrew --target check_mpi
```

The suite checks communicator splitting, global element coverage, mapped H(div)
mass symmetry and linearity, divergence adjointness, a complete distributed
gravity analysis/solve, and profiling with rank-local labels. Catch2 uses a
fixed synchronized seed so all ranks enter tests and collectives in the same
order. Only rank zero emits the report.

The current measurements use multiple MPI ranks on one host. The suite is
appropriate for a multi-node launcher, but inter-host fabric behavior still
needs validation on actual multi-node hardware.

## Known high-level limitation

The reduced gravity and virial paths run on multiple ranks. The complete
spectral equilibrium experiment does not yet do so because logical radial
surface interpolation explicitly requires one rank. Supporting that path needs
distributed ownership and transpose-consistent communication for interpolation
rows whose surface element may be remote. This is a separate architectural
feature, not a failure of the corrected gravity kernels.

## Current reference results

- Prepared spectral Jacobian, one rank: 3.964707 s.
- Gravity virial: 205.843 s (one rank), 118.926 s (two), 65.210 s (four).
- Centrifugal h refinement: 254.984 s, historically 525.601 s.
- Release MPI CTest jobs: 18.12 s (two ranks), 10.11 s (four ranks).
- Full release suite: 410 cases and 5,086,850 assertions passed in 2,807.831 s.

No tolerance or resolution was changed to obtain these results.