Profiling and MPI validation

This guide describes the profiling and distributed-test tools added after checkpoint 25510008ddf9810eaf9821149734d526c76b087a.

Build modes

The profiling macros compile out by default. For a profiling build, enable them and leave IPO off so profile attribution is easier to interpret:

./format
cmake -S . -B cmake-build-profile-homebrew-llvm -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DMEAN_FIELD_ENABLE_PROFILING=ON \
  -DMEAN_FIELD_ENABLE_IPO=OFF
./format
cmake --build cmake-build-profile-homebrew-llvm --target tests mpi_tests -j 6

For production timings, use profiling off and IPO on:

./format
cmake -S . -B cmake-build-release-homebrew -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DMEAN_FIELD_ENABLE_PROFILING=OFF \
  -DMEAN_FIELD_ENABLE_IPO=ON
./format
cmake --build cmake-build-release-homebrew --target tests mpi_tests experiments -j 6

Always run ./format immediately before a command that can compile.

Bounded test runner

tools/profile_tests.py accepts a Catch2 filter, rank counts, repetitions, and a per-run timeout. It writes one log per run and updates the summary CSV after every repetition.

python3 tools/profile_tests.py \
  --executable cmake-build-profile-homebrew-llvm/tests \
  --ranks 1 --repeat 1 --timeout 360 \
  --output-directory /private/tmp/meanfield-profile \
  "Gravity Field Virial Consistency Across Volume Preserving Deformation" \
  -- --reporter console

MPI sweep:

python3 tools/profile_tests.py \
  --executable cmake-build-release-homebrew/tests \
  --ranks 1,2,4 --repeat 3 --timeout 360 \
  --output-directory /private/tmp/meanfield-rank-sweep \
  "Gravity Field Virial Consistency Across Volume Preserving Deformation"

Arguments after -- go to Catch2. Runner options may appear before or after the filter. On timeout, the runner sends SIGTERM to the entire process group, waits for --kill-grace, and then sends SIGKILL if necessary. This prevents MPI workers from surviving a timed-out launcher.

The CSV records rank count, repetition, status, exit code, wall time, timeout, log path, and exact command. Profile-region output is in the corresponding log.

Profile columns

Samples: recorded observations after warmups.
Warmups: discarded warmup observations.
Work/rank: minimum-to-maximum work-unit count across ranks.
Avg max ms: maximum per-rank average duration.
Min ms and Max ms: global extrema across all samples and ranks.
Total max s: maximum accumulated duration on any rank.

MPI aggregation first unions label sets across ranks. A region may therefore be visited on only one rank without causing collective-order divergence.

MPI regression suite

The dedicated mpi_tests executable covers distributed semantics without replicating the complete serial suite on every process. Registered CTest jobs use two and four ranks:

ctest --test-dir cmake-build-release-homebrew \
  --label-regex mpi --output-on-failure --timeout 180

Or build and run through the convenience target (remember that the target can compile, so format first):

./format
cmake --build cmake-build-release-homebrew --target check_mpi

The suite checks communicator splitting, global element coverage, mapped H(div) mass symmetry and linearity, divergence adjointness, a complete distributed gravity analysis/solve, and profiling with rank-local labels. Catch2 uses a fixed synchronized seed so all ranks enter tests and collectives in the same order. Only rank zero emits the report.

The current measurements use multiple MPI ranks on one host. The suite is appropriate for a multi-node launcher, but inter-host fabric behavior still needs validation on actual multi-node hardware.

Known high-level limitation

The reduced gravity and virial paths run on multiple ranks. The complete spectral equilibrium experiment does not yet do so because logical radial surface interpolation explicitly requires one rank. Supporting that path needs distributed ownership and transpose-consistent communication for interpolation rows whose surface element may be remote. This is a separate architectural feature, not a failure of the corrected gravity kernels.

Current reference results

Prepared spectral Jacobian, one rank: 3.964707 s.
Gravity virial: 205.843 s (one rank), 118.926 s (two), 65.210 s (four).
Centrifugal h refinement: 254.984 s, historically 525.601 s.
Release MPI CTest jobs: 18.12 s (two ranks), 10.11 s (four ranks).
Full release suite: 410 cases and 5,086,850 assertions passed in 2,807.831 s.

No tolerance or resolution was changed to obtain these results.

4.6 KiB Raw Blame History