127 lines
4.6 KiB
Markdown
127 lines
4.6 KiB
Markdown
# Profiling and MPI validation
|
|
|
|
This guide describes the profiling and distributed-test tools added after
|
|
checkpoint `25510008ddf9810eaf9821149734d526c76b087a`.
|
|
|
|
## Build modes
|
|
|
|
The profiling macros compile out by default. For a profiling build, enable them
|
|
and leave IPO off so profile attribution is easier to interpret:
|
|
|
|
```bash
|
|
./format
|
|
cmake -S . -B cmake-build-profile-homebrew-llvm -G Ninja \
|
|
-DCMAKE_BUILD_TYPE=Release \
|
|
-DMEAN_FIELD_ENABLE_PROFILING=ON \
|
|
-DMEAN_FIELD_ENABLE_IPO=OFF
|
|
./format
|
|
cmake --build cmake-build-profile-homebrew-llvm --target tests mpi_tests -j 6
|
|
```
|
|
|
|
For production timings, use profiling off and IPO on:
|
|
|
|
```bash
|
|
./format
|
|
cmake -S . -B cmake-build-release-homebrew -G Ninja \
|
|
-DCMAKE_BUILD_TYPE=Release \
|
|
-DMEAN_FIELD_ENABLE_PROFILING=OFF \
|
|
-DMEAN_FIELD_ENABLE_IPO=ON
|
|
./format
|
|
cmake --build cmake-build-release-homebrew --target tests mpi_tests experiments -j 6
|
|
```
|
|
|
|
Always run `./format` immediately before a command that can compile.
|
|
|
|
## Bounded test runner
|
|
|
|
`tools/profile_tests.py` accepts a Catch2 filter, rank counts, repetitions, and
|
|
a per-run timeout. It writes one log per run and updates the summary CSV after
|
|
every repetition.
|
|
|
|
```bash
|
|
python3 tools/profile_tests.py \
|
|
--executable cmake-build-profile-homebrew-llvm/tests \
|
|
--ranks 1 --repeat 1 --timeout 360 \
|
|
--output-directory /private/tmp/meanfield-profile \
|
|
"Gravity Field Virial Consistency Across Volume Preserving Deformation" \
|
|
-- --reporter console
|
|
```
|
|
|
|
MPI sweep:
|
|
|
|
```bash
|
|
python3 tools/profile_tests.py \
|
|
--executable cmake-build-release-homebrew/tests \
|
|
--ranks 1,2,4 --repeat 3 --timeout 360 \
|
|
--output-directory /private/tmp/meanfield-rank-sweep \
|
|
"Gravity Field Virial Consistency Across Volume Preserving Deformation"
|
|
```
|
|
|
|
Arguments after `--` go to Catch2. Runner options may appear before or after the
|
|
filter. On timeout, the runner sends `SIGTERM` to the entire process group,
|
|
waits for `--kill-grace`, and then sends `SIGKILL` if necessary. This prevents
|
|
MPI workers from surviving a timed-out launcher.
|
|
|
|
The CSV records rank count, repetition, status, exit code, wall time, timeout,
|
|
log path, and exact command. Profile-region output is in the corresponding log.
|
|
|
|
## Profile columns
|
|
|
|
- `Samples`: recorded observations after warmups.
|
|
- `Warmups`: discarded warmup observations.
|
|
- `Work/rank`: minimum-to-maximum work-unit count across ranks.
|
|
- `Avg max ms`: maximum per-rank average duration.
|
|
- `Min ms` and `Max ms`: global extrema across all samples and ranks.
|
|
- `Total max s`: maximum accumulated duration on any rank.
|
|
|
|
MPI aggregation first unions label sets across ranks. A region may therefore be
|
|
visited on only one rank without causing collective-order divergence.
|
|
|
|
## MPI regression suite
|
|
|
|
The dedicated `mpi_tests` executable covers distributed semantics without
|
|
replicating the complete serial suite on every process. Registered CTest jobs
|
|
use two and four ranks:
|
|
|
|
```bash
|
|
ctest --test-dir cmake-build-release-homebrew \
|
|
--label-regex mpi --output-on-failure --timeout 180
|
|
```
|
|
|
|
Or build and run through the convenience target (remember that the target can
|
|
compile, so format first):
|
|
|
|
```bash
|
|
./format
|
|
cmake --build cmake-build-release-homebrew --target check_mpi
|
|
```
|
|
|
|
The suite checks communicator splitting, global element coverage, mapped H(div)
|
|
mass symmetry and linearity, divergence adjointness, a complete distributed
|
|
gravity analysis/solve, and profiling with rank-local labels. Catch2 uses a
|
|
fixed synchronized seed so all ranks enter tests and collectives in the same
|
|
order. Only rank zero emits the report.
|
|
|
|
The current measurements use multiple MPI ranks on one host. The suite is
|
|
appropriate for a multi-node launcher, but inter-host fabric behavior still
|
|
needs validation on actual multi-node hardware.
|
|
|
|
## Known high-level limitation
|
|
|
|
The reduced gravity and virial paths run on multiple ranks. The complete
|
|
spectral equilibrium experiment does not yet do so because logical radial
|
|
surface interpolation explicitly requires one rank. Supporting that path needs
|
|
distributed ownership and transpose-consistent communication for interpolation
|
|
rows whose surface element may be remote. This is a separate architectural
|
|
feature, not a failure of the corrected gravity kernels.
|
|
|
|
## Current reference results
|
|
|
|
- Prepared spectral Jacobian, one rank: 3.964707 s.
|
|
- Gravity virial: 205.843 s (one rank), 118.926 s (two), 65.210 s (four).
|
|
- Centrifugal h refinement: 254.984 s, historically 525.601 s.
|
|
- Release MPI CTest jobs: 18.12 s (two ranks), 10.11 s (four ranks).
|
|
- Full release suite: 410 cases and 5,086,850 assertions passed in 2,807.831 s.
|
|
|
|
No tolerance or resolution was changed to obtain these results.
|