Function utils::mpiAllReceivedSigterm

Function Documentation

inline auto utils::mpiAllReceivedSigterm(const bool local) -> bool

Reconcile a per-rank boolean flag to true if any rank’s own is true.

A signal like SIGTERM is caught independently, per-process (see utils::sigtermWasReceived()) – under MPI, only the rank(s) an external SIGTERM was actually sent to (in principle, an entire job’s process group, but not guaranteed, and not even guaranteed to be observed at the same moment by every rank that did receive it) would ever see it, and every rank still needs to reach the exact same decision at the exact same point for OutputManagerH5::checkpoint()/its own destructor’s barrier- synchronized calls to stay in lockstep across ranks: a rank stopping alone, while others continue on to their own next mpiBarrier() call, would otherwise deadlock those other ranks forever, since nothing they call is left to make that barrier’s matching call on the now-stopped rank’s behalf. This is a single cheap collective (one MPI_Allreduce), called at the same, infrequent points sigtermWasReceived() itself already was (once per batch, not per trial), so its own cost is negligible next to those already-existing per-batch/per-checkpoint collectives.

Parameters:

local – This rank’s own value of the flag

Returns:

local, unchanged, when SLUG_MPI is not defined or mpiSize() == 1; otherwise true if any rank passed true, false only if every rank passed false