You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<p>MFC: An open-source high-order multi-component, multi-phase, and multi-scale compressible flow solver. <ahref="https://doi.org/10.1016/j.cpc.2020.107396">S. H. Bryngelson, K. Schmidmayer, V. Coralic, K. Maeda, J. Meng, T. Colonius (2021) Computer Physics Communications <b>266</b>, 107396</a></p>
<li><aclass="anchor" id="Allaire02"></a>Allaire, G., Clerc, S., and Kokh, S. (2002). A five-equation model for the simulation of interfaces between compressible fluids. Journal of Computational Physics, 181(2):577–616.</li>
<p>MFC can be run using <code>mfc.sh</code>'s <code>run</code> command. It supports interactive and batch execution. Batch mode is designed for multi-node distributed systems (supercomputers) equipped with a scheduler such as PBS, SLURM, or LSF. A full (and up-to-date) list of available arguments can be acquired with <code>./mfc.sh run -h</code>.</p>
140
140
<p>MFC supports running simulations locally (Linux, MacOS, and Windows) as well as several supercomputer clusters, both interactively and through batch submission.</p>
141
141
<dlclass="section important"><dt>Important</dt><dd>Running simulations locally should work out of the box. On supported clusters, you can append <code>-c <computer name></code> on the command line to instruct the MFC toolchain to make use of the template file <code>toolchain/templates/<computer name>.mako</code>. You can browse that directory and contribute your own files. Since systems and their schedulers do not have a standardized syntax to request certain resources, MFC can only provide support for a restricted subset of common or user-contributed configuration options. <br/>
@@ -149,7 +149,7 @@
149
149
</ul>
150
150
</dd></dl>
151
151
<p>Please refer to <code>./mfc.sh run -h</code> for a complete list of arguments and options, along with their defaults.</p>
152
-
<h1><aclass="anchor" id="autotoc_md108"></a>
152
+
<h1><aclass="anchor" id="autotoc_md122"></a>
153
153
Interactive Execution</h1>
154
154
<p>To run all stages of MFC, that is <ahref="https://github.com/MFlowCode/MFC/tree/master/src/pre_process/">pre_process</a>, <ahref="https://github.com/MFlowCode/MFC/tree/master/src/simulation/">simulation</a>, and <ahref="https://github.com/MFlowCode/MFC/tree/master/src/post_process/">post_process</a> on the sample case <ahref="https://github.com/MFlowCode/MFC/tree/master/examples/2D_shockbubble/">2D_shockbubble</a>,</p>
155
155
<divclass="fragment"><divclass="line">./mfc.sh run examples/2D_shockbubble/case.py</div>
<li>Running <ahref="https://github.com/MFlowCode/MFC/tree/master/src/simulation/">simulation</a> and <ahref="https://github.com/MFlowCode/MFC/tree/master/src/post_process/">post_process</a> using 4 cores:</li>
164
164
</ul>
165
165
<divclass="fragment"><divclass="line">./mfc.sh run examples/2D_shockbubble/case.py -t simulation post_process -n 4</div>
166
-
</div><!-- fragment --><h1><aclass="anchor" id="autotoc_md109"></a>
166
+
</div><!-- fragment --><h1><aclass="anchor" id="autotoc_md123"></a>
167
167
Batch Execution</h1>
168
168
<p>The MFC detects which scheduler your system is using and handles the creation and execution of batch scripts. The batch engine is requested via the <code>-e batch</code> option. The number of nodes can be specified with the <code>-N</code> (i.e., <code>--nodes</code>) option.</p>
169
169
<p>We provide a list of (baked-in) submission batch scripts in the <code>toolchain/templates</code> folder.</p>
<p>As an example, one might request GPUs on a SLURM system using the following:</p>
180
180
<p><b>Disclaimer</b>: IBM's JSRUN on LSF-managed computers does not use the traditional node-based approach to allocate resources. Therefore, the MFC constructs equivalent resource sets in the task and GPU count.</p>
181
-
<h2><aclass="anchor" id="autotoc_md110"></a>
181
+
<h2><aclass="anchor" id="autotoc_md124"></a>
182
182
GPU Profiling</h2>
183
-
<h3><aclass="anchor" id="autotoc_md111"></a>
183
+
<h3><aclass="anchor" id="autotoc_md125"></a>
184
184
NVIDIA GPUs</h3>
185
185
<p>MFC provides two different arguments to facilitate profiling with NVIDIA Nsight. <b>Please ensure the used argument is placed at the end so their respective flags can be appended.</b></p><ul>
186
186
<li>Nsight Systems (Nsys): <code>./mfc.sh run ... -t simulation --nsys [nsys flags]</code> allows one to visualize MFC's system-wide performance with <ahref="https://developer.nvidia.com/nsight-systems">NVIDIA Nsight Systems</a>. NSys is best for understanding the order and execution times of major subroutines (WENO, Riemann, etc.) in MFC. When used, <code>--nsys</code> will run the simulation and generate <code>.nsys-rep</code> files in the case directory for all targets. These files can then be imported into Nsight System's GUI, which can be downloaded <ahref="https://developer.nvidia.com/nsight-systems/get-started#latest-Platforms">here</a>. To keep the report files small, it is best to run case files with a few timesteps. Learn more about NVIDIA Nsight Systems <ahref="https://docs.nvidia.com/nsight-systems/UserGuide/index.html">here</a>.</li>
187
187
<li>Nsight Compute (NCU): <code>./mfc.sh run ... -t simulation --ncu [ncu flags]</code> allows one to conduct kernel-level profiling with <ahref="https://developer.nvidia.com/nsight-compute">NVIDIA Nsight Compute</a>. NCU provides profiling information for every subroutine called and is more detailed than NSys. When used, <code>--ncu</code> will output profiling information for all subroutines, including elapsed clock cycles, memory used, and more after the simulation is run. Adding this argument will significantly slow the simulation and should only be used on case files with a few timesteps. Learn more about NVIDIA Nsight Compute <ahref="https://docs.nvidia.com/nsight-compute/NsightCompute/index.html">here</a>.</li>
188
188
</ul>
189
-
<h3><aclass="anchor" id="autotoc_md112"></a>
189
+
<h3><aclass="anchor" id="autotoc_md126"></a>
190
190
AMD GPUs</h3>
191
191
<ul>
192
192
<li>Rocprof Systems (RSYS): <code>./mfc.sh run ... -t simulation --rsys --hip-trace [rocprof flags]</code> allows one to visualize MFC's system-wide performance with <ahref="https://ui.perfetto.dev/">Perfetto UI</a>. When used, <code>--roc</code> will run the simulation and generate files in the case directory for all targets. <code>results.json</code> can then be imported in <ahref="https://ui.perfetto.dev/">Perfetto's UI</a>. Learn more about AMD Rocprof <ahref="https://rocm.docs.amd.com/projects/rocprofiler/en/docs-5.5.1/rocprof.html">here</a> It is best to run case files with few timesteps to keep the report file sizes manageable.</li>
193
193
<li>Rocprof Compute (RCU): <code>./mfc.sh run ... -t simulation --rcu -n <name> [rocprof-compute flags]</code> allows one to conduct kernel-level profiling with <ahref="https://rocm.docs.amd.com/projects/rocprofiler-compute/en/latest/what-is-rocprof-compute.html">ROCm Compute Profiler</a>. When used, <code>--rcu</code> will output profiling information for all subroutines, including rooflines, cache usage, register usage, and more, after the simulation is run. Adding this argument will moderately slow down the simulation and run the MFC executable several times. For this reason, it should only be used with case files with few timesteps.</li>
<p>When running a simulation, MFC generates a <code>./restart_data</code> folder in the case directory that contains <code>lustre_*.dat</code> files that can be used to restart a simulation from saved timesteps. This allows a user to simulate some timestep $X$, then continue it to run to another timestep $Y$, where $Y > X$. The user can also choose to add new patches at the intermediate timestep.</p>
<p>To run MFC's test suite, run </p><divclass="fragment"><divclass="line">./mfc.sh test -j <thread count></div>
140
140
</div><!-- fragment --><p>It will generate and run test cases, comparing their output to previous runs from versions of MFC considered accurate. <em>golden files</em>, stored in the <code>tests/</code> directory contain this data, aggregating <code>.dat</code> files generated when running MFC. A test is considered passing when our error tolerances are met in order to maintain a high level of stability and accuracy. <code>./mfc.sh test</code> has the following unique options:</p><ul>
141
141
<li><code>-l</code> outputs the full list of tests</li>
@@ -148,7 +148,7 @@
148
148
</ul>
149
149
<p>To specify a computer, pass the <code>-c</code> flag to <code>./mfc.sh run</code> like so: </p><divclass="fragment"><divclass="line">./mfc.sh test -j <thread count> -- -c <computer name></div>
150
150
</div><!-- fragment --><p> where <code><computer name></code> could be <code>phoenix</code> or any of the others in the <ahref="https://github.com/MFlowCode/MFC/tree/master/toolchain/templates">templates</a>). You can create new templates with the appropriate run commands or omit this option. The use of <code>--</code> in the above command passes options to the <code>./mfc.sh run</code> command underlying the <code>./mfc.sh test</code>.</p>
151
-
<h2><aclass="anchor" id="autotoc_md116"></a>
151
+
<h2><aclass="anchor" id="autotoc_md130"></a>
152
152
Creating Tests</h2>
153
153
<p>Creating and updating test cases can be done with the following command line arguments:</p><ul>
154
154
<li><code>--generate</code> to generate golden files for a new test case</li>
<p>If a trace is empty (that is, the empty string <code>""</code>), it will not appear in the final trace, but any case parameter variations associated with it will still be applied.</p>
197
197
<p>Finally, the case is appended to the <code>cases</code> list, which will be returned by the <code>list_cases</code> function.</p>
198
-
<h2><aclass="anchor" id="autotoc_md117"></a>
198
+
<h2><aclass="anchor" id="autotoc_md131"></a>
199
199
Testing Post Process</h2>
200
200
<p>To test the post-processing code, append the <code>-a</code> or <code>--test-all</code> option: </p><divclass="fragment"><divclass="line">./mfc.sh test -a -j 8</div>
201
201
</div><!-- fragment --><p>This argument will re-run the test stack with ‘parallel_io='T’<code>, which generates silo_hdf5 files. It will also turn most write parameters (</code>*_wrt<code>) on. Then, it searches through the silo files using</code>h5dump<code>to ensure that there are no</code>NaN<code>s or</code>Infinity<code>s. Although adding this option does not guarantee that accurate</code>.silo` files are generated, it does ensure that the post-process code does not fail or produce malformed data. </p>
0 commit comments