Performance and energy efficiency of CFD codes on AMD EPYC clusters

Computational fluid dynamics (CFD) codes such as OpenFOAM are limited by memory traffic more than by arithmetic. On modern AMD EPYC processors their speed depends on how well the cells handled by each MPI rank fit into the L3 cache and on how much data the ranks exchange. Before submitting a job, a user must decide how many nodes to request and how many ranks to place on each core complex, and the fastest choice is often not the cheapest one in node hours or energy.

My work measures and models this behaviour so that users and HPC centres can choose configurations that save time, energy and cost. It grew out of benchmarking for the HiDALGO and HiDALGO2 Centres of Excellence and is carried out with colleagues from Częstochowa University of Technology, Széchenyi István University and HLRS Stuttgart.

Comparing AMD EPYC architectures for CFD

We evaluated six generations of AMD EPYC processors (Rome, Milan, Milan-X, Genoa, Genoa-X and Bergamo), with 64 to 128 cores, 256 to 1152 MB of L3 cache and 8-channel DDR4 or 12-channel DDR5 memory, on two memory-bound OpenFOAM models: the motorBike case and an urban air pollution model from HiDALGO2. To compare architectures directly we used the FVOPS metric (Finite Volumes solved Per Second). Performance shows local maxima for particular grid sizes per process, tied to each processor's attributes, and profiling with AMD uProf shows that effective use of the L2 and L3 caches and the bandwidth of memory channels are the decisive factors.

Modelling the performance–energy trade-off

A prediction model of the performance–energy trade-off helps select the number of compute nodes that reduces energy consumption, computation time, or both, for a given CFD case on an AMD EPYC cluster.

Predicting MPI rank placement and node count from data

The most recent study revisits that model with 560 measured configurations of the motorBike case on five Zen 4 and Zen 5 EPYC processors, including socket power from processor telemetry. On up to four nodes, small meshes are both fastest and cheapest on a single node on most processors; for the largest mesh the largest node count is both fastest and cheapest because its speedup is superlinear. A genuine trade-off arises only for medium meshes, where full speed costs a median of 20 to 97% more node hours. For a new mesh, reusing the configuration that was optimal on the nearest measured mesh is near optimal in 75 to 85% of decisions, and running its two recommendations raises this to 85 to 90%. Neither gradient boosted trees nor the earlier polynomial model does better.

Beyond CPUs: GPU machine learning and Python applications

The same profiling approach applies to other workloads. For multi-GPU machine learning on NVIDIA H100 systems we compared data-parallel and distributed strategies for image recognition and the cost of LLM tuning methods: switching the PyTorch tensor layout from NCHW to NHWC and enabling pinned memory gave large gains at little effort, LoRA tunes faster and with less GPU memory than DPO, and QAT is the most resource-intensive. For Python-based social-science applications using MPI we removed load imbalance with a two-stage task reordering and splitting, and resolved memory bottlenecks on ccNUMA systems with several NUMA domains, on both x86-64 and ARM processors.

Publications

Software: Quare, a pipeline that ingests and visualises profiling data from OpenFOAM runs with AMD uProf counters and MPI profilers. Talk: Efficient HPC on AMD EPYC: modelling the performance-energy trade-off for CFD codes (institute seminar, 2026). Benchmarking deliverables: HiDALGO2 and HiDALGO.

Other research topics: Divisible load scheduling · Digital twins and environmental simulation · IT for natural history collections · Virtual Laboratory and remote instrumentation · all research themes.

Go to top of the page.