Beruflich Dokumente
Kultur Dokumente
Werner Rehm
Juli 2001
The Author: Dr. rer. nat. Werner Rehm, Year of Birth 1942
2
CFD with HPC
Kurzfassung
3
CFD with HPC
Abstract
This report describes the R&D work performed within the scope of
EC/HGF project activities concerning the numerical simulation of reacting flow in
complex geometries. The aim is the refinement of numerical methods used in
computational fluid dynamics (CFD) by introducing high-performance
computations (HPC) to analyse explosion processes in technical systems in more
detail. Application examples concern conventional and nuclear energy systems,
especially the safety aspects of future hydrogen technology.
The project work is mainly focused on the modelling of the accident-
related behaviour of hydrogen in safety enclosures regarding the distribution and
combustion of burnable gas mixtures, ranging from slow to fast or even rapid
flames. For fire and explosion protection, special models and criteria are being
developed for the assessment of adequate safety measures to control deflagration-
to-detonation transition (DDT) processes. Therefore, the physical mixing concept
with dilution and inertization media is applied and recommended.
4
CFD with HPC
Preface .............................................................................................................. 6
Overview ........................................................................................................... 8
Synopsis ............................................................................................................ 9
References ....................................................................................................... 44
Acknowledgments ........................................................................................... 47
Nomenclature .................................................................................................. 61
5
CFD with HPC
Preface
6
CFD with HPC
in Bonn and the EC directorate in Brussels within the scope of joint hydrogen
safety projects based on an international cooperation.
This productive cooperation was focused on hydrogen distribution,
combustion, flame acceleration and DDT processes, including CFD code impro-
vement with validation and the exploration of mitigation or prevention features.
Main aspects are related to scientific high-performance supercomputing (HPSC)
for large-scale simulations of reactive flows in complex geometries, using vector
and parallel processing capabilities, including aspects of fire protection and
explosion prevention for innovative energy systems, especially for nuclear safety
and hydrogen safety with relevance to industrial safety. Besides the optimization
of modern CFD methods with supercomputing for safety analyses, further studies
were performed for hydrogen control systems, which are used in conventional fire
protection and industrial explosion prevention related to process and
environmental safety. Therefore, several industrial companies were contacted,
e.g. ESMG, INBUREX, FIKE, BINDER, TOTAL WALTHER, KIDDE-
DEUGRA, and HYDROGAS.
In this context, we also would like to thank foreign partners from Europe,
Canada, USA, and Japan for their helpful discussions in the field of fluid dynamic
safety analyses and hydrogen safety concepts for nuclear and conventional
applications relevant in research and practice.
7
CFD with HPC
Overview
The final documentation of the HGF-H2 Project in FZJ report Part 2: CFD
Software Optimization and Special Application consists of the following
reports:
1. Final Report Part 1: Advanced CFD methods with supercomputing -
improvement, validation, and performance, by W. Rehm, C. Nae, et al.,
Aug. 2001, FZJ Report
2. Detailed Report Part 2: Recent DDT simulations and criteria based on
experimental and numerical results, by B. Wang, W. Rehm, Sep. 2001, FZJ
Report
3. Analysis Report Part 3: Benchmark calculations and performance analysis
for special application examples, by W. Jahn, W. Rehm, Oct. 2001, FZJ
Report
4. Summary Report Part 4: Evaluation and application plan for fluid
dynamic safety analysis, by W. Rehm, W. Jahn, C. Nae, B. Wang, Nov.
2001, FZJ Report
8
CFD with HPC
Synopsis
In this short summary report, we present the R&D work that is being
performed within the framework of applied computational fluid dynamics
(ACFD) with high-performance supercomputing clusters (HPSC) for joint
EC/HGF-funded CFD/DDT projects, aimed at the numerical simulation of
turbulent reacting flows in complex geometries at the so-called GGG computing
level to gain high numerical resolution in space and time. As a result, it is shown
that scientific-technical high-performance supercomputing networking (HPSN)
has the capability to enhance a close interdisciplinary collaboration between
universities, research centres, and industry. Major aspects are related to the
prediction of physical-chemical phenomena using a modern field code cluster
(MFCC), solving the reactive Navier-Stokes/Euler equations with higher order
turbulence and combustion models on a powerful platform, including code
verification and validation of numerical results by experimental observation.
In this context, special CFD project results are outlined in five selected
chapters relating to hydrogen behaviour - distribution and combustion - as well as
the transition from deflagration to detonation (DDT) in explosive hydrogen
mixtures using CRAY supercomputing systems and different parallelization
strategies. The report focuses on CFD optimization and application studies. The
main task is to obtain a sufficiently high numerical resolution of reactive flows in
complex geometries using supercomputing to study fire and explosion loads. In
addition, it is shown in the appendix that each grid size has an optimum number
of processor nodes for parallel processing. This was analysed in more detail for
large-scale multi-dimensional simulations of non/reacting flows based on several
distribution and combustion test cases. Furthermore, these project studies are
relevant with regard to the maintenance of scientific competence and expertise at
9
CFD with HPC
10
CFD with HPC
Technical Summary
12
CFD with HPC
Work Tasks: Within the scope of Part 2, special subtasks were planned in
more detail and described in the project proposal with the following topics :
• Improvement, optimization, and application of modern field codes for Fluid
Dynamic Safety Analysis in the FZJ for technical systems, especially for
hydrogen safety.
• A probabilistic safety analysis (PSA) was not performed since the working
group moved to a different department. However, PSA studies were
performed by NNC Knutsford as a partner in the EU-H2DDT project so that
these results are also available.
• A safety analysis of structural mechanics could not be performed because of
the closing period of the working group. But the finite element program
13
CFD with HPC
DYNA was ported, parallelized and tested on the T3E in a separate FZJ
project and is now also ready to use.
Contents: The final report consists of five selected chapters. The first
chapter outlines applied computational fluid dynamics (ACFD) using high-
performance supercomputer clusters (HPSC). The second chapter describes
large-scale simulations for complex engineering flows with recent solvers,
including LES methods. The third chapter summarizes experimental and
numerical studies of explosion phenomena in hydrogen-air mixtures under severe
accident conditions. The fourth chapter highlights reactive flow simulations in
complex geometries with high-performance supercomputing. The fifth chapter
gives an overview of application examples for hydrogen safety technology,
including safety systems for fire protection and explosion prevention used in
14
CFD with HPC
process and industrial safety. Besides the conclusion with outlook, an extensive
bibliography has also been compiled. Finally, the appendix collects field code
equations, the main results of parallel processing with a large number of
performance and analysis tests, including representative input data for some CFD
codes. Great effort was involved in preparing the input data sets for the several
field codes used in various test cases, including the handling of input/output data
for pre- and post-processing with visualization or animation.
Basic Equations and Field Codes: The fluid motion can be described by
field codes solving the conservation equations for mass, momentum, and energy,
which form together with the equation of state a closed mathematical system, i.e.
the so-called Navier-Stokes (Na-St.) equations. The governing equations are
formulated, as follows:
(1) ∂ρ / ∂t + div (ρ u) = 0;
(2) ∂(ρ uj) / ∂t + div(ρ uj u) = - grad p + div(µ grad uj) + SM ;
(3) ∂(ρ i) / ∂t + div(ρ i u) = - p div u + div(k grad T) + Si ;
(4) p = p (ρ, T) and i = i (ρ, T).
15
CFD with HPC
ch ) for T > T ig using the chemical induction time (τ ch) and the ignition
temperature (T ig.).
The basic partial differntial equations (PDEs) are integrated in a computing
domain, using the conservative form of finite volume methods (FVM) with
various accurate differencing schemes and robust solver techniques for the
convection, diffusion and source terms with ambient initial and boundary
conditions. The resulting set of algebraic equations is solved with direct or
indirect algorithms (i.e. iterative or coupled solver), as to: a(p) Q(p) = a(w) Q(w)
+ a(e) Q(e) for the residual reduction to the defined values and aimed at grid-
independent solutions. Here, we select from verification of codes versus
validation of results, as to an independent end-user point of view.
The basic equations contain all of the physics of a given turbulent flow, due
to the fact that turbulence is a continuum phenomenon. Turbulence modelling is
one of three key elements in CFD using RANS, LES or DNS methods, (Fig. 1).
Mathematical theories have been developed for the other two key elements, i.e.
grid generation and algorithm development. An other issue is related to the
combustion modelling of premixed flames, (Fig.2).
For the numerical simulation of transient, compressible, turbulent, chemi-
cally reacting flows, we have developed special field codes for specific studies
providing new versions of reactive Navier-Stokes and on reactive Euler flow sol-
ver codes. The software and hardware systems of the modern field code cluster
are based on an innovative methodology for CFD predictions of engineering
flows in complex geometries using high-performance supercomputing facilities.
16
CFD with HPC
high DNS
DNS
LES
LES
Resolution
Fig. 1: Overview on Navier-Stokes flow solver methods with DNS, LES, VLES,
and RANS techniques used in research, development and standard applications.
Mixed-is-Burnt Model
PDF Model DDT
Combustor Eddy
Dissipation
Model Engine
Flamelet Model
Fig. 2 Combustion models for premixed flames as to the Borghi diagram, i.e.
flame velocity ratio versus flame length scale ratio with the parameters
Damköhler and Karlovitz numbers.
17
CFD with HPC
18
CFD with HPC
Modelling Phenomena
Flame AICC
Slow Flame
Scenario
Codes subsonic Pressure
Deflagration Acceleration
CFX
Fast Flame Shock
AIXCO
turbulent, sonic Pressure
SHOCKIN
Detonation Propagation
Figs. 4: Hydrogen flame scenario (above) and reactive field codes for hydrogen
safety technology (below) of technical energy systems.
19
CFD with HPC
20
CFD with HPC
21
CFD with HPC
channel flows with obstacles. Throughout this approach, the subgrid viscosity is
computed by the subgrid scale model ( ν s ) which can be written in the form of:
ν s = (c s D ∆ ) 2 ( 2 S ij S ij ) 1/2
with the S ij resolved strain rate tensor using ∆ the grid cell, D the wall daming,
and c s as model constant. The CFX-4 version was ported to the T90 for the PVP
mode and CFX-5 was optimized on the T3E for the MPP mode. Performance
tests showed a good scalability with increasing grid cells and number of
processor nodes. Both new versions of CFX-4/5 were extensively used and
validated in experiments for hydrogen distribution and combustion tests,
especially for flame acceleration with repeated obstacles (RUT tests), for hot jet
ignition in multi-room tests (BMC tests), and for flame-quenching tests (ENEL
test). Empirical parameters were fine tuned for the selected test cases indicating
that the experimental results agreed quite well with the numerical simulations.
The CFX-4/5 code system is a comprehensive design and analysis tool for
numerical simulations of industrial flows including fire and explosion protection.
For instance, Fig. 5 displays the numerical simulation of a turbulent H2-flame
acceleration in the channel with repeated obstacles into the cavity of the large-
scale explosion RUT facility with about 500 m**3 volume.
22
CFD with HPC
derived from combustion theory or experimental results. The associated level set
function evolution equation can be written, as follows:
G t + ( v + s n ) ∇ G = 0; and | ∇ G | = 1
with G ( x, t ) = flame front level set, s = normal burning velocity, and v = flow
velocity. The effects of turbulence can be taken into account through classical
two-equation models (e.g. k-eps.). In addition, a two-step reaction model is
implemented to study self-ignition characteristics of various hydrogen-air
mixtures based on reduced chemical reaction schemes.
Furthermore, the code is able to simulate the transition from a deflagration
wave to a detonation wave, using a switching criterion defined by the critical
ignition temperature. We ported, parallelized and optimized the AIXCO code on
the CRAY-T3E, obtaining good performance tests and reasonable speedup
factors with increasing grid cells and processor nodes up to 500 processors (600
Mhz). The code has now been tested in small-scale laboratory DDT tests
provided by SWL at RWTH Aachen and large-scale explosion tests performed in
RUT-KI, Moscow, reviewing the DDT modelling. Finally, the code was extended
for very large eddy simulation (VLES) using two-scale turbulence modelling with
down- and up-scaling between the integral and the subgrid scales for non-reacting
flows, e.g. mixing layers, jets, wakes, and channel flows with obstacles.
Recently, the VLES turbulence model is adapted in combination with the flamelet
combustion model for the numerical simulation of turbulent reacting flows. In
conclusion, the AIXCO code is a modern research tool for the prediction of
turbulent combustion, based on RANS, VLES, or DNS flow solver options used
for combustion engines and hydrogen safety. For example, Fig, 6 demonstrates
the numerical simulation of a H2-flame interaction with shock wave focusing in
the corner of the RUT test cavity reaching auto-ignition conditions in the unburnt
gas ahead of the flame with DDT pre-conditioning.
23
CFD with HPC
Fig. 5: CFX (k-e, EDM) test example for flame acceleration (FA) in a channel
with temperature and pressure contours (11% H2-air at NTP).
burnt gas
unburnt gas
Density (kg/m**3)
Time = 0.178 s
Fig. 6: AIXCO (k-e, FC) test case for DDT flame-shock wave interaction in a
cavity with density contour (15% H2-air at NTP).
in out
Cavity
DDT Formation of Detonation at 20 ms
Pressure (bar)
0.3 36
Fig.7: IFSAS (1-step, AG) test case for shock wave focusing in a corner with
DDT self-ignition and detonation formation in an explosive hydrogen-air
mixture as to pressure contour (15% H2-air at NTP).
24
CFD with HPC
( ρ f ) t + ∇ ( ρ v f ) = - ρ f b exp ( - E / RT ) with T = p / ρ R
or a more sophisticated chemical rate law can be employed. The code uses
adaptive grids for static and dynamic mesh refinement, including slanted
boundaries. The code was tested and validated for representative explosion
processes (SWL/RUT tests), especially for single-effect studies associated with
shock ignition, detonation propagation over obstacles with resulting loadings in
confined and partly confined geometries for various explosive mixtures and
ambient conditions. For practical applications, an extended version is under
development. Indeed, a typical test case is presented in Fig. 7 showing the
numerical simulation of shock wave focusing with self-ignition and detonation
formation of an explosive H2-air mixture in the RUT test cavity with high DDT
pressure peaks.
25
CFD with HPC
non/reacting flow field. Most of the modules of the MFCC software system were
optimized for parallel processing on the CRAY supercomputer complex at FZJ as
a center of excellence. The CRAY complex consists of the J90, T90, and T3E,
which are integrated in a high-speed network operating in the upper GGG-
computing class, whereas the teraflops class is already under investigation for
even more challenging tasks. In this context, we studied several parallelization
strategies, focusing on massively parallel processing (MPP) using multi-blocks,
domain decomposition and message passing tools (HPF, PVM, and MPI). The
performance results were compared with the theoretical values for linear speeding
(Sp) defined by the Amdahl‘s law, as to:
26
CFD with HPC
Fig. 8: Speedup for massively parallel processing on the CRAY-T3E with the
CFX-LES test case in channel flows over obstacles using 3.000.000 grid points.
Fig. 9: Fire protection system with FM-200 spreeding in the control tower of the
airport at Düsseldorf by KD.
27
CFD with HPC
28
CFD with HPC
tentative agreement, but we used the COM code more or less as a black box, due
to the curtailed cooperation time and lack of data needed for further studies.
29
CFD with HPC
mixing and turbulent combustion. We found that the eddy dissipation combustion
model generally resulted in appropriate results, but the model constants must be
fitted to the special combustion mode, for example, related to flame acceleration
or hot jet ignition. We assumed that for the flamelet combustion models fewer
empirical contants would be required using standarized burning laws, [Ref. 18,
19].
However, predictions with scaling of turbulent combustion and DDT
processes are still a very complex task and remain an open question for hydrogen
safety applications in real geometries. Concerning DDT conditions, our modern
field code cluster offers the ability to analyse DDT conditions in complex
geometries with regard to critical flame speeds, critical shock Mach number,
critical self-ignition temperatures, onset of detonation formation and propagation
(without the cell structure). More progress has been made in simulating DDT
mode A ahead of the flame front with shock wave focusing and self-ignition in
the unburnt gas compared with DDT mode B in the flame brush with turbulent
mixing and re-ignition triggering DDT processes, which would need further
investigations and huge computer resources. In any case, the numerical simulation
of DDT conditions allows a more objective evaluation basis than in the past in
contrast to the use of simple DDT criteria in a more subjective way (e.g. only
based on detonation cell size classes and geometrical ranking). Such simple DDT
criteria must be carefully used in realistic safety studies, where it is not quite clear
whether or not simple criteria are a conservative approach and where the real
mechanisms are not taken into account. DDT results have shown that very high
reflected detonation pressure peaks can be produced in explosive H2-air mixtures
in the order of about 100 bar and more, which need suitable hydrogen control
systems to limit the resulting combustion loads inside and outside safety
enclosures, Fig. 10. The hydrogen control options are related to mitigation,
prevention, and protection, as discussed below with some pros and cons.
30
CFD with HPC
Recombiners
Lean Mixtures
4 % H2
9 % H2
Slow Flame
Explosion, DDT:
Hazard AIXCO, SHOCKIN
Regime
Fast Flame
18 % H2
Inertization
Steam Detonation
N2, CO2, He Rapid Flame
DET, IFSAS
Mixing Media
Micro Sprays
58 % H2
Limiting
Conditions
Loads: IFSAS Code
(DYNA)
70 % H2
75 % H2
Rich Mixtures
Fig. 10: Hydrogen safety features for fire and explosion protection in technical
systems, consisting of safety measures, including recombiners, igniters, or
inertization to control slow, fast or rapid flames, especially deflagration-to-
detonation transition (DDT) for anticipated events in severe accident conditions,
including predictive CFD methodology for safety assessment
31
CFD with HPC
32
CFD with HPC
systems which are strongly safety-orientated and should be considered for further
investigation and optimization. The diluation/inertization system can be designed
as an active and/or passive safety system, as already used in conventional safety
technology. The amount of inertization defines the quenching of rapid, fast, or
slow hydrogen flames. Furthermore, a cold injection of post-inertization lowers
the containment pressure as part of the accident management strategy (including
water sprays or filtered venting). Finally, an insensitive containment atmosphere
can be handled more easily and safely than a sensitive one. For
dilution/inertization measures, experimental data are available, e.g. for CO2, N2,
He, etc., or modern mixture media for fire protection and explosion prevention.
There exist several safety reasons for partial- or post-inertization measures in
PWR types, e.g. with CO2, whereas N2 pre-inertization is already used in BWR
types during operation (but not in the startup and shutdown period). Last but not
least, dilution/inertization measures make the hydrogen safety problem less
complex and it is therefore more or less reduced to the question of hydrogen
distribution and mixing quality to exclude hydrogen combustion loads in
containments. More recently, modern CFD methods with supercomputing have
been established to predict hydrogen distribution and inertization mixing rates
with a high numerical resolution in complex geometries. Therefore, we improved
our modern field code cluster (MFCC) with high-performance supercomputing
(HPSC) with special emphasis on using classical turbulence modelling for the
mean flow quantities of standard applications as well as introducing large eddy
simulations (LES) for more detailed flow quantities of engineering flows. LES
developments are now powerful numerical tools for fluid flow predictions, which
will become more effective with increasing supercomputer resources towards the
teraflops class in the near future so that more precise numerical simulations will
be possible.
33
CFD with HPC
34
CFD with HPC
active or passive injection, which should be also considered for nuclear safety
and hydrogen technology, especially for the combination of nuclear and chemical
power plants (e.g. HTR type with H2 production). Finally, microwater spray
systems can effectively suppress fires and explosions depending on the droplet
size, as experimental observations have shown. Last but not least, water is
relatively cheap and normally available in sufficient quantities. Presently, we
studying recent fire and explosion control systems for more inherent safety (i.e. a
new approach in contrast to risk reduction as acceptably low) in further project
activities.
35
CFD with HPC
36
CFD with HPC
loads with the resulting uncertainty band, it is proposed that best-estimate as well
as conservative assumptions should be used. For this reason, a huge number of
large-scale simulations will be necessary, taking into account coarse to fine grid
techniques and modern CFD flow solver algorithms, which should be preferably
executed with high-performance computing and parallel processing to obtain a
sufficiently high numerical resolution. Advanced CFD methods with
supercomputing are already in use for various applications worldwide. Also pre-,
partial- and post-inertization systems with modern mixing media should be
studied in research and practice for fire protection and explosion prevention with
relevance to industrial safety to reduce the accident consequences inside the plant
with a hydrogen risk.
37
CFD with HPC
2.5 PJ/a was required. During the following working period of the HGF-H2
project, the ISR1 was reorganized at FZ Jülich, which constrained the project
work of Part 2. The reduced working group (1-2 PJ/a) was supplemented by a
guest scientist (8 ma mo) and a post-doc (1.5 PJ) so that the project team was
now, in principle, capable of undertaking the essential working tasks with some
modifications and restrictions. The project work of Part 2 was involved in another
FZJ project for CFD simulations with CRAY supercomputing so that we received
the computer resources for the T90 (50 CPU-h/mo) and for the T3E (2500 CPU-
h/mo), which were sufficient for test cases, but limited for large-scale simulations.
Hence, we performed large-scale simulations in the order of 1 to 10 million grid
points with maximal processor nodes and an effective performance in the MPP
mode.
For the scientific-technical results obtained in Part 2, a special application
plan for further R&D work in research and practice was proposed in the field of
fire protection and explosion prevention for technical systems, e.g. for hydrogen
technology, to continue and apply the findings in a specific project group for
applied computational fluid dynamics (ACFD) with high-performance
supercomputing clusters (HPSC) in FZ Jülich (s. Tab. 2, 3). In this context, the
modern field code cluster (MFCC) developed for supercomputing, networking
and clustering in Europe forms the basis of two recently accepted EC proposals,
i.e. THERMO-NET (nuclear safety) and EXPRO (industrial safety), s. Tab. 4.
The overall project aims are to proceed with the successful project work using
scientific-technical high-performance supercomputing for innovative numerical
simulations of non/reacting flows in complex geometries with modern CFD flow
solvers for the next generation of codes, ranging from the GGG computing level
towards the teraflops computer class. Main results of the EC/HGF-CFD/DDT
project work have been published in international journals and conference papers,
e.g. Ref. [20 - 27].
38
CFD with HPC
39
CFD with HPC
40
CFD with HPC
41
CFD with HPC
42
CFD with HPC
43
CFD with HPC
References
44
CFD with HPC
45
CFD with HPC
46
CFD with HPC
Acknowledgments
This R&D work was performed within the scope of European projects on
H2-DDT (Model and Criteria Verification), as well as a FZJ-VSR project on
supercomputing (Reactive Flow Simulations) and of a German HGF-H2 project
(CFD Optimization and Application).
We acknowledge the support of the FZJ (VS, ISR and ZAM). The author is
grateful to all project participants and in particular to the EC in Brussels (Dr. G.
Van Goethem), FU Berlin (Prof. R. Klein), FZ Karlsruhe (Dr. W. Breitung), and
RU-Bochum (Prof. H. Unger) for their cooperation, especially to RWTH-
ITM/SWL Aachen (Prof. N. Peters, Dr. B. Binninger, Dr. H. Herrmann, Prof. H.
Olivier) as well as to AEA Technology (Dr. G. Scheuerer, Dr. F. Unger, Dr.
Kuntz, Dr. Forkel) for their services, including CDL in Canada (Dr. P. Thibault).
We would like to thank everybody involved at FZJ (Dr. M. Gerndt, DI W. Jahn,
Dr. C. Nae, Dr. R. Vogelsang, Dr. B. Wang) for their valuable contributions,
including the administration and infrastucture at FZ Jülich. Finally, we also thank
the ESMG group and KIDDE-DEUGRA GmbH for their helpful information and
material.
Last but not least, I would like to thank cordially my wife Leni-Keil-Rehm
for her permant patience and great interest in preparing the project work with the
documentation for publication, especially all other persons involved for many
productive discussions about further R&D developments.
47
CFD with HPC
Objectives: The overall project aim is to develop modern computational fluid dynamic
methods for hydrogen combustion studies using high-performance supercomputing clusters
with field code optimization and special application, including refinement, validation and
performance analysis. The modern field code cluster developed with parallel processing
capability aims to obtain a high numerical resolution of non/reacting flows in complex
geometries. Special application are related to hydrogen safety and explosion loads under severe
accident conditions for nuclear and conventional energy systems, including transition from
deflagration to detonation processes (DDT), s. Tab. 1 and Figs. 1 to 12, as follows.
Summary: We have developed a modern field code cluster (MFCC) with high-performance
supercomputing (HPSC) using the CRAY computer complex (T90, T3E, J90) at FZ Jülich
with parallel vector processing (PVP) and massively parallel processing (MPP). The MFCC
software system consists of Euler / Navier-Stokes flow solvers: D3UNS, CFX, AIXCO,
SHOCKIN, IFSAS and DET, which are suitable for numerically simulating slow, fast, and
rapid hydrogen flames with resulting combustion loads inside and outside of safety enclosures.
The MFCC software system with supercomputing is used as an advanced research and analysis
tool for fire and explosion protection of innovative energy systems with respect to nuclear
reactor safety and conventional industrial safety (e.g. for hydrogen technology as a future
energy option to limit the increasing climate problem). The R&D work for hydrogen safety at
FZ Jülich is performed in the framework of joint projects supported by the EC in Brussels and
the HGF in Bonn, whose cooperations are gratefully acknowledged.
48
CFD with HPC
49
CFD with HPC
∂
∂t
ρ+
∂
∂x j
(
ρ ⋅ u~ j = 0)
∂
∂t
( )
ρ ⋅u~i +
∂
∂x j
( )
ρ ⋅u~i ⋅ u~ j = −
∂
∂xi
Π+
∂
∂x j
( µ + µt )⋅ 2 ⋅ S~ij − 2 ⋅ S~ii ⋅ δij
3
∂
∂t
( )
ρ ⋅~
e +
∂
∂x j
[( ) ]
ρ ⋅ e~ + Π ⋅ u~ j =
∂
∂xi
~ ~ 2 ~ ∂
u j ⋅ µ ⋅ 2 ⋅ Sij − 3 ⋅ Sii ⋅ δij + ∂x
( k + kt )⋅
∂
Θ
∂x j
j
µ
∂
∂t
( )~
ρ ⋅Y +
∂
∂x j
( ~
ρ ⋅ u j ⋅Y =
∂
∂x j
) ⋅
∂ ~
ScY ∂x j
Y − ηY , j + ωY (x, t )
where :
C p ⋅µ
Π = ρRΘ 1
Π = p − Dii k=
3 Pr
1
e~ = C v Θ + u~i u~i ~ 1 1
Θ=T −
C p ⋅ µt
2 Dii kt =
2 Cv ρ Prt
and :
{ C
( )
U J → U JI = U J − (1 − β ) ⋅ ∇U + β ⋅ ∇U J ⋅ nIJ
1
2
( )}
D
and:
ε NUM
rε (β, γ, ∆, C S ) =
ε SGS
ε NUM =
1
V
(
⋅ (u, v, w )⋅ φ ρu , φ ρv , φ ρw )Tupwind = V1 ⋅ (u ⋅ φ ρu + v ⋅ φ ρv + w ⋅ φ ρw )upwind
2 2
~
= µ t ⋅ 2 ⋅ S ij − ⋅ S kk ⋅ δ ij = ρ ⋅ (C S ∆ ) ⋅ S ⋅ 2 ⋅ S ij − ⋅ S kk ⋅ δ ij
2 2
ε SGS
2
3 3
50
CFD with HPC
The governing continuum equations of turbulent reactive fluid flows are written in the
following two-dimensional conservative form:
U t + Fx + G y + H = Rx + S y + Q
ρ ρu ρv ρv
ρ u ρ u 2
+ p ρ vu ρ
vu
where U = ρv , F = ρuv , G = ρv + p , H =
2 i ρv 2 ,
y
ρE ρuE +up ρvE + vp ρvE + vp
ρY
ρuY
ρvY
ρvY
0 0 0
Y1 ω1
τ τ
0
Y2 ω 2
xx xy
R= τ xy , S = τ yy
, Q = 0 , Y =
... , ω = ... ,
uτ xx + vτ xy + κTx uτ xy + vτ yy + κTy 0
ρω
N
Y ω N
0 0
and τ xx = ( λ + 2 µ )ux + λv y , τ xy = τ yx = µ ( u y + v x ), τ yy = (λ + 2 µ )v y + λu x .
Here p, ρ, T, E, u and v denote pressure, density, temperature, total specific energy and
velocity vector components in the Cartesian coordinates, respectively. Yj and ωj represent the
mass fraction and production rate of jth species. λ and µ indicate coefficients of viscosity, and
κ the thermal conductivity. In the above equations i = 0 and i = 1 correspond to a planar and
an axis-symmetric flow. For the application of a detailed kinetic scheme, the production rate (ω
j) of the jth species is given by:
m=1
where ν rjm and ν ljm are the coefficients of the jth species in the mth reaction on the right- and
left-hand side, respectively. The forward and backward reaction rates of the mth reaction, ωfm
and ωbm, must be defined by the user.
51
CFD with HPC
1 2
E =e+ (u + v 2 ) ,
2
where e denotes the specific internal energy, which can be calculated according to the
thermodynamic database.
α ω
In case of utilising the two-step kinetic model, Y = and ω = 1 . α and β (0 < α and
β ω 2
β < 1) represent the remainder of the progress of the induction reaction and that of the
exothermal reaction. The increasing rates of α and β are expressed as:
dα E
ω1 = = − k1 ρ exp( − 1 ),
dt RT
dβ E E +q
ω2 = = − k 2 p 2 ( β 2 exp( − 2 ) − (1 − β ) 2 exp( − 2 ))
dt RT RT
and the total specific energy is determined by
p 1
E= + qβ + ( u 2 + v 2 )
(γ − 1) ρ 2
/Yu/: Q. Yu: Experimental and Numerical Study of Unsteady Shock and Detonation Waves,
Doctoral thesis, Stoßwellenlabor, RWTH Aachen, 1996.
/Kee/: R.J. Kee, F.M. Rupley, J.A. Miller CHEMKIN-II: a FORTRAN Chemical Kinetics
Package for the Analysis of Chemical Kinetics, SAND89-8009, 1989.
/B. Wang, W. Rehm/: Numerical Studies of Shock Reflection and Ignition, Intern. Colloqium
on Dynamics of Explosions ICDERS 2001, Seattle, USA, Aug. 2001.
52
CFD with HPC
Tab. 1: CFD/DDT-related computer models used for hydrogen safety studies at FZJ.
MFCC: D2UNS, CFX, COM, AIXCO, DET, DYNA (1) State GMD-SP2
Fig. 1: Modern field code cluster with high-performance supercomputing on the CRAY
computer complex at FZJ for the simulation of reactive flows in complex geometries (e.g.
CFX-PVM and AXICO-MPI on the T3E with massively parallel processing).
53
CFD with HPC
D E T - 2 D /T 3 E /H P F
H ig h - S p e e d F lo w w ith 5 0 0 . 0 0 0 C e lls
250
T
i 200
m 150
e 100
50
s
0
10 20 30 40 60 80 100
N u m b e r o f P ro c e s s o rs
Fig. 2a: Parallel performance of the reactive Euler solver DET-2D/HPF on the CRAY-T3E
for a detonation test case in a model geometry with obstacles using 500.000 cells.
C FX-3D /T 3E/PVM
C hannel Flow 700.000 C ells
800
Time (s)
600
400
200
0
32 64 96
Processor N odes
Fig. 2b: Parallel performance of the reactive Na-St. solver CFX-3D/PVM on the CRAY-T3E
for a complex flow test case in a tube geometry using 700.000 cells.
A IX C O - 2 D / T 3 E /M P I
B e n c h m a r k te s t
T im e ( s )
400
300
200
100
0
8 16 24 48
N um be r o f P ro c e s s o rs
Fig. 2c: Parallel performance of the reactive Na-St. solver AIXCO-2D/MPI on the CRAY-
T3E for a deflagration test case in a channel geometry with obstacles using 50.000 cells.
54
CFD with HPC
Fig. 3: Unsteady iso-vortex representation using RANS code and k-eps turbulence model
(Mach = 0.2, Reynolds = 22.000)
´
Fig. 4: Domain partitioning using load balancing for MPP
Fig. 5: Iso-temperature profile for a 2D jet using LES turbulence model (Mach = 0.10,
Reynolds = 5.000)
55
CFD with HPC
Fig. 6a: Instantaneous LES µt/µl ratio distribution (Mach = 0.15, Reynolds = 22.000)
1.E+06
beta = 0.05
beta = 0.10
1.E+05
1.E+04
1.E+03
1.E+02
1.E+01
1.E+00
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
r
Fig. 6b: Global statistics for the r criteria distribution for compressible LES simulations
56
CFD with HPC
Fig. 7: Complex analysis and postprocessing for a 3D LES test case with obstacles
57
CFD with HPC
20 µs 20 µs
Lower wall
Reaction wave
40 µs 40 µs
Reaction wave
60 µs 60 µs
Explosion wave
80 µs 80 µs
Detonation wave
58
CFD with HPC
The SHOCKIN code was applied to a containment geometry for DDT studies using a two-
step reaction model where a high pressure region is located at the lower corners of the dome.
CFX calculation for a helium injection and distribution test in a multi-compartment geometry
(Batelle Modell Containment).
BMC description :
Volume: 600m³
Air: 1 bar, 304K
Helium Injection:
Diameter: 95mm
Speed : 42m/s
Rate: ca. 0.049 kg/s
Computational details :
Code : CFX with k-eps
Blocks : 91
Elements : 68.522
Points : 99.883 Fig. 12a: BMC model geometry
59
CFD with HPC
Figs. 12b: Light gas distribution during the first 220 seconds after injection.
Figs. 12c: Light gas distribution during the first 512 seconds after injection.
References
/C. Nae/: Efficient LES using a Betta-Gamma Scheme and Wall Laws, International
Conference on Computational Fluid Dynamics ICFD 2001, Oxford, U.K., 2001.
/B. Wang, W. Jahn, W. Rehm/: Application of a CFD Code for Reactive Flows in H2-Air
Mixtures, CFD 2001 Conference - A CFD Odyssey , Waterloo, Canada, 2001.
60
CFD with HPC
Nomenclature
a Iteration coefficient
b Pre-exponential factor
c Model constant
c0 Speed of sound
f Fuel fraction
k Turbulent kinetic energy
mlim Minimal mixture fraction
n Normal vector
p Pressure
r Reaction coefficient
sl Laminar flame velocity
st Turbulent flame velocity
t Time
u Fluid velocity
u' Velocity intensity
vj Velocity vector
x Space coordinate
AEA AEA Technology GmbH
AC Combustion constant
AT Burning constant
BR Block ratio
CFDS CFD Society Canada
CDL Combustion Dynamics Ltd.
Cµ Flow constant
D Diffusion coefficient
Da Damköhler number
DNS Direct numerical simulation
E Energy
EC European Commission
EDC Eddy dissipation concept
ERCOFTAC European Research on Turbulent Flow and Combustion
61
CFD with HPC
F Flux vector
FC Flamelet concept
FZJ Forschungszentrum Jülich
FZK Forschungszentrum Karlsruhe
FUB Freie Universität Berlin
G G equation
ISR Institute for Safety Research and Reactor Technology
ITM Institute for Technical Mechanics
Ka Karlovitz number
MS Shock Mach number
LES Large eddy simulation
Q Iteration variable
R Gas constant
Re Reynolds number
RANS Reynolds-averaged numerical simulation
RWTH Rheinisch-Westfälische Technische Hochschule
S Source term
SWL Shock Wave Laboratory
ST Turbulent burning rate
SC Shock heating rate
T Temperature
Tig Ignition temperature
U Velocity vector
UC3 University Carlos 3, Madrid
UPM University Politecnica, Madrid
V Volume
W Progress variable
ZAM Central Institute for Applied Mathematics
ε Dissipation rate µ Viscosity
µT Turbulent viscosity µE/σf Schmidt-Prandtl No,
ρ Density τ Time
Φ Variable Γ Diffusion term
% H2 Volume fraction of hydrogen
62
CFD with HPC
Curriculum Vitae
63