Parallel I/O¶
For large-scale applications using thousands of processors, doing proper I/O is often as important as having efficient parallel algorithms. 2DECOMP&FFT provides a parallel I/O module to help applications handle large data set in parallel. This module takes advantage of the decomposition information available in the library kernel and uses either the MPI-IO or ADIOS2 backends (selected at compile time) to implement some most frequently used I/O functions for applications based on 3D Cartesian data structures.
All the I/O functions have been packed in a Fortran module:
use decomp_2d_io
To write a single three-dimensional array to a file¶
call decomp_2d_write_one(ipencil,var,directory,filename,icoarse,io_name)
where ipencil
describes how the data is distributed (valid values are: 1 for X-pencil; 2 for
Y-pencil and 3 for Z-pencil); var
is the reference to the data array, which can be either real or
complex; directory
is the path to where I/O should be written; filename
is the name of the
file to be written; icoarse
indicates whether the I/O should be coarsend (valid values are: 0
for no; 1 for the nstat
and 2 for the nvisu
coarsenings); io_name
is the name of the I/O
group to be used. A more general form of the subroutine is:
call decomp_2d_write_one(ipencil,var,directory,filename,icoarse,io_name,opt_decomp,reduce_prec,opt_deferred_writes)
where the global size of the data array is described by the decomposition object opt_decomp
(as
discussed in the Advanced 2D Decomposition API), allowing distributed array of arbitrary size to be
written. The file written would contain the 3D array in its natural ijk-order so that it can be
easily post-processed (for example by a serial code). A corresponding read routine is also
available. The optional logical
arguments reduce_prec
and opt_deferred_writes
enable
reduced precision writing (when enabled at compile time, MPI-IO only; default .true.
) and
enabling ADIOS2’s deferred writer mode (default .true.
).
To write multiple three-dimensional variables into a file¶
The 2DECOMP&FFT library supports two ways of writing multiple variables to a single file which may be used for check-pointing purposes, for example. The newer interface is described first and allows codes to use the ADIOS2 and MPI-IO backends, the older interface is supported for backwards compatibility.
When decomp_2d_write_one
is called, the directory
and io_name
are combined to check
whether a particular output location is already opened, if not then a new file will be opened and
written to - this is the “standard” use. If, however, a file is opened first then the call to
decomp_2d_write_one
will append to the current file, resulting in a single file with multiple
fields. Once the check-pointing is complete the file can then be closed.
The following describes the original interface for writing multiple variables to a file and is only supported by the MPI-IO backend. This function is very useful when creating check-point files or result files. It is in the form of:
call decomp_2d_write_var(fh,disp,ipencil,var)
where fh
is a MPI-IO file handle provided by the application (file opened using MPI_FILE_OPEN);
ipencil
describes the distribution of the input data (valid values are: 1 for X-pencil; 2 for
Y-pencil and 3 for Z-pencil); disp
(meaning displacement) is a variable of kind MPI_OFFSET_KIND
and of intent INOUT - it is like a pointer or file cursor tracking the location where the next chunk
of data would be written. It is assumed that the data array is in default size, otherwise the
function also takes a second and more general form:
call decomp_2d_write_var(fh,disp,ipencil,var,opt_decomp)
where the decomposition object opt_decomp
describes the arbitrary size of the global array.
To create a restart/checkpointing file, it is often necessary to save key scalar variables as well. This can be done using:
call decomp_2d_write_scalar(fh,disp,n,var)
where var
is a 1D array containing n scalars of the same data type. The supported data types
are: real, complex and integer.
These subroutines have corresponding read routines with exactly the same set of parameters. Applications are responsible for closing the file after everything is written (using MPI_FILE_CLOSE). Again, each chunk of data in the file contains one variable stored in its natural order. One major benefit is that it becomes very easy to read the data back into the application using a different number of MPI processes.
To write a 2D slice of data from a 3D variable¶
call decomp_2d_write_plane(ipencil,var,iplane,n,filename,io_name,opt_decomp)
where ipencil
describes the distribution of the 3D variable var
; iplane
defines the
direction of the desired 2D slice (1 for X-plane; 2 for Y-plane and 3 for Z-plane); n
specifies
which plane to write out (in global coordinate system) when positive, a value n<=1
will output a
plane of the average along the iplane
axis; filename
is the name of the file to be
written; and io_name
is the I/O group performing the write. As before, opt_decomp
is an
optional parameter that can be used when var
is of non-default size.
To write out a 3D variable in lower resolution to a file¶
Applications using the 2D decomposition often handle quite large data sets. It may not be practical to write out everything for post-processing. The following subroutine provide a convenient way to write only a subset of the data for analysis.
call decomp_2d_write_every(ipencil,var,iskip,jskip,kskip,filename,from1)
where once again ipencil
describes the distribution of the 3D variable var and filename
defines the name of the file to be written. Every iskip
-th points of the data in X-direction,
jskip
-th in Y-direction and every kskip
-th in Z-direction are to be written. Finally
from1
is a boolean flag. Assuming every n-th data points are to be written out, the data points
have indices of:
1,n+1,2n+1… if from1 is
.true.
n,2n,3n… ff from1 is
.false.
Quick I/O Reference¶
The following table summarises the supported I/O types and data types of the subroutines:
IO functions |
I/O type |
Data type |
decomp * |
|||
---|---|---|---|---|---|---|
Read |
Write |
Real |
Cmplx |
Int |
||
_one |
X |
X |
X |
X |
X |
|
_var |
X |
X |
X |
X |
X |
|
_scalar |
X |
X |
X |
X |
X |
N/A |
_plane |
X |
X |
X |
X |
||
_every |
X |
X |
X |
- *
decomp refers to a decomposition object that describes an arbitrary-size global data set.
ADIOS2 backend for I/O¶
By default 2DECOMP&FFT will build with the MPI-IO backend for I/O. The alternative ADIOS2 backend
can be selected at compile time, by either specifying -DIO_BACKEND=adios2
during configure or by
modifying the build configuration via ccmake
. Due to the way ADIOS2 works, there are a few
changes necessary to allow codes to work with either backend interchangeably.
Registering variables for I/O¶
Registering a variable for I/O informs ADIOS2 about the variables size and type, with MPI-IO this
call becomes a no-op
.
subroutine decomp_2d_register_variable(io_name,varname,ipencil,icoarse,iplane,type,opt_decomp,opt_nplanes)
The variable is associated with an I/O group through io_name
; given a name varname
;
ipencil
, icoarse
and iplane
determine the orientation and size of the data for I/O (see
previous descriptions, use iplane=0
for 3D data); and type
specifies the kind
of the
data, only real
data is currently supported in ADIOS2, i.e. real(kind=type)
. The optional
arguments opt_decomp
and opt_nplanes
are a decomposition object for non-standard sizes (see
previous descriptions) and opt_nplanes
allows controlling how many planes are written in planar
output (default 1).
Opening a file for reading or writing¶
The mode of operation for ADIOS2 is to open a file for I/O and keep this open, buffering multiple fields before performing the I/O.
subroutine decomp_2d_open_io(io_name, io_dir, mode)
The output destination io_dir
is where all data will be written to from the group io_name
,
the mode
can take the values decomp_2d_write_mode
, decomp_2d_read_mode
or
decomp_2d_append_mode
to write, read or append to a file, respectively. This is required by all
ADIOS2 I/O, when using the MPI-IO backend I/O operations will open the file on-demand, in which case
I/O is file-per-field, or if the file is explicitly opened then subsequent I/O call will be into the
same file.
Beginning an I/O step¶
ADIOS2 performs I/O in “steps”, this subroutine marks the beginning of a step to queue up I/O operations.
subroutine decomp_2d_start_io(io_name,io_dir)
The arguments io_name
and io_dir
are as described above. This is a no-op
in the MPI-IO
backend.
Ending an I/O step¶
subroutine decomp_2d_end_io(io_name, io_dir)
This subroutine marks the end of an I/O step and ADIOS2 can begin performing I/O operations. This is
a no-op
in the MPI-IO backend.
Closing a file¶
subroutine decomp_2d_close_io(io_name, io_dir)
Closes the I/O destination, must be matched with corresponding call to decomp_2d_open_io
. By
closing the I/O ADIOS2 is forced to perform a flush
operation.