============ Parallel I/O ============ For large-scale applications using thousands of processors, doing proper I/O is often as important as having efficient parallel algorithms. 2DECOMP&FFT provides a parallel I/O module to help applications handle large data set in parallel. This module takes advantage of the decomposition information available in the library kernel and uses either the MPI-IO or ADIOS2 backends (selected at compile time) to implement some most frequently used I/O functions for applications based on 3D Cartesian data structures. All the I/O functions have been packed in a Fortran module: :: use decomp_2d_io To write a single three-dimensional array to a file --------------------------------------------------- :: call decomp_2d_write_one(ipencil,var,directory,filename,icoarse,io_name) where ``ipencil`` describes how the data is distributed (valid values are: 1 for X-pencil; 2 for Y-pencil and 3 for Z-pencil); ``var`` is the reference to the data array, which can be either real or complex; ``directory`` is the path to where I/O should be written; ``filename`` is the name of the file to be written; ``icoarse`` indicates whether the I/O should be coarsend (valid values are: 0 for no; 1 for the ``nstat`` and 2 for the ``nvisu`` coarsenings); ``io_name`` is the name of the I/O group to be used. A more general form of the subroutine is: :: call decomp_2d_write_one(ipencil,var,directory,filename,icoarse,io_name,opt_decomp,reduce_prec,opt_deferred_writes) where the global size of the data array is described by the decomposition object ``opt_decomp`` (as discussed in the Advanced 2D Decomposition API), allowing distributed array of arbitrary size to be written. The file written would contain the 3D array in its natural ijk-order so that it can be easily post-processed (for example by a serial code). A corresponding read routine is also available. The optional ``logical`` arguments ``reduce_prec`` and ``opt_deferred_writes`` enable reduced precision writing (when enabled at compile time, MPI-IO only; default ``.true.``) and enabling ADIOS2's deferred writer mode (default ``.true.``). To write multiple three-dimensional variables into a file ......................................................... The 2DECOMP&FFT library supports two ways of writing multiple variables to a single file which may be used for check-pointing purposes, for example. The newer interface is described first and allows codes to use the ADIOS2 and MPI-IO backends, the older interface is supported for backwards compatibility. When ``decomp_2d_write_one`` is called, the ``directory`` and ``io_name`` are combined to check whether a particular output location is already opened, if not then a new file will be opened and written to - this is the "standard" use. If, however, a file is opened first then the call to ``decomp_2d_write_one`` will append to the current file, resulting in a single file with multiple fields. Once the check-pointing is complete the file can then be closed. The following describes the original interface for writing multiple variables to a file and is only supported by the MPI-IO backend. This function is very useful when creating check-point files or result files. It is in the form of: :: call decomp_2d_write_var(fh,disp,ipencil,var) where ``fh`` is a MPI-IO file handle provided by the application (file opened using MPI_FILE_OPEN); ``ipencil`` describes the distribution of the input data (valid values are: 1 for X-pencil; 2 for Y-pencil and 3 for Z-pencil); ``disp`` (meaning displacement) is a variable of kind MPI_OFFSET_KIND and of intent INOUT - it is like a pointer or file cursor tracking the location where the next chunk of data would be written. It is assumed that the data array is in default size, otherwise the function also takes a second and more general form: :: call decomp_2d_write_var(fh,disp,ipencil,var,opt_decomp) where the decomposition object ``opt_decomp`` describes the arbitrary size of the global array. To create a restart/checkpointing file, it is often necessary to save key scalar variables as well. This can be done using: :: call decomp_2d_write_scalar(fh,disp,n,var) where ``var`` is a 1D array containing n scalars of the same data type. The supported data types are: real, complex and integer. These subroutines have corresponding read routines with exactly the same set of parameters. Applications are responsible for closing the file after everything is written (using MPI_FILE_CLOSE). Again, each chunk of data in the file contains one variable stored in its natural order. One major benefit is that it becomes very easy to read the data back into the application using a different number of MPI processes. To write a 2D slice of data from a 3D variable ---------------------------------------------- :: call decomp_2d_write_plane(ipencil,var,iplane,n,filename,io_name,opt_decomp) where ``ipencil`` describes the distribution of the 3D variable ``var``; ``iplane`` defines the direction of the desired 2D slice (1 for X-plane; 2 for Y-plane and 3 for Z-plane); ``n`` specifies which plane to write out (in global coordinate system) when positive, a value ``n<=1`` will output a plane of the average along the ``iplane`` axis; ``filename`` is the name of the file to be written; and ``io_name`` is the I/O group performing the write. As before, ``opt_decomp`` is an optional parameter that can be used when ``var`` is of non-default size. To write out a 3D variable in lower resolution to a file -------------------------------------------------------- Applications using the 2D decomposition often handle quite large data sets. It may not be practical to write out everything for post-processing. The following subroutine provide a convenient way to write only a subset of the data for analysis. :: call decomp_2d_write_every(ipencil,var,iskip,jskip,kskip,filename,from1) where once again ``ipencil`` describes the distribution of the 3D variable var and ``filename`` defines the name of the file to be written. Every ``iskip``-th points of the data in X-direction, ``jskip``-th in Y-direction and every ``kskip``-th in Z-direction are to be written. Finally ``from1`` is a boolean flag. Assuming every n-th data points are to be written out, the data points have indices of: * 1,n+1,2n+1... if from1 is ``.true.`` * n,2n,3n... ff from1 is ``.false.`` Quick I/O Reference ------------------- The following table summarises the supported I/O types and data types of the subroutines: +--------------+------+-------+------+-------+---------+-------------+ | IO functions | I/O type | Data type | decomp [*]_ | + +------+-------+------+-------+---------+ + | | Read | Write | Real | Cmplx | Int | | +==============+======+=======+======+=======+=========+=============+ | _one | X | X | X | X | | X | +--------------+------+-------+------+-------+---------+-------------+ | _var | X | X | X | X | | X | +--------------+------+-------+------+-------+---------+-------------+ | _scalar | X | X | X | X | X | N/A | +--------------+------+-------+------+-------+---------+-------------+ | _plane | | X | X | X | | X | +--------------+------+-------+------+-------+---------+-------------+ | _every | | X | X | X | | | +--------------+------+-------+------+-------+---------+-------------+ .. [*] decomp refers to a decomposition object that describes an arbitrary-size global data set. ADIOS2 backend for I/O --------------------------------------- By default 2DECOMP&FFT will build with the MPI-IO backend for I/O. The alternative ADIOS2 backend can be selected at compile time, by either specifying ``-DIO_BACKEND=adios2`` during configure or by modifying the build configuration via ``ccmake``. Due to the way ADIOS2 works, there are a few changes necessary to allow codes to work with either backend interchangeably. Registering variables for I/O ............................. Registering a variable for I/O informs ADIOS2 about the variables size and type, with MPI-IO this call becomes a ``no-op``. :: subroutine decomp_2d_register_variable(io_name,varname,ipencil,icoarse,iplane,type,opt_decomp,opt_nplanes) The variable is associated with an I/O group through ``io_name``; given a name ``varname``; ``ipencil``, ``icoarse`` and ``iplane`` determine the orientation and size of the data for I/O (see previous descriptions, use ``iplane=0`` for 3D data); and ``type`` specifies the ``kind`` of the data, only ``real`` data is currently supported in ADIOS2, i.e. ``real(kind=type)``. The optional arguments ``opt_decomp`` and ``opt_nplanes`` are a decomposition object for non-standard sizes (see previous descriptions) and ``opt_nplanes`` allows controlling how many planes are written in planar output (default 1). Opening a file for reading or writing ..................................... The mode of operation for ADIOS2 is to open a file for I/O and keep this open, buffering multiple fields before performing the I/O. :: subroutine decomp_2d_open_io(io_name, io_dir, mode) The output destination ``io_dir`` is where all data will be written to from the group ``io_name``, the ``mode`` can take the values ``decomp_2d_write_mode``, ``decomp_2d_read_mode`` or ``decomp_2d_append_mode`` to write, read or append to a file, respectively. This is required by all ADIOS2 I/O, when using the MPI-IO backend I/O operations will open the file on-demand, in which case I/O is file-per-field, or if the file is explicitly opened then subsequent I/O call will be into the same file. Beginning an I/O step ..................... ADIOS2 performs I/O in "steps", this subroutine marks the beginning of a step to queue up I/O operations. :: subroutine decomp_2d_start_io(io_name,io_dir) The arguments ``io_name`` and ``io_dir`` are as described above. This is a ``no-op`` in the MPI-IO backend. Ending an I/O step .................. :: subroutine decomp_2d_end_io(io_name, io_dir) This subroutine marks the end of an I/O step and ADIOS2 can begin performing I/O operations. This is a ``no-op`` in the MPI-IO backend. Closing a file .............. :: subroutine decomp_2d_close_io(io_name, io_dir) Closes the I/O destination, must be matched with corresponding call to ``decomp_2d_open_io``. By closing the I/O ADIOS2 is forced to perform a ``flush`` operation.