ASAGI
1.0
a pArallel Server for Adaptive GeoInformation
|
These are minimal C, C++ and Fortran examples that load a 2-dimensional grid and print the value at (0,0). In each case the grid contains floating point values.
C example:
C++ example:
Fortran example:
ASAGI distinguishes between three different grid types:
Full storage does not automatically mean, that the full grid is stored on every CPU. If asagi::Grid::setComm() and/or asagi::Grid::setThreads() are called, the initial grid will be distributed among all nodes resp. CPUs. If the cache-grid is used and asagi::Grid::setThreads() and/or asagi::Grid::setComm() are set, ASAGI will copy the data from other NUMA domains and/or other MPI processes. Only if it is not available in another cache, the data will be fetched from the file.
ASAGI supports grids with up to MAX_DIMENSIONS
dimensions. (MAX_DIMENSIONS
is 4 by default, but can be changed during compilation of ASAGI.) The number of actual dimensions in the grid cannot be specified by calling an ASAGI function but depends on the netCDF input file.
A grid can have multiple resolutions. Each resolution is identified by a level id (level of detail). If the number of levels is not specified when creating a grid, the grid will contain only one level of detail. In this case you can also omit the level id in all other functions, since level 0 will be used by default. (C does not support default arguments or overloading, therefore omitting arguments is not possible when using the C interface.)
For grids with multiple levels asagi::Grid::open() must be called once for each level. Several levels can be stored in a single NetCDF file with different variable names. (Use asagi::Grid::setParam() to specify the variable name.) The coarsest resolution should have the level id 0. With ascending level id, the resolution should get finer. When accessing values with any get
function, the level of detail can be selected with the last argument. The function asagi::Grid::close() has to be called only once for the whole grid.
ASAGI distinguishes between actual coordinates and internal array indexes. All functions, that return a grid value, expect actual coordinates. ASAGI maps each coordinate to an array index using the coordinate variables from the NetCDF file (see section NetCDF files on how specify coordinate variables in NetCDF files). If no coordinate variable is available, the mapping is omitted. After the mapping, the coordinate is rounded to the nearest array index. ASAGI does not interpolate between array values.
The actual range of the grid can be obtained with asagi::Grid::getMin()/asagi::Grid::getMax(). They also return coordinates, not array indexes. It is erroneous to access values outside range of the grid.
The range of a dimension can be . This is the case if the size of the dimension in the netCDF file is one.
ASAGI supports cell-centered and vertex-centered grids. The value position can be switched with asagi::Grid::setParam().
All NetCDF files opened with ASAGI should respect the COARDS conventions (http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html). However, ASAGI has some further limitations:
scale_factor
and add_offset
are ignored. Besides conversion between data types, ASAGI does not modify the values. _FillValue
and missing_value
, are not supported. It is possible to open a NetCDF file by different grids or levels at the same time. This allows you, for example, to store all levels of one grid in a single NetCDF file. In this case the levels must be distinguished by the variable names.
When compiled with THREADSAFE=ON
(see section Compilation) all functions are thread-safe. However, there are some restrictions due to MPI implementations. If your MPI library is not thread-safe, you have to add the additional flag THREADSAFE_MPI=ON
which will mare sure that ASAGI does not call MPI functions from different threads at the same time. However, in this case, you are not allowed to call MPI and ASAGI functions at the same time.
Multi-thread support is required if you want to use ASAGI's NUMA functionality (see NUMA).
In addition, for ASAGI to work correctly, it has to now about all threads the application is using. Use asagi::Grid::setThreads() to set the number of threads and call asagi::Grid::open() from all threads. asagi::Grid::open() is a collective operation for all threads.
ASAGI is able to detect the NUMA domains of your node. If more than one NUMA domain is detected, ASAGI will place a cache on each NUMA domain to increase locality. You can control the NUMA detection, with the configuration parameter NUMA_COMMUNICATION
(see Parameters).
ASAGI supports two different MPI communication patterns: Via MPI remote memory access (MPI windows) or a separate communication thread. The MPI windows are used by default since they do not have any special requirement and are easy to use. However, in some MPI libraries, RMA is poorly tested and does not work well, especially with hybrid parallelization.
Therefore, you can use the communication thread. In this mode, a separate thread is required which is responsible for answering remote requests. You have to start the thread with asagi::Grid::startCommThread before any grid using the communication thread is opened. Multiple grids will share one communication thread does you must not start more than communication. However, you have to make sure that the MPI communicator for the communication thread includes all grid communicators. Once the last grid using the communication thread is closed, you should stop the additional with asagi::Grid::stopCommThread. To use the communication thread it is also necessary to have a thread-safe MPI implementation.
To disable MPI communication completely, set MPI_COMMUNICATION
to OFF
ASAGI supports several parameters for each grid:
Name | Values | Description | Grid-global (*) |
---|---|---|---|
GRID | FULL | CACHE | PASS-THROUGH | The grid type (see Grid types) | yes |
NUMA_COMMUNICATION | ON | OFF | CACHE | Enable/disable NUMA detection. CACHE can be used in combination with the "full grid". It enables NUMA detection and in addition ASAGI will look into all node local NUMA caches before activating MPI communication. (default: ON if compiled with NUMA support) | yes |
MPI_COMMUNICATION | OFF | THREAD | WINDOW | Use a communication thread or MPI RMA (windows) for MPI communication (default: WINDOW, see MPI Communication) | yes |
VALUE-POSITION | CELL-CENTERED | VERTEX-CENTERED | The value position (see Value position) | yes |
TIME-DIMENSION | int | The dimension that holds the time (default is -1 which means no time dimension exists). ASAGI treats time dimension specially. | yes |
VARIABLE | string | The variable name in the netCDf file. (default: z) | no |
BLOCK-SIZE-X | int | The size of a block in dimension X. Use a negative value to set the block size equal to the total number cells in this dimension. | no |
CACHE-SIZE | int | The size of the cache (in blocks) on each CPU. | no |
CACHE-HAND-SPREAD | int | ASAGI uses the clock algorithm to approx. LRU. This parameter specifies the difference of the 2 hands in the clock. Lower values result in a faster algorithm but a worse approximation. | no |
(*) If yes, the parameter can only be set for all levels at the same time. Set the parameter level
in asagi::Grid::setParam() to 0 to change value.
ASAGI supports several access counters to measure the throughput of the library and get information about effectiveness of the caches:
Name | Description |
---|---|
accesses | Total number of data accesses |
numa_transfers | Number of blocks transfered between CPUs |
mpi_transfers | Number of blocks transfered between processes |
file_load | Number of blocks loaded from file (after initialization) |
local_hits | Number values that where already in local NUMA domain |
node_hits | Number values that where already on the local node |
local_misses | Number of values that where not already in local memory |
THREADSAFE_COUNTER=ON
, the counters might be inaccurate.