Skip to content

Commit 7576460

Browse files
jhendersonHDFjhendersonHDF
committed
Add initial README file
1 parent 7d16e29 commit 7576460

File tree

1 file changed

+175
-0
lines changed

1 file changed

+175
-0
lines changed

‎README.md‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
# HDF5 Nvidia GPUDirect Storage VFD
2+
3+
## Table of Contents
4+
1. [Description](#1-Description)
5+
2. [Installation](#2-Installation)
6+
* [Prerequisites](#Prerequisites)
7+
* [Build instructions](#Build-instructions)
8+
* [CMake](#CMake-build-instructions)
9+
* [CMake options](#CMake-build-options)
10+
* [Driver options](#CMake-driver-options)
11+
3. [Usage and Testing](#3-Usage-And-Testing)
12+
* [Linked into application](#Linked-into-application)
13+
* [Dynamically loaded by FAPL](#Dynamic-load-fapl)
14+
* [Dynamically loaded by environment variable](#Dynamic-load-variable)
15+
* [Usage](#Driver-Usage)
16+
4. [Known Issues and Limitations](#4-Issues-And-Limitations)
17+
5. [More Information](#5-More-Information)
18+
19+
## 1. Description
20+
21+
The HDF5 GPUDirect Storage VFD is a Virtual File Driver (VFD) for HDF5 that can
22+
be used to interface with Nvidia's GPUDirect Storage (GDS) API. The driver is
23+
built as a plugin library that is external to HDF5.
24+
25+
## 2. Installation
26+
27+
### Prerequisites
28+
29+
To build the GDS VFD, the following libraries are required:
30+
31+
+ `libhdf5` - The [HDF5](https://www.hdfgroup.org/downloads/hdf5/) library.
32+
Minimum version required is 1.13.0. HDF5 must be compiled with
33+
support for parallel I/O if building the GDS VFD's tests. Shared
34+
libraries must be enabled if loading the GDS VFD as an HDF5 plugin.
35+
36+
+ `libcuda` - Nvidia [CUDA](https://developer.nvidia.com/about-cuda) library.
37+
Cufile support must also be available.
38+
39+
+ `libmpi` - MPI library. The GDS VFD's tests use MPI to test functionality
40+
of the driver.
41+
42+
Compiled libraries must either exist in the system's library paths or must be
43+
pointed to during the GDS VFD build process.
44+
45+
### Build instructions
46+
47+
The HDF5 GDS VFD is built using CMake.
48+
49+
#### CMake
50+
51+
CMake version 2.8.12.2 or greater is required for building the driver.
52+
53+
After obtaining the driver's source code, you can create a build directory
54+
within the source tree and run the `ccmake` or `cmake` command from it:
55+
56+
cd vfd-gds-X
57+
mkdir build
58+
cd build
59+
ccmake ..
60+
61+
If using `ccmake`, type `'c'` multiple times and choose suitable options or if
62+
using `cmake`, pass these options with `-D`. Some of these options may be needed
63+
if, for example, the required components mentioned previously are not located in
64+
default paths.
65+
66+
Setting include directory and library paths may require you to toggle to
67+
the advanced mode by typing `'t'`. Once you are done and do not see any
68+
errors, type `'g'` to generate makefiles. Once you exit the CMake
69+
configuration screen and are ready to build the targets, do:
70+
71+
make
72+
73+
Verbose build output can be generated by appending `VERBOSE=1` to the
74+
`make` command.
75+
76+
Assuming that the `CMAKE_INSTALL_PREFIX` has been set and that you have
77+
write permissions to the destination directory, you can install the driver
78+
by simply doing:
79+
80+
make install
81+
82+
##### CMake options
83+
84+
* `CMAKE_INSTALL_PREFIX` - This option controls the install directory that the
85+
resulting output files are written to. The default value is `/usr/local`.
86+
* `CMAKE_BUILD_TYPE` - This option controls the type of build used for the VFD.
87+
Valid values are `Release`, `Debug`, `RelWithDebInfo`, `MinSizeRel`; the default
88+
build type is `RelWithDebInfo`.
89+
90+
##### Driver options
91+
92+
* `BUILD_EXAMPLES` - This option is used to enable/disable building of the
93+
GDS VFD's HDF5 example programs. The default value is `ON`.
94+
* `BUILD_TESTING` - This option is used to enable/disable building of the
95+
GDS VFD's tests. The default value is `ON`.
96+
* `HDF5_VFD_GDS_CUFILE_DIR` - This option controls the directory used when
97+
searching for the NVIDIA GPUDirect Storage cuFile library. This option may
98+
need to be set if CMake cannot automatically resolve the path.
99+
* `HDF5_VFD_GDS_CUFILE_LIB` - This option controls the library used for
100+
NVIDIA GPUDirect Storage cuFile support. This option may need to be set if
101+
CMake cannot automatically resolve it.
102+
* `HDF5_C_COMPILER_EXECUTABLE` - This option controls the HDF5 compiler
103+
wrapper script used by the VFD build process. It should be set to the full
104+
path to the HDF5 compiler wrapper (usually `bin/h5cc`), including the name
105+
of the wrapper script. The following two options may also need to be set.
106+
* `HDF5_C_LIBRARY_hdf5` - This option controls the HDF5 library used by the
107+
VFD build process. It should be set to the full path to the HDF5 library,
108+
including the library's name (e.g., `/path/libhdf5.so`). Used in conjunction
109+
with the `HDF5_C_INCLUDE_DIR` option.
110+
* `HDF5_C_INCLUDE_DIR` - This option controls the HDF5 include directory used
111+
by the VFD build process. Used in conjunction with the `HDF5_C_LIBRARY_hdf5`
112+
variable.
113+
* `MPI_C_COMPILER` - This option controls the MPI C compiler used by the VFD
114+
build process. It should be set to the full path to the MPI C compiler,
115+
including the name of the executable.
116+
117+
## 3. Usage and Testing
118+
119+
To use the HDF5 GDS VFD in an HDF5 application, the driver can either be linked
120+
into the application, or it can be dynamically loaded as a plugin. If dynamically
121+
loading the GDS VFD, users should ensure that the `HDF5_PLUGIN_PATH` environment
122+
variable points to the directory containing the built VFD library if the VFD has
123+
been installed to a non-standard location.
124+
125+
### Linked into application
126+
127+
To link the GDS VFD into an HDF5 application, the application should include the
128+
`H5FDgds.h` header that gets installed on the system and should link the installed
129+
VFD library (`libhdf5_vfd_gds.so`, or similar) into the application. Once this has
130+
been done, GDS VFD access can be setup by calling `H5Pset_fapl_gds(...)` on a FAPL
131+
within the HDF5 application. Refer to the HDF5 example programs under the `examples`
132+
folder to see how this is done.
133+
134+
### Dynamically loaded by FAPL
135+
136+
To explicitly load the GDS VFD inside an HDF5 application, a call to the
137+
`H5Pset_driver_by_name(...)` routine should be made to setup GDS VFD access on a
138+
FAPL. This will cause HDF5 to load the VFD as a plugin and set the VFD on the
139+
given FAPL. The string "gds" should be given for the `driver_name` parameter. NULL
140+
should be given for the `driver_config` parameter, as the driver does not currently
141+
accept configuration strings.
142+
143+
### Dynamically loaded by environment variable
144+
145+
To implicitly load the GDS VFD inside an HDF5 application, the `HDF5_DRIVER`
146+
environment variable may be set to the string "gds". During library initialization,
147+
HDF5 will check this environment variable, load the GDS VFD as a plugin and set the
148+
VFD as the default file driver on File Access Property Lists. Therefore, any file
149+
access that uses `H5P_DEFAULT` for its FAPL, or which uses a FAPL that hasn't had a
150+
specific VFD set on it, will automatically use the GDS VFD for file access.
151+
152+
### Usage
153+
154+
Users of the GDS VFD should be aware that using the VFD in a normal, unmodified HDF5
155+
application will cause the VFD to behave similarly to the sec2 VFD; that is, the VFD
156+
will make use of POSIX I/O API routines to do file I/O. To make proper use of
157+
GPUDirect storage-capable devices with the VFD, the HDF5 application should make use
158+
of CUDA memory management API routines (e.g., cudaMalloc) to allocate memory on the
159+
device. Then, the device-allocated memory buffer should be passed to `H5Dread`/`H5Dwrite`.
160+
For examples of this, refer to the HDF5 programs under the `examples` folder.
161+
162+
## 4. Known Issues and Limitations
163+
164+
* For chunked datasets, HDF5 may attempt to cache chunks in its chunk cache
165+
under certain circumstances, which can incur overhead when copying between
166+
CUDA device memory and host memory. This caching mechanism cannot currently be
167+
properly disabled; therefore, performance may suffer for chunked datasets if:
168+
- filters are applied to the dataset's chunks
169+
- fill values need to be written to chunks
170+
171+
## 5. More Information
172+
173+
[GPUDirect Storage](https://developer.nvidia.com/blog/gpudirect-storage/)
174+
175+
[HDF5 VFD Plugins RFC](https://github.com/HDFGroup/hdf5doc/blob/master/RFCs/HDF5_Library/VFL_DriverPlugins/RFC__A_Plugin_Interface_for_HDF5_Virtual_File_Drivers.pdf)

0 commit comments

Comments
 (0)