HDF5 File Image Operations Dana Robinson The HDF

slide1. HDF5 File Image Operations Dana Robinson
The HDF Group Efficient Use of HDF5 With High Data Rate X-Ray Detectors
Paul Scherrer Institut<br>
slide2. Purpose Allow users to work with files like how they do on disk.

No disk I/O when file images are opened, created, read from, or written to.

Faster access to data.

Need to be careful about minimizing your memory footprint when working with large files.<br>
slide3. Use in pipelines Processing 1 HDF5 file 1) Processing step 1 opens the HDF5 file with the core VFD. HDF5 file<br>
slide4. Process 2 Use in pipelines Process 1 2) Buffer is sent to processing step 2.

This saves writing the file to the filesystem. HDF5 file IPC<br>
slide5. Useful for MPI Process 0 Process 1 Process n config file 1) Process 0 reads configuration file. 2) File image is broadcast to other processes
to open in memory. Process 2<br>
slide6. New API Functions H5Pget/set_file_image

H5Pget/set_file_image_callbacks

H5Fget_file_image

H5LTopen_file_image (high-level convenience function)<br>
slide7. H5Pget/set_file_image herr_t
H5Pget/set_file_image(hid_t fapl_id,
void *buffer, size_t buf_len)

Requires allocating a buffer, which is passed to the function.

Designed for the core VFD but most VFDs can be modified to support using an initial image file.

The buffer is copied in/out of the HDF5 library, not passed by reference!
We will discuss ways around this…<br>
slide8. Digression: Property Lists Used frequently in the HDF5 library.

Allow us to be flexible in passing parameters to functions without breaking the API.

Use "pass by value" semantics, not "pass by reference"

- Designed for moving small amounts of data around.

- Greatly simplifies copying and deletion of property lists.<br>
slide9. Digression: Property Lists Unfortunately, passing very large buffers by value can bring a severe performance penalty.

We have a few ways around this:

- Use the H5LTopen_image_file call (discussed later)

- Ignore the problem if the overhead is low (small files)

- Use file image callbacks to implement call-by-reference
semantics (discussed later)<br>
slide10. H5Fget_file_image ssize_t
H5Fget_file_image(hid_t fapl_id,
void *buffer, size_t buf_len)

Provides a simple way to retrieve a copy of the image of an open file.

You provide the buffer and the size.

You can call the function with a null buffer pointer to get the current file size.

Can be used with files opened with the SEC2, STDIO, and core VFDs.<br>
slide11. H5P_get/set_file_image_callbacks herr_t
H5Pget/set_file_image_callbacks(hid_t fapl_id,
H5_file_image_callbacks_t *callbacks_ptr)

H5_file_image_callbacks_t is a struct containing function pointers which can be invoked on:
image re/allocation image free
image copy

It also contains a void pointer for user-specific data and function pointers which can be invoked on:
user data copy user data free<br>
slide12. H5P_get/set_file_image_callbacks The purpose of the callback functions are twofold:

1) Allow the user more careful control over the image in memory when resources are scarce (e.g.: when handling large files).

2) Allow the user to implement pass-by-reference semantics for the buffer.<br>
slide13. H5_file_image_callbacks_t typedef struct {
void *(image_malloc)(size_t size,
H5_file_image_op_t file_image_op, void *udata),
void *(image_memcpy)(void *dest, void *src,
size_t size, H5_file_image_op_t file_image_op,
void *udata),
void *(image_realloc)(void *ptr, size_t size,
H5_file_image_op_t file_image_op, void *udata),
herr_t *(image_free)(void *ptr,
H5_file_image_op_t file_image_op, void *udata),
void *(udata_copy)(void *udata),
herr_t *(udata_free)(void *udata),
void *udata;
} H5_file_image_callbacks_t;<br>
slide14. H5P_get/set_file_image_callbacks void *(image_malloc)(size_t size,
H5_file_image_op_t file_image_op, void *udata),

The file_image_op parameter indicates which operation is taking place when the function was invoked.

The image_malloc, realloc, free, and memcpy functions must have the same semantics as their C counterparts.

Note that the memcpy and free calls can indicate failure in their return values, unlike their C counterparts.<br>
slide15. H5_file_image_op_t typedef enum {
H5_FILE_IMAGE_OP_PROPERTY_LIST_SET,
H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY,
H5_FILE_IMAGE_OP_PROPERTY_LIST_GET,
H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE,
H5_FILE_IMAGE_OP_FILE_OPEN,
H5_FILE_IMAGE_OP_FILE_RESIZE,
H5_FILE_IMAGE_OP_FILE_CLOSE,
} H5_file_image_op_t; These values indicate the operation in progress when the callback was invoked.<br>
slide16. Warning!!! Creating a full set of callback functions to control the behavior of the file image is likely to be a difficult process!

The high-level function H5LTopen_file_image (discussed later) should be suitable in most cases.<br>
slide17. H5LTopen_file_image hid_t
H5LTopen_file_image (void *buffer,
size_t buf_len, unsigned flags)

High-level convenience function to simplify buffer use.

Allows the library to take control of the buffer with reasonable default behavior.

Suitable for most uses.<br>
slide18. H5LTopen_file_image Bitwise flags:

H5LT_FILE_IMAGE_OPEN_RW
Open the image file with read/write permissions.

H5LT_FILE_IMAGE_DONT_COPY
Pass the buffer by reference instead of by value.
The library will call free on the buffer.

H5LT_FILE_IMAGE_DONT_RELEASE
Do not free the image when the library is done with it.
Also prohibits resizing/reallocating the buffer.<br>
slide19. Reading an in-memory image <allocate and initialize the buffer>
<allocate the fapl>
<set fapl to use the core VFD>

H5Pset_file_image(fapl_id, buffer, buf_len);

<discard buffer any time after this point>

<open file>

<discard fapl any time after this point>
<read/write file as desired, close><br>
slide20. H5LTopen_file_image example <allocate and initialize the buffer>

hid_t file_id;
unsigned flags = H5LT_FILE_IMAGE_DONT_COPY;

file_id = H5LTopen_file_image(buffer, buf_len, flags);

<read/write file as desired, close><br>
slide21. Status Complete. Appeared in HDF5 1.8.9 (May 2012).

Java and Fortran equivalents are not available at this time.<br>