For research purposes, I must read a lot of large hdf5 files generated by an environmental model. Right now, I open them in rstudio using the rhdf5 package. I recently noticed that the most time-limiting step was the reading of the hdf5 files and so I wanted to speed up that part of the code. Switching from rhdf5 to the old hdf5 package, the speed was considerably reduced (it goes something like ten times faster). Is there any reasonable explanation that explains such a speed difference? I would prefer to keep rhdf5 as it is still supported, so is there a way to reach better performances? As it could be due to some bad practice, please find below the two lines of code I use to open and read the hdf5 files:

With rhdf5(slow)

mymont    = h5read(h5file,'/')

and with hdf5 (fast):

mymont    = hdf5load(file=h5file,load=FALSE,verbosity=0,tidy=TRUE)

It looks like your files are quite different in structure from my 'large matrix' example. You have hundreds of small datasets, and h5read() takes a relatively long time to recurse the file and read them individually.

Here's a quick an dirty function that simplifies the reading for your file type, and performs a fair bit quicker than h5read(), although still nowhere near as fast as h5load() . It assumes that you have a flat heirachy in your h5 files and that you don't want anything extra like the attribute data.

library(rhdf5)
library(hdf5)
library(microbenchmark)

h5File <- "paracou-Q-2036-01-00-000000-g01.h5"

f0 <- function() hdf5load(file = h5File, load=FALSE, 
                          verbosity=0, tidy=TRUE)
f1 <- function() h5read(h5File, "/")

h5read_optimised <- function( h5File ) {

    dset_names <- h5ls(h5File, recursive = FALSE, datasetinfo = FALSE)$name

    fid <- H5Fopen(h5File)
    contents <- sapply(dset_names, FUN = function(fid, dset_name) {
        did <- H5Dopen(fid, name = dset_name)
        res <- H5Dread(did)
        H5Dclose(did)
        res
    }, fid = fid)
    H5Fclose(fid)

    return(contents)
}

f2 <- function() h5read_optimised(h5File)

> microbenchmark(f0(), f1(), f2(), times = 5)
Unit: milliseconds
expr        min         lq      mean    median         uq        max neval
f0()   47.07749   47.11715   48.1333   47.2934   48.24441   50.93405     5
f1() 1576.38092 1680.43496 1680.1064 1683.5432 1694.62365 1765.54923     5
f2()  540.33196  544.24837  562.4359  553.9433  577.82355  595.83262     5
> identical(f1(), f2())
[1] TRUE

The remaining time diffence is mostly spent by rhdf5 being very careful and checking everytime it is passed a file or dataset handle that the type is correct. For the single large matrix example it only does this once, and spends the rest of the time reading, but for your files it does this check hundreds of times and spends a significant portion of the runtime there.

Here's a version where all the checking is removed, and you're right down at the interface with the C functions. Almost all of this is undocumented, and it's obviously tuned to the structure of the file you shared using the default arguments, but it's pretty quick. Based on this I am wondering whether making this type checking could be optional, so some internal functions can skip it.

h5read_optimised_more <- function( h5file ) {

    dset_names <- h5ls(h5File, recursive = FALSE, datasetinfo = FALSE)$name

    fid <- H5Fopen(h5File)
    dapl <- H5Pcreate("H5P_DATASET_ACCESS")
    contents <- sapply(dset_names, FUN = function(fid, dset_name) {
        did <- .Call("_H5Dopen", fid@ID, dset_name, dapl@ID, PACKAGE='rhdf5')
        res <- .Call("_H5Dread", did, NULL, NULL, NULL, TRUE, 0L, FALSE, PACKAGE='rhdf5')
        invisible(.Call("_H5Dclose", did, PACKAGE='rhdf5'))
        return(res)
    }, fid = fid)

    H5Pclose(dapl)
    H5Fclose(fid)

    return(contents)
}

f3 <- function() h5read_optimised_more(h5File)

> microbenchmark(f0(), f1(), f2(), f3(), times = 5)
Unit: milliseconds
 expr        min         lq       mean     median         uq        max neval
 f0()   48.79590   49.02661   52.35011   51.39961   52.22597   60.30245     5
 f1() 1544.00682 1550.06203 1576.59195 1573.27627 1603.67828 1611.93634     5
 f2()  539.84307  562.46172  576.96321  566.65601  598.58858  617.26666     5
 f3()   37.99232   38.52886   39.36942   39.39166   40.29512   40.63916     5
 > identical(f1(), f3())
[1] TRUE

Interesting question, I've never really compared the various HDF5 reading packages against each other.  Since they all rely on the same underlying C library, my assumption was that read speeds would be comparable between them, and factors like compression levels, chunk size, and disk I/O would determine performance - maybe this is not the case.  Given that last update to the hdf5 package was in 2009 and it's no longer on CRAN it definitely wasn't on my radar.

That said, I did recently find some small performance bottlenecks in rhdf5, but they reduced read times by more like 25% rather than many times faster.  You can see if they help your use case by trying the version in the devel branch of Bioconductor.

As far as your commands go, they are fine if you want to read the entire file.  If you only need a subset of the data then it is almost certainly quicker to use something like h5readDataset() to extract only the parts that you need.  Similarly, if you're accessing the same file multiple times, it can be quicker to use the 'low-level' functions and open it once using H5Fopen() followed by H5Dread() on the parts you need.  That really depends what exactly you're doing - it won't be beneficial for a single read of the whole file.