This message means the data were compressed with an algorithm (filter in HDF5 terms) that isn't supplied with the standard HDF5 distribution.
I've been working on adding support for extra filters in hdf5filters. It's in the devel branch of Bioconductor, but you can't really use it yet as I haven't added support to rhdf5.
It'd be great if you could open an issue at https://github.com/grimbough/rhdf5 and we can work on getting this integrated. If it's possible for you to share the file that'd also be really helpful.
I have the same problem even after I installed hdf5filters. Do you have a suggestion?
Thanks!