You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support for HSDS/REST VOL in HighFive – interest in upstreaming? #39
Hi folks,
We're a company building large-scale physics simulators, and we've been happily using HighFive for C++ HDF5 bindings. As we port parts of our stack to an online compute platform, we've added support for HSDS via the REST VOL.
To enable this, we’ve forked HighFive and made some patches here to get things working. Before we go further, we wanted to ask: would you be interested in incorporating support for REST VOL in the official HighFive releases?
If so I could polish the PR (e.g. setting up CI with REST VOL), improve some features (like containers of containers of ... std::string's), and ensure everything aligns with your coding conventions and contribution guidelines.
There aren't that many options out there for HDF5 C++ bindings + REST VOL support so I thought this could be a useful addition for the community. Let us know your thoughts!
Cheers.
It sounds like a nice feature and I'm glad you got it working.
Three comments (to start):
There's a lot of #ifdef happening. Does this mean that the choice of using REST VOL must be made at compile time? If yes, would this prevent applications from reading both regular HighFive files and HSDS files? This will then lead to questions like are the limitations imposed on us by HDF5 or are they a result of the implementation? The former means we might need to use inheritence or something pimpl like to be able to create HSDS File, DataSet, etc. The latter would likely require to keep refining until the #ifdefs vanish.
Why are strings so problematic?
As a preview of the types of things that I would like to have differently: The files h5?_wrapper contain an extremely simplistic, mechanical and dumb wrapper of the C-API. They have one task: call the HDF5 function with a similar name and check the error code. Nothing else. I don't want to pollute this layer with complex logic. This makes it easy to see when we're not checking an error code, and makes it easier to review if we're checking the correct condition.
I seems like a worthwhile feature, but I'm skeptical if it's close to mergeable.
Thanks for the feedback @1uc ; just saw this now as it appears the notification got lost in the spam.
I seems like a worthwhile feature, but I'm skeptical if it's close to mergeable.
100% agree on both counts. That’s why I’m reaching out, I’d like to understand what direction would make this contribution mergeable from your point of view.
HDF5 itself requires the VOL connector to be configured either via environment variables or through FAPL property lists before opening a file. Once a file is open, the VOL is effectively opaque to HighFive's abstraction layer. In our current setup, we went with the simpler route: assume the environment is correctly configured (e.g., via HDF5_PLUGIN_PATH, HDF5_VOL_CONNECTOR, etc.), and that users avoid relative file paths. In this case, things “just work” with REST VOL, albeit with some limitations. You can see how we configure this in .devcontainer/Dockerfile. That said, we could certainly move toward a cleaner abstraction - for instance, using FAPLs explicitly or isolating the backend differences using polymorphism or traits. I fully agree that the current #ifdef sprawl is not sustainable long-term and I’m open to refactor it if you think a particular design direction makes more sense.
I ran into trouble with variable-length strings via the C interface - HSDS doesn’t seem to handle them cleanly. So, as a workaround, I converted std::string and std::vector<std::string> to fixed-length strings internally. This is hacky so will have to be revisited for sure, but it got me past the std::string and std::vector<std::string> tests.
A few C-API primitives simply aren’t supported by HSDS (e.g., H5Dget_offset). In such cases, I threw exceptions directly from the h5*_wrapper layer. Likewise, for functions like H5Ldelete, which behave differently under HSDS (e.g., returning 0 instead of -1 if a link doesn’t exist), I added a call to h5l_exists to preserve expected behavior. In this case, it returns 0 instead of -1 if the link does not exists (which wouldn't throw when it should). I can move all such logic higher up the stack, or refactor this however you see fit.
The current patch is rough, it was built to explore feasibility and unblock our development. But I think REST VOL support in HighFive could be valuable and I’d be glad to work toward something that’s clean and upstream-worthy. Would it make sense to start by formalizing the compile-time option + environment setup approach, and build from there?
to make sure that HighFive with HSDS can still read regular files,
unless impossible, the ifdefs need to be eliminated.
The reason I think we need both is that HighFive is almost popular enough that a very unlucky user might be dependent on two libraries that themselves use HighFive. Having two versions of HighFive that use the same symbols in incompatible ways, seems like a way to create and unsolvable case of dependency hell.
I've too little experience (i.e. heard one short presentation) with HSDS to be sure, but the rest of HDF5, works through setting property lists, and calling well-defined APIs. For example, one configures the file to have properties, X, Y and Z. But then still calls H5Dread (independent of what X, Y and Z were). It's unfortunate (and surprising) that HSDS support is this invasive. One example is the H5Dget_offset, while I can vaguely understand why there can't be an offset, it's surprising that they don't simply return HADDR_UNDEF (the error code). If they did HighFive would convert the error code to an exception. Then if someone calls H5Dget_offset in a context it's not permitted they get a runtime error (no special code or ifdefs required).
Some of the arguments sound like HSDS is buggy. Which given how new it is, seems not entirely impossible.
@vincentmr on the one hand I'm very happy to see interest in HighFive and happy to have other developers involved. At the same time, I really don't want to get into a situation where you've worked for weeks of months on polishing the PR; and I feel I have to reject it because I'm concerned it would break for other users or because it would add to much "tech debt".
I think the best way forward that protects you from wasting time, is if you continue using and refining the fork; but you do so as and when you need it for your company or otherwise suits you. It'll either get cleaner as you go; messier (because HSDS is even more invasive than thought); or just stay the way it is. If it naturally reaches a cleaner state, submit it again :) If you see that there's 10 or 20 thumbs ups or hearts on your issue we know there's real interest and we can revisit it more urgently. We might get lucky and by the time there's real interest in HSDS with HighFive the quirky behaviour you saw in HSDS will have sorted itself out.
The reason I think we need both is that HighFive is almost popular enough that a very unlucky user might be dependent on two libraries that themselves use HighFive.
Fair point.
One example is the H5Dget_offset, while I can vaguely understand why there can't be an offset, it's surprising that they don't simply return HADDR_UNDEF (the error code).
In this case they do return an error (so no #ifdef required), so this may have been a bad example to bring up on my part. A better example of an invasive feature is REST VOL does not support H5Iget_file_id() which is used by HighFive to get the file ID when creating a PathTraits object. I guess we might pass the ID around in the constructor from the parent objects, but that would be a bit of a pain.
Some of the arguments sound like HSDS is buggy. Which given how new it is, seems not entirely impossible.
That is also my impression. To be fair, I think this is more a REST VOL issue than HSDS.
@vincentmr on the one hand I'm very happy to see interest in HighFive and happy to have other developers involved. At the same time, I really don't want to get into a situation where you've worked for weeks of months on polishing the PR; and I feel I have to reject it because I'm concerned it would break for other users or because it would add to much "tech debt".
Make sense, thanks for your concern.
I think the best way forward that protects you from wasting time, is if you continue using and refining the fork; but you do so as and when you need it for your company or otherwise suits you. [...]
That sounds like the best course of action for now, indeed. Let's do that. In the end, I thought I would let you know there is some interest and some work done toward supporting REST VOL. If someone else is interested, they can reach out and we can trade ideas and share effort.
One last thing, I was wondering if someone more familiar with the library than I am would know a nice trick to convert all std::string objects in a container (or container of containers, etc.) into padded fixed-length strings? I'd like to replace such hacks and I thought there might be a utility or something that does it.
This reply is poorly structured: first it rambles, then it contains a potential solution (second block of code); only to continue rambling some more; but hopefully some of it is useful to youl
Sorry, I keep forgetting to answer the question.
I don't know of a nice trick to convert a container of std::string objects into a container of padded fixed-length strings.
Somehow I doubt one exists for the following reason. HDF5 expects the fixed-length and variable-length strings to be laid out in memory very differently. Variable length strings, are passed to HDF5 as an array of char*, e.g. char**. While fixed-length strings are passed to HDF5 as an array of chars with the understanding that a new string begins every N bytes, e.g. char*. Therefore, I believe we have no choice but to copy. So no trick.
The place in HighFive where this copy happens is the StringBuffer:
While it might count as a trick, it's most definitely not nice. It interacts with the other gnarly part of HighFive: the "Inspector". Essentially, the game is that the Inspector flattens containers into C-style arrays; while the String/Shallow/Copy buffer provides the memory for the C-style "flat" array. The StringBuffer additionally sends provides the Inspector with a proxy object that so that the can copy one string to another string without needing to know how it's stored in the C-style array. This way std::string can be serialized into a buffer suitable for fixed-length (of any flavour) or variable-length strings.
For your purposes, maybe you can demand that your applications don't write (and then never have to read) variable-length strings. You'd do this by specifying the string type, instead of letting HighFive deduce it for you. Here's an example:
// When reading into an `std::string` it doesn't matter if the HDF5 datatype
// is fixed or variable length. HighFive will internally read into a buffer
// and then write to the final destination.
auto from_variable = file.getDataSet("single_variable").read<std::string>();
auto from_fixed = file.getDataSet("single_fixed_nullterm").read<std::string>();
WARNING: HDF5 "null-terminated strings" are not required to contain a null-character, i.e. they may, or may not be null-terminated. HighFive deals with this, hence you might notice that a N-byte fixed-length string marked as null-terminated in HDF5, results in an std::string of size N and requires N+1 bytes of storage (remember, std::string::size doesn't include the null-charater). HighFive will refuse to write null-terminated fixed-length strings without a \0, but programs that don't use HighFive can easily create files that contain such strings.
If you do so, you can remove the new code that does the conversion and replace it with a check that throws if it sees variable-length strings and HSDS. After that you'd be a big step closer to upstreamable, because now there's only one problem left: remove the #ifdefs. At least one wont spend time fixing one problem simply to learn that one can't solve the other.
Activity
1uc commentedon Jun 3, 2025
It sounds like a nice feature and I'm glad you got it working.
Three comments (to start):
There's a lot of
#ifdefhappening. Does this mean that the choice of using REST VOL must be made at compile time? If yes, would this prevent applications from reading both regular HighFive files and HSDS files? This will then lead to questions like are the limitations imposed on us by HDF5 or are they a result of the implementation? The former means we might need to use inheritence or something pimpl like to be able to create HSDS File, DataSet, etc. The latter would likely require to keep refining until the#ifdefs vanish.Why are strings so problematic?
As a preview of the types of things that I would like to have differently: The files
h5?_wrappercontain an extremely simplistic, mechanical and dumb wrapper of the C-API. They have one task: call the HDF5 function with a similar name and check the error code. Nothing else. I don't want to pollute this layer with complex logic. This makes it easy to see when we're not checking an error code, and makes it easier to review if we're checking the correct condition.I seems like a worthwhile feature, but I'm skeptical if it's close to mergeable.
vincentmr commentedon Jun 5, 2025
Thanks for the feedback @1uc ; just saw this now as it appears the notification got lost in the spam.
100% agree on both counts. That’s why I’m reaching out, I’d like to understand what direction would make this contribution mergeable from your point of view.
.devcontainer/Dockerfile. That said, we could certainly move toward a cleaner abstraction - for instance, using FAPLs explicitly or isolating the backend differences using polymorphism or traits. I fully agree that the current#ifdefsprawl is not sustainable long-term and I’m open to refactor it if you think a particular design direction makes more sense.std::stringandstd::vector<std::string>to fixed-length strings internally. This is hacky so will have to be revisited for sure, but it got me past thestd::stringandstd::vector<std::string>tests.H5Dget_offset). In such cases, I threw exceptions directly from theh5*_wrapperlayer. Likewise, for functions likeH5Ldelete, which behave differently under HSDS (e.g., returning 0 instead of -1 if a link doesn’t exist), I added a call toh5l_existsto preserve expected behavior. In this case, it returns0instead of-1if the link does not exists (which wouldn't throw when it should). I can move all such logic higher up the stack, or refactor this however you see fit.The current patch is rough, it was built to explore feasibility and unblock our development. But I think REST VOL support in HighFive could be valuable and I’d be glad to work toward something that’s clean and upstream-worthy. Would it make sense to start by formalizing the compile-time option + environment setup approach, and build from there?
1uc commentedon Jun 9, 2025
I think the most important aspect would be:
The reason I think we need both is that HighFive is almost popular enough that a very unlucky user might be dependent on two libraries that themselves use HighFive. Having two versions of HighFive that use the same symbols in incompatible ways, seems like a way to create and unsolvable case of dependency hell.
I've too little experience (i.e. heard one short presentation) with HSDS to be sure, but the rest of HDF5, works through setting property lists, and calling well-defined APIs. For example, one configures the file to have properties, X, Y and Z. But then still calls
H5Dread(independent of what X, Y and Z were). It's unfortunate (and surprising) that HSDS support is this invasive. One example is theH5Dget_offset, while I can vaguely understand why there can't be an offset, it's surprising that they don't simply returnHADDR_UNDEF(the error code). If they did HighFive would convert the error code to an exception. Then if someone callsH5Dget_offsetin a context it's not permitted they get a runtime error (no special code or ifdefs required).Some of the arguments sound like HSDS is buggy. Which given how new it is, seems not entirely impossible.
@vincentmr on the one hand I'm very happy to see interest in HighFive and happy to have other developers involved. At the same time, I really don't want to get into a situation where you've worked for weeks of months on polishing the PR; and I feel I have to reject it because I'm concerned it would break for other users or because it would add to much "tech debt".
I think the best way forward that protects you from wasting time, is if you continue using and refining the fork; but you do so as and when you need it for your company or otherwise suits you. It'll either get cleaner as you go; messier (because HSDS is even more invasive than thought); or just stay the way it is. If it naturally reaches a cleaner state, submit it again :) If you see that there's 10 or 20 thumbs ups or hearts on your issue we know there's real interest and we can revisit it more urgently. We might get lucky and by the time there's real interest in HSDS with HighFive the quirky behaviour you saw in HSDS will have sorted itself out.
vincentmr commentedon Jun 12, 2025
Fair point.
In this case they do return an error (so no
#ifdefrequired), so this may have been a bad example to bring up on my part. A better example of an invasive feature is REST VOL does not supportH5Iget_file_id()which is used by HighFive to get the file ID when creating a PathTraits object. I guess we might pass the ID around in the constructor from the parent objects, but that would be a bit of a pain.That is also my impression. To be fair, I think this is more a REST VOL issue than HSDS.
Make sense, thanks for your concern.
That sounds like the best course of action for now, indeed. Let's do that. In the end, I thought I would let you know there is some interest and some work done toward supporting REST VOL. If someone else is interested, they can reach out and we can trade ideas and share effort.
One last thing, I was wondering if someone more familiar with the library than I am would know a nice trick to convert all std::string objects in a container (or container of containers, etc.) into padded fixed-length strings? I'd like to replace such hacks and I thought there might be a utility or something that does it.
1uc commentedon Jun 18, 2025
This reply is poorly structured: first it rambles, then it contains a potential solution (second block of code); only to continue rambling some more; but hopefully some of it is useful to youl
Sorry, I keep forgetting to answer the question.
I don't know of a nice trick to convert a container of std::string objects into a container of padded fixed-length strings.
Somehow I doubt one exists for the following reason. HDF5 expects the fixed-length and variable-length strings to be laid out in memory very differently. Variable length strings, are passed to HDF5 as an array of
char*, e.g.char**. While fixed-length strings are passed to HDF5 as an array ofchars with the understanding that a new string begins every N bytes, e.g.char*. Therefore, I believe we have no choice but to copy. So no trick.The place in HighFive where this copy happens is the StringBuffer:
highfive/include/highfive/bits/H5Converter_misc.hpp
Line 165 in 240c07e
While it might count as a trick, it's most definitely not nice. It interacts with the other gnarly part of HighFive: the "Inspector". Essentially, the game is that the Inspector flattens containers into C-style arrays; while the String/Shallow/Copy buffer provides the memory for the C-style "flat" array. The StringBuffer additionally sends provides the Inspector with a proxy object that so that the can copy one string to another string without needing to know how it's stored in the C-style array. This way
std::stringcan be serialized into a buffer suitable for fixed-length (of any flavour) or variable-length strings.For your purposes, maybe you can demand that your applications don't write (and then never have to read) variable-length strings. You'd do this by specifying the string type, instead of letting HighFive deduce it for you. Here's an example:
highfive/src/examples/read_write_std_strings.cpp
Lines 42 to 62 in 240c07e
WARNING: HDF5 "null-terminated strings" are not required to contain a null-character, i.e. they may, or may not be null-terminated. HighFive deals with this, hence you might notice that a N-byte fixed-length string marked as null-terminated in HDF5, results in an std::string of size
Nand requiresN+1bytes of storage (remember, std::string::size doesn't include the null-charater). HighFive will refuse to write null-terminated fixed-length strings without a\0, but programs that don't use HighFive can easily create files that contain such strings.If you do so, you can remove the new code that does the conversion and replace it with a check that throws if it sees variable-length strings and HSDS. After that you'd be a big step closer to upstreamable, because now there's only one problem left: remove the
#ifdefs. At least one wont spend time fixing one problem simply to learn that one can't solve the other.1uc commentedon Jun 18, 2025
I'd like this open, because it's a nice idea; and that way I might remember to check back later.
vincentmr commentedon Jun 20, 2025
Yes, I'll consider that.
Thanks for the tips and advice. I've been pulled to other fires, but I'll revisit the PR in light of your suggestions in a couple weeks.