Fix CVE-2025-2153 (#5795)

This PR fixes #5329. Previously, the message flags field was able to be modified such that a message that is not sharable according to the share_flags field in H5O_msg_class_t could be treated as sharable. A check has been added to make sure messages that are not sharable can't be modified so that they indicate they can be shared.

The bug was first reproduced using the fuzzer and the POC file from #5329. With this change, the heap based buffer overflow no longer occurs.

Author: glennsong09

release_docs/CHANGELOG.md

@@ -579,6 +579,12 @@ Added Fortran wrapper h5fdsubfiling_get_file_mapping_f() for the subfiling file
 
    Fixes GitHub issue #5861
 
+### Fixed security issue CVE-2025-2153
+   
+   The message flags field could be modified such that a message that is not sharable according to the share_flags field in H5O_msg_class_t can be treated as sharable. An assert has been added in H5O__msg_write_real to make sure messages that are not sharable can't be modified to shared. Additionally, the check in H5O__chunk_deserialize that catches unsharable messages being marked as sharable has been improved.
+
+   Fixes GitHub issue #5329
+
 ### Fixed security issue CVE-2025-6857
 
    An HDF5 file had a corrupted v1 B-tree that would result in a stack overflow when performing a lookup on it. This has been fixed with additional integrity checks.

src/H5Ocache.c

@@ -1400,8 +1400,8 @@ H5O__chunk_deserialize(H5O_t *oh, haddr_t addr, size_t chunk_size, const uint8_t
             else {
                 /* Check for message of unshareable class marked as "shareable"
                  */
-                if ((flags & H5O_MSG_FLAG_SHAREABLE) && H5O_msg_class_g[id] &&
-                    !(H5O_msg_class_g[id]->share_flags & H5O_SHARE_IS_SHARABLE))
+                if (((flags & H5O_MSG_FLAG_SHARED) || (flags & H5O_MSG_FLAG_SHAREABLE)) &&
+                    H5O_msg_class_g[id] && !(H5O_msg_class_g[id]->share_flags & H5O_SHARE_IS_SHARABLE))
                     HGOTO_ERROR(H5E_OHDR, H5E_CANTLOAD, FAIL,
                                 "message of unshareable class flagged as shareable");
 

src/H5Omessage.c

@@ -354,6 +354,9 @@ H5O__msg_write_real(H5F_t *f, H5O_t *oh, const H5O_msg_class_t *type, unsigned m
          */
         assert(!(mesg_flags & H5O_MSG_FLAG_DONTSHARE));
 
+        /* Sanity check to see if the type is not sharable */
+        assert(type->share_flags & H5O_SHARE_IS_SHARABLE);
+
         /* Remove the old message from the SOHM index */
         /* (It would be more efficient to try to share the message first, then
          *      delete it (avoiding thrashing the index in the case the ref.