You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
|[Chunking in HDF5](advanced_topics/chunking_in_hdf5.html)| Provides detailed information regarding chunking in HDF5 |
13
-
|[HDF5 File Image Operations (PDF)](https://docs.hdfgroup.org/hdf5/rfc/HDF5FileImageOperations.pdf)| Describes how to work with HDF5 files in memory. Disk I/O is not required when file images are opened, created, read from, or written to |
14
-
|[Copying Committed Datatypes with H5Ocopy (PDF)](advanced_topics/CopyingCommittedDatatypesWithH5Ocopy.pdf)| Describes how to copy to another file a dataset that uses a committed datatype or an object with an attribute that uses a committed datatype so that the committed datatype in the destination file can be used by multiple objects|
|[Enabling a Strict Consistency Semantics Model in Parallel HDF5 (PDF)](https://docs.hdfgroup.org/hdf5/rfc/RFC%20PHDF5%20Consistency%20Semantics%20MC%20120328.docx.pdf)| Describes how to improve file access consistency semantics with the Parallel HDF5 library |
17
-
|[File Space Management](advanced_topics/FileSpaceManagement.html)| Describes mechanisms to manage space in an HDF5 file |
18
-
|[Fine-tuning the Metadata Cache](advanced_topics/FineTuningMetadataCache.html)| Describes how to provide better control of the metadata cache |
19
-
|[Freeing Memory Allocated by the HDF5 Library (PDF)](https://docs.hdfgroup.org/hdf5/rfc/RFC%20H5free_memory%20v2.pdf)| Describes how inconsistent memory management can cause heap corruption or resource leaks and possible solutions |
20
-
|[HDF5 Data Flow Pipeline for H5Dread](advanced_topics/data_flow_pline_H5Dread.html)| Describes data flow when reading raw data from an HDF5 dataset |
21
-
|[HDF5 Metadata](advanced_topics/)| Provides a comprehensive overview of the types of metadata used in HDF5 |
22
-
|[Introduction to Single-Writer_Multiple-Reader (SWMR)](advanced_topics/intro_SWMR.html)| Enables writing to a file while multiple readers are accessing it |
23
-
|[Introduction to the Virtual Dataset - VDS](advanced_topics/intro_VDS.html)| Describes how to present data stored in several HDF5 datasets and files as a single dataset and access the data using HDF5 APIs |
24
-
|[Modified Region Writes (PDF)](advanced_topics/ModifiedRegionWrites.pdf)| Describes how to set write operations for in-memory files so that only modified regions are written to storage. Available when the Core (Memory) VFD is used |
25
-
|[Page Buffering (PDF)](https://docs.hdfgroup.org/hdf5/rfc/RFC-Page_Buffering.pdf)| Describes how to reduce the number of small accesses in HDF5 by buffering metadata and raw data accesses |
12
+
|[Chunking in HDF5](https://support.hdfgroup.org/releases/hdf5/documentation/advanced_topics/chunking_in_hdf5.html)| Provides detailed information regarding chunking in HDF5 |
13
+
|[HDF5 File Image Operations (PDF)](https://support.hdfgroup.org/releases/hdf5/documentation/rfc/HDF5FileImageOperations.pdf)| Describes how to work with HDF5 files in memory. Disk I/O is not required when file images are opened, created, read from, or written to |
14
+
|[Copying Committed Datatypes with H5Ocopy (PDF)](https://support.hdfgroup.org/releases/hdf5/documentation/advanced_topics/CopyingCommittedDatatypesWithH5Ocopy.pdf)| Describes how to copy to another file a dataset that uses a committed datatype or an object with an attribute that uses a committed datatype so that the committed datatype in the destination file can be used by multiple objects|
|[Enabling a Strict Consistency Semantics Model in Parallel HDF5 (PDF)](https://support.hdfgroup.org/releases/hdf5/documentation/rfc/RFC%20PHDF5%20Consistency%20Semantics%20MC%20120328.docx.pdf)| Describes how to improve file access consistency semantics with the Parallel HDF5 library |
17
+
|[File Space Management](https://support.hdfgroup.org/releases/hdf5/documentation/advanced_topics/FileSpaceManagement.html)| Describes mechanisms to manage space in an HDF5 file |
18
+
|[Fine-tuning the Metadata Cache](https://support.hdfgroup.org/releases/hdf5/documentation/advanced_topics/FineTuningMetadataCache.html)| Describes how to provide better control of the metadata cache |
19
+
|[Freeing Memory Allocated by the HDF5 Library (PDF)](https://support.hdfgroup.org/releases/hdf5/documentation/rfc/RFC%20H5free_memory%20v2.pdf)| Describes how inconsistent memory management can cause heap corruption or resource leaks and possible solutions |
20
+
|[HDF5 Data Flow Pipeline for H5Dread](https://support.hdfgroup.org/releases/hdf5/documentation/advanced_topics/data_flow_pline_H5Dread.html)| Describes data flow when reading raw data from an HDF5 dataset |
21
+
|[HDF5 Metadata](https://support.hdfgroup.org/releases/hdf5/documentation/advanced_topics/)| Provides a comprehensive overview of the types of metadata used in HDF5 |
22
+
|[Introduction to Single-Writer_Multiple-Reader (SWMR)](https://support.hdfgroup.org/releases/hdf5/documentation/advanced_topics/intro_SWMR.html)| Enables writing to a file while multiple readers are accessing it |
23
+
|[Introduction to the Virtual Dataset - VDS](https://support.hdfgroup.org/releases/hdf5/documentation/advanced_topics/intro_VDS.html)| Describes how to present data stored in several HDF5 datasets and files as a single dataset and access the data using HDF5 APIs |
24
+
|[Modified Region Writes (PDF)](https://support.hdfgroup.org/releases/hdf5/documentation/advanced_topics/ModifiedRegionWrites.pdf)| Describes how to set write operations for in-memory files so that only modified regions are written to storage. Available when the Core (Memory) VFD is used |
25
+
|[Page Buffering (PDF)](https://support.hdfgroup.org/releases/hdf5/documentation/rfc/RFC-Page_Buffering.pdf)| Describes how to reduce the number of small accesses in HDF5 by buffering metadata and raw data accesses |
26
26
|[Partial Edge Chunks]()| Describes how to control the filtering of partial edge chunks to improve performance when extending datasets |
27
27
|[Thread-safe HDF5]()| Information on thread-safety and concurrent access in HDF5 |
28
-
|[Using the Direct Chunk Write Function (PDF)](https://docs.hdfgroup.org/hdf5/rfc/DECTRIS%20Integration%20RFC%202012-11-29.pdf)| Describes another way that chunks can be written to datasets |
29
-
|[Virtual Object Layer](https://docs.hdfgroup.org/hdf5/develop/_h5_v_l__u_g.html)| An abstraction layer within the HDF5 library that enables different methods for accessing data and objects that conform to the HDF5 data model |
28
+
|[Using the Direct Chunk Write Function (PDF)](https://support.hdfgroup.org/releases/hdf5/documentation/rfc/DECTRIS%20Integration%20RFC%202012-11-29.pdf)| Describes another way that chunks can be written to datasets |
29
+
|[Virtual Object Layer](https://support.hdfgroup.org/documentation/hdf5/latest/_h5_v_l__u_g.html)| An abstraction layer within the HDF5 library that enables different methods for accessing data and objects that conform to the HDF5 data model |
Copy file name to clipboardExpand all lines: documentation/hdf5-docs/chunking_in_hdf5.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -216,11 +216,11 @@ Luckily, because each slot in the hash table only occupies the size of the point
216
216
217
217
## Additional Resources
218
218
219
-
The slide set [HDF5 Advanced Topics: Chunking in HDF5 (PDF)](advanced_topics/Chunking_Tutorial_EOS13_2009.pdf), a tutorial from HDF and HDF-EOS Workshop XIII (2009) provides additional HDF5 chunking use cases and examples.
219
+
The slide set [HDF5 Advanced Topics: Chunking in HDF5 (PDF)](https://support.hdfgroup.org/releases/hdf5/documentation/advanced_topics/Chunking_Tutorial_EOS13_2009.pdf), a tutorial from HDF and HDF-EOS Workshop XIII (2009) provides additional HDF5 chunking use cases and examples.
220
220
221
221
The page [HDF5 Examples](https://docs.hdfgroup.org/hdf5/develop/_h_d_f5_examples.html) lists many code examples that are regularly tested with the HDF5 library. Several illustrate the use of chunking in HDF5, particularly “Read/Write Chunked Dataset” and any examples demonstrating filters.
222
222
223
-
[Dataset Chunking Issues](advanced_topics/DSChunkingIssues.md) provides additional information regarding chunking that has not yet been incorporated into this document.
223
+
[Dataset Chunking Issues](https://support.hdfgroup.org/releases/hdf5/documentation/advanced_topics/DSChunkingIssues.md) provides additional information regarding chunking that has not yet been incorporated into this document.
224
224
225
225
Directions for Future Development
226
226
As seen above, the HDF5 chunk cache currently requires careful control of the parameters in order to achieve optimal performance. In the future, we plan to improve the chunk cache to be more foolproof in many ways, and deliver acceptable performance in most cases even when no thought is given to the chunking parameters.
|[Environment Variables Used by HDF5](hdf5_topics/EnvVarsUsedByHDF5.html)| Environment variables that can be used when building or using HDF5 |
14
-
|[Using Compression in HDF5](hdf5_topics/UsingCompressionInHDF5.html)| Resources regarding compression |
15
-
|[Improving I/O Performance When Working with HDF5 Compressed Datasets](hdf5_topics/HDF5ImprovingIOPerformanceCompressedDatasets.pdf)| A description of the factors that should be considered when storing compressed data in HDF5 files and how to tune those parameters to optimize the I/O performance of an HDF5 application when working with compressed datasets (PDF) |
16
-
|[Parallel HDF5](hdf5_topics/ParallelHDF5.html)| Information on Parallel HDF5 |
13
+
|[Environment Variables Used by HDF5](https://support.hdfgroup.org/releases/hdf5/documentation/hdf5_topics/EnvVarsUsedByHDF5.html)| Environment variables that can be used when building or using HDF5 |
14
+
|[Using Compression in HDF5](https://support.hdfgroup.org/releases/hdf5/documentation/hdf5_topics/UsingCompressionInHDF5.html)| Resources regarding compression |
15
+
|[Improving I/O Performance When Working with HDF5 Compressed Datasets](https://support.hdfgroup.org/releases/hdf5/documentation/hdf5_topics/HDF5ImprovingIOPerformanceCompressedDatasets.pdf)| A description of the factors that should be considered when storing compressed data in HDF5 files and how to tune those parameters to optimize the I/O performance of an HDF5 application when working with compressed datasets (PDF) |
16
+
|[Parallel HDF5](https://support.hdfgroup.org/releases/hdf5/documentation/hdf5_topics/ParallelHDF5.html)| Information on Parallel HDF5 |
17
17
|[Fill Value and Space Allocation Behavior]()| A table summarizing of the behavioral interactions of HDF5 fill value and storage allocation settings |
18
18
|[H5Fill Behavior]()| The library's fill value behavior for property list settings |
19
-
|[Using Identifiers](hdf5_topics/UsingIdentifiers.html)| Description of how identifiers work |
20
-
|[UTF-8 encoding in HDF5 applications](hdf5_topics/UsingUTF-8EncodinginHDF5Apps.html)| Information on using UTF-8 encoding in HDF5 applications |
21
-
|[HDF5 Dimension Scale Specification and Design Notes](hdf5_topics/H5DS_Spec.pdf)||
19
+
|[Using Identifiers](https://support.hdfgroup.org/releases/hdf5/documentation/hdf5_topics/UsingIdentifiers.html)| Description of how identifiers work |
20
+
|[UTF-8 encoding in HDF5 applications](https://support.hdfgroup.org/releases/hdf5/documentation/hdf5_topics/UsingUTF-8EncodinginHDF5Apps.html)| Information on using UTF-8 encoding in HDF5 applications |
21
+
|[HDF5 Dimension Scale Specification and Design Notes](https://support.hdfgroup.org/releases/hdf5/documentation/hdf5_topics/H5DS_Spec.pdf)||
22
22
23
23
<!--- In doxygen/technical notes -->
24
24
<!--- | [HDF5 Library Release Versions Numbers]() | A description of HDF5 library release version numbering | -->
25
25
26
-
For more advanced features, see [Advanced Topics in HDF5](advanced_topics_list.html)
26
+
For more advanced features, see [Advanced Topics in HDF5](https://support.hdfgroup.org/releases/hdf5/documentation/advanced_topics_list.html)
Copy file name to clipboardExpand all lines: documentation/hdf5-docs/registered_virtual_file_drivers_vfds.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -21,6 +21,6 @@ Please contact the maintainer of a VFD for help implementing the plugin.
21
21
| CUDA GPU | 512 | gds | The HDF5 GPUDirect Storage VFD is a Virtual File Driver (VFD) for HDF5 that can be used to interface with Nvidia's GPUDirect Storage (GDS) API. The driver is built as a plugin library that is external to HDF5. |[https://github.com/hpc-io/vfd-gds](ttps://github.com/hpc-io/vfd-gds)| Suren Byna (sbyna at lbl dot gov)|
22
22
| GDAL vsil | 513 | vsil | The HDF5 GDAL vsil Storage VFD is a Virtual File Driver (VFD) for the GDAL HDF5 driver that can be used to access any file supported by the GDAL Virtual File System Interface ([https://gdal.org/user/virtual_file_systems.html](https://gdal.org/user/virtual_file_systems.html)). |[https://github.com/OSGeo/gdal/blob/master/frmts/hdf5/hdf5vfl.h](https://github.com/OSGeo/gdal/blob/master/frmts/hdf5/hdf5vfl.h)| Even Rouault (even dot rouault at spatialys dot com)|
23
23
| Unidata/UCAR NetCDF-C ByteRange | 514 | byte-range | The Unidata H5FDhttp.[ch] VFD driver is used to support accessing remote files using the HTTP byte range mechanism. It is part of the Unidata Netcdf-C library. |[https://github.com/Unidata/netcdf-c/blob/main/libhdf5/H5FDhttp.c](https://github.com/Unidata/netcdf-c/blob/main/libhdf5/H5FDhttp.c)| Dennis Heimbigner (dmh at ucar dot edu) |
24
-
| MathWorks (MW) VFD | 515 | no search name | The MathWorks (MW) VFD is a Virtual File Driver (VFD) for HDF5 which can be used for reading and writing hdf5 files from/to remote locations including (but not limited to) AWS S3, Azure Blob Storage and Hadoop Distributed File System (HDFS). |[](https://url.com)|Abhi Baruah (abaruah at mathworks dot com)|
24
+
| MathWorks (MW) VFD | 515 | no search name | The MathWorks (MW) VFD is a Virtual File Driver (VFD) for HDF5 which can be used for reading and writing hdf5 files from/to remote locations including (but not limited to) AWS S3, Azure Blob Storage and Hadoop Distributed File System (HDFS). |[](https://url.com)|mwhdf5 at groups dot mathworks dot com |
25
25
26
26
*The Search Name provides a mechanism for searching for a VFD.
*[Software Changes from Release to Release](https://support.hdfgroup.org/releases/hdf5/documentation/release_specifics/sw_changes_1.14.html)
12
+
*[Migrating to HDF5 1.14 from previous releases](https://support.hdfgroup.org/releases/hdf5/documentation/release_specifics/Migrating_from_HDF5_1.12_to_HDF5_1.14.html)
*[Software Changes from Release to Release](https://support.hdfgroup.org/releases/hdf5/documentation/release_specifics/sw_changes_1.12.html)
17
+
*[Migrating from HDF5 1.10 to HDF5 1.12](https://support.hdfgroup.org/releases/hdf5/documentation/release_specifics/Migrating_from_HDF5_1.10_to_HDF5_1.12.html)
*[Why should I care about the HDF5-1.10.2 release? (blog)]()
22
-
*[Software Changes from Release to Release](release_specifics/sw_changes_1.10.html)
23
-
*[Migrating from HDF5 1.8 to HDF5 1.10](release_specifics/Migrating_from_HDF5_1.8_to_HDF5_1.10.html)
22
+
*[Software Changes from Release to Release](https://support.hdfgroup.org/releases/hdf5/documentation/release_specifics/sw_changes_1.10.html)
23
+
*[Migrating from HDF5 1.8 to HDF5 1.10](https://support.hdfgroup.org/releases/hdf5/documentation/release_specifics/Migrating_from_HDF5_1.8_to_HDF5_1.10.html)
0 commit comments