Avoid allocation of internal array to get _distance code #97
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
So with #98 and lzbench ( dinosaure@turbine:~/dev/lzbench$ ./lzbench -ezlib,4/decompress,3 news
lzbench 1.8 (64-bit Linux) Assembled by P.Skibinski
Compressor name Compress. Decompress. Compr. size Ratio Filename
memcpy 14079 MB/s 14869 MB/s 377109 100.00 news
zlib 1.2.11 -4 32 MB/s 200 MB/s 149282 39.59 news
decompress dev -3 1.91 MB/s 74 MB/s 292951 77.68 news
done... (cIters=1 dIters=1 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)
dinosaure@turbine:~/dev/lzbench$ ./lzbench -ezlib,4/decompress,3 news
lzbench 1.8 (64-bit Linux) Assembled by P.Skibinski
Compressor name Compress. Decompress. Compr. size Ratio Filename
memcpy 15176 MB/s 15324 MB/s 377109 100.00 news
zlib 1.2.11 -4 32 MB/s 201 MB/s 149282 39.59 news
decompress dev -3 20 MB/s 74 MB/s 292951 77.68 news
done... (cIters=1 dIters=1 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)In other words, 1.91 MB/s to 20 MB/s! Good catch @Engil! I will merge it. |
dinosaure
added a commit
to dinosaure/opam-repository
that referenced
this pull request
Mar 3, 2021
CHANGES: - Add a little executable to benchmark inflation into the distribution (@dinosaure, mirage/decompress#93) - Add instructions for running benchmark (@gs0510, mirage/decompress#94) - Clarify the description (@XVilka, mirage/decompress#96) - Improve the benchmark and outputs (@dinosaure, @gs0510, mirage/decompress#95) - Avoid allocation of distance table (@Engil, @dinosaure, mirage/decompress#97) - Swapping from arithmetic to logical bitshifts on `d.hold` (@clecat, mirage/decompress#99) - Make the use of all `Higher.compress` arguments (@vect0r-vicall, mirage/decompress#103) - Apply ocamlformat.0.16.0 (@dinosaure, mirage/decompress#105, mirage/decompress#107) - Improve Lz77 algorithms (@dinosaure, mirage/decompress#108) **breaking changes** the deflation expects a new window: `De.Lz77.make_window` instead of `De.make_window` (which is twice larger to improve the compression algorithm) Depending on the level and your corpus, we did not observe performance regression on deflation (and mirage/decompress#97 improves a lot performances). An higher level is slower (but the compression ratio is better). We advise, by default, to use the level 6. Note that the user is able to make its own compression algorithm according to his corpus. An example of such implementation is available on the new `decompress.lz` libraries which fills a queue and compress the input. **breaking changes** decompress expects a level between 0 and 9 (inclusive) (instead of 0 and 3). - Add tests about level compression (@dinosaure, mirage/decompress#109) - Add level on GZip layer (@dinosaure, mirage/decompress#110) - Provide a `ctypes` reverse binding (@dinosaure, mirage/decompress#98) - Provide a binary `decompress.pipe` which can compress/uncompress with deflate, zlib or gzip format.
dinosaure
added a commit
to dinosaure/opam-repository
that referenced
this pull request
Mar 3, 2021
CHANGES: - Add a little executable to benchmark inflation into the distribution (@dinosaure, mirage/decompress#93) - Add instructions for running benchmark (@gs0510, mirage/decompress#94) - Clarify the description (@XVilka, mirage/decompress#96) - Improve the benchmark and outputs (@dinosaure, @gs0510, mirage/decompress#95) - Avoid allocation of distance table (@Engil, @dinosaure, mirage/decompress#97) - Swapping from arithmetic to logical bitshifts on `d.hold` (@clecat, mirage/decompress#99) - Make the use of all `Higher.compress` arguments (@vect0r-vicall, mirage/decompress#103) - Apply ocamlformat.0.16.0 (@dinosaure, mirage/decompress#105, mirage/decompress#107) - Improve Lz77 algorithms (@dinosaure, mirage/decompress#108) **breaking changes** the deflation expects a new window: `De.Lz77.make_window` instead of `De.make_window` (which is twice larger to improve the compression algorithm) Depending on the level and your corpus, we did not observe performance regression on deflation (and mirage/decompress#97 improves a lot performances). An higher level is slower (but the compression ratio is better). We advise, by default, to use the level 6. Note that the user is able to make its own compression algorithm according to his corpus. An example of such implementation is available on the new `decompress.lz` libraries which fills a queue and compress the input. **breaking changes** decompress expects a level between 0 and 9 (inclusive) (instead of 0 and 3). - Add tests about level compression (@dinosaure, mirage/decompress#109) - Add level on GZip layer (@dinosaure, mirage/decompress#110) - Provide a `ctypes` reverse binding (@dinosaure, mirage/decompress#98) - Provide a binary `decompress.pipe` which can compress/uncompress with deflate, zlib or gzip format.
dinosaure
added a commit
to dinosaure/opam-repository
that referenced
this pull request
Mar 3, 2021
CHANGES: - Add a little executable to benchmark inflation into the distribution (@dinosaure, mirage/decompress#93) - Add instructions for running benchmark (@gs0510, mirage/decompress#94) - Clarify the description (@XVilka, mirage/decompress#96) - Improve the benchmark and outputs (@dinosaure, @gs0510, mirage/decompress#95) - Avoid allocation of distance table (@Engil, @dinosaure, mirage/decompress#97) - Swapping from arithmetic to logical bitshifts on `d.hold` (@clecat, mirage/decompress#99) - Make the use of all `Higher.compress` arguments (@vect0r-vicall, mirage/decompress#103) - Apply ocamlformat.0.16.0 (@dinosaure, mirage/decompress#105, mirage/decompress#107) - Improve Lz77 algorithms (@dinosaure, mirage/decompress#108) **breaking changes** the deflation expects a new window: `De.Lz77.make_window` instead of `De.make_window` (which is twice larger to improve the compression algorithm) Depending on the level and your corpus, we did not observe performance regression on deflation (and mirage/decompress#97 improves a lot performances). An higher level is slower (but the compression ratio is better). We advise, by default, to use the level 6. Note that the user is able to make its own compression algorithm according to his corpus. An example of such implementation is available on the new `decompress.lz` libraries which fills a queue and compress the input. **breaking changes** decompress expects a level between 0 and 9 (inclusive) (instead of 0 and 3). - Add tests about level compression (@dinosaure, mirage/decompress#109) - Add level on GZip layer (@dinosaure, mirage/decompress#110) - Provide a `ctypes` reverse binding (@dinosaure, mirage/decompress#98) - Provide a binary `decompress.pipe` which can compress/uncompress with deflate, zlib or gzip format.
dinosaure
added a commit
to dinosaure/opam-repository
that referenced
this pull request
Mar 3, 2021
CHANGES: - Add a little executable to benchmark inflation into the distribution (@dinosaure, mirage/decompress#93) - Add instructions for running benchmark (@gs0510, mirage/decompress#94) - Clarify the description (@XVilka, mirage/decompress#96) - Improve the benchmark and outputs (@dinosaure, @gs0510, mirage/decompress#95) - Avoid allocation of distance table (@Engil, @dinosaure, mirage/decompress#97) - Swapping from arithmetic to logical bitshifts on `d.hold` (@clecat, mirage/decompress#99) - Make the use of all `Higher.compress` arguments (@vect0r-vicall, mirage/decompress#103) - Apply ocamlformat.0.16.0 (@dinosaure, mirage/decompress#105, mirage/decompress#107) - Improve Lz77 algorithms (@dinosaure, mirage/decompress#108) **breaking changes** the deflation expects a new window: `De.Lz77.make_window` instead of `De.make_window` (which is twice larger to improve the compression algorithm) Depending on the level and your corpus, we did not observe performance regression on deflation (and mirage/decompress#97 improves a lot performances). An higher level is slower (but the compression ratio is better). We advise, by default, to use the level 6. Note that the user is able to make its own compression algorithm according to his corpus. An example of such implementation is available on the new `decompress.lz` libraries which fills a queue and compress the input. **breaking changes** decompress expects a level between 0 and 9 (inclusive) (instead of 0 and 3). - Add tests about level compression (@dinosaure, mirage/decompress#109) - Add level on GZip layer (@dinosaure, mirage/decompress#110) - Provide a `ctypes` reverse binding (@dinosaure, mirage/decompress#98) - Provide a binary `decompress.pipe` which can compress/uncompress with deflate, zlib or gzip format.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I was thinking that OCaml was able to allocate
tonly one time but it seems not true and I think it really slow-down the deflation when, at each call of_distance, we do a major-allocation. This behavior was discovered by @Engil. This issue appear when we enlarge the minor heap and whenmallochardly try to allocate this bigarray.In our case, it's a static array, so this PR should fix this slow-down.