Skip to content

fd_write is slow because of repeatedly concatenating buffers #93

@TerrorJack

Description

@TerrorJack

when working on https://github.com/haskell-wasm/bsdtar-wasm i noticed that it takes about 2 seconds to extract an archive to native file system with wasmtime or nodejs/uvwasi, but more than a minute to extract to in-memory vfs when using this library. so i took a look at the profile and noticed that the hotspot is fd_write, which concatenates buffers when appending a new buffer to a file. so when we do a lot of small sequential writes to a file, this quickly blows up due to the O(n^2) time complexity in repeated concatenation.

we can fix it by implementing exponential growth of the file data buffer, so the amortized time complexity of appending data would drop down to O(n) where n is the length of data chunk to be appended. though that'll have the side effect of increased memory overhead, so i'd like to hear your opinion: would you be okay to accept a patch to address this issue at the cost of increased memory overhead? or would you rather gate it behind a feature flag to be opted in? thanks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions