fix: keep scratch dirs off NFS - #18
Merged
Merged
Conversation
tempfile.mkdtemp() lands on /tmp, which on a SEPAL sandbox is NFS. For the sample-points layer that decides how long tile generation takes, since tippecanoe scratches alongside its output. Route the three temp dirs through pysepal's scratch_dir(). Needs a pysepal that ships pysepal.scripts.scratch.
0.2.1 passes tippecanoe --temporary-directory, so the local dest_dir this branch introduces also moves tippecanoe's scratch off NFS. With 0.2.0 only the output moves and tile generation stays slow.
3.8.1 ships pysepal.scripts.scratch, so the git dependency is no longer needed. Also drops comments whose context lives in the PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tempfile.mkdtemp()resolves to/tmp, which on a SEPAL sandbox is an nfs4 mount of the home export — slow for many small writes, and counted against the storage quota. The sample-points layer feels it most: tippecanoe scratches alongside its output, so the directorybuild_sample_points_layerpicks decides how long tile generation takes. Profiling a sandbox, 1000 points took 28.3s with the scratch on NFS and 0.44s with it on local disk, and the cost tracks tile count rather than point count, so small designs pay the same.Routes the three temp dirs (
map.py,state_manager.py,geospatial.py) through pysepal'sscratch_dir(), which prefers container-local/var/tmpon SEPAL and the standard temp dir everywhere else. Also bumpsvectortileserverto>=0.2.1: that release passes tippecanoe--temporary-directory, without which only the output moves and generation stays slow.Blocked on a publish: vectortileserver 0.2.1 is tagged on GitHub but not yet on PyPI, so the environment build — and therefore CI — will fail until it is. pysepal 3.8.1 (which ships
pysepal.scripts.scratch) is released, and the env already tracks pysepal main.198 tests pass locally. One test that monkeypatched
map_mod.tempfile.mkdtempnow patchesscratch_dir; two comments namingmkdtempwere updated.