NOTES.md

1. Two algorithms for searching top files are implemented. Both methods first split a list of files into several chunks depending on the nthreads parameter. Then, for each chunk, top largest files are found either by linear search or by sorting depending on the top to log(len(chunk)) ratio. For smaller top and larger chunks, the linear search is faster (The precise ratio should be calibrated). Then, the top files from all chunks are merged, sorted, and top files are selected.
2. Compression is parallelized via parallel compressing files into intermediate directory, and sequential writing compressed files into zip file. Extraction is implemented similarly. Writing into zip cannot be parallelized. Parallel compression without writing intermediate compressed files to disk can be implemented via buffering, although an implementation of this approach is not simple using standard library. I can suugest the following approach. Files are being read and compressed concurrently into buffers. If a compressed file is smaller than the buffer, the zip writer is locked and compressed is flushed to the zip file, then zip file is released and can be used by another thread. If buffer is full, then the zip writer is locked, the buffer is flushed, and the rest of the file is being read by blocks, compressed, and flushed to the zip file. The zip writer is released only when the whole file is written.
3. Compressing top from a directory and extracting top files from a zip file is implemented
4. Filtering of files with pattern for compression/extraction is implemented
5. Several compression algorithms are implemented, gzip, flate, lzw, zlib
6. Background extraction is not implemented
7. Unit tests are written but need to be extended
8. There was also an idea to write E2E tests using Robot Framework