Zcompress

You delete the original folder anyway. Keep the .zcmp archive.

zcompress is designed to be memory-conscious. It allows users to define the "window size" or compression level. By lowering the compression level, the tool uses significantly less RAM, making it an ideal choice for embedded devices or legacy hardware where memory is a scarce resource. zcompress

decompressed = zcompress.decompress(compressed) assert json_str == decompressed.decode('utf-8') You delete the original folder anyway

zcompress : original size 2.3 GB → compressed size 410 MB. the tool uses significantly less RAM

Assuming you are using the open-source libzcompress library (a hypothetical but representative implementation), here is a step-by-step guide.

zcmp --threads=16 large_dataset.tar

# Train a dictionary on a representative sample zcmp --train --dict-size=1MB sample/*.json -o my_dict.zcmpdict