Is there any point for (new) .bz2 archives in the era of Zstd?

It takes years for bzip2 be in every Linux Distro, and we _still_ doing gzip.

LZMA / xz tool are start to get more support, but they are nowhere near universal.

No idea when how long zstd will need.

For Windows systems, 7-zip (.7z, similar compression to .xz) is a free download for Windows 10, and Windows 11 can open up a .7z file with a simple double click.

.zip and .gz no longer need to be used here in 2026.

.gz/deflate is used when something very cheap and very fast is needed. xz/lzma is quite often too slow or requires too much memory even on decompression.

so no, .zip and .gz are very much needed in 2026.

you need python 3.14 for zstd.

uv taking over basically ensures that dependencies won't become managed properly and nothing will work without uv

    >>> 'x\u2009   a'.split()
    ['x', 'a']
    # incorrect; in bytes mode, `\S` doesn't know about unicode whitespace
    >>> list(re.finditer(br'\S+', 'x\u2009   a'.encode()))
    [<re.Match object; span=(0, 4), match=b'x\xe2\x80\x89'>, <re.Match object; span=(7, 8), match=b'a'>]
    # correct, in unicode mode
    >>> list(re.finditer(r'\S+', 'x\u2009   a'))
    [<re.Match object; span=(0, 1), match='x'>, <re.Match object; span=(5, 6), match='a'>]

edit: OP's fully native C++ version using Pystd

    import mmap, codecs

    from collections import Counter

    def word_count(filepath):

        freq = Counter()
    
        decode = codecs.getincrementaldecoder('utf-8')().decode
    
        with open(filepath, 'rb') as f, mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
        
                for chunk in iter(lambda: mm.read(65536), b''):
            
                        freq.update(decode(chunk).split())
            
                    freq.update(decode(b'', final=True).split())
        
                return freq

That's two neat tricks that I'm definitely adding to my bag of python trickery.

... Ah, but I suppose the existing code hasn't avoided that anyway. (It's also creating regex match objects, but those get disposed each time through the loop.) I don't know that there's really a way around that. Given the file is barely a KB, I rather doubt that the illustrated techniques are going to move the needle.

In fact, it looks as though the entire data structure (whether a dict, Counter etc.) should a relatively small part of the total reported memory usage. The rest seems to be internal Python stuff.

If you don't care about efficiency you can just do len(set(text.split())), but that's barely worth making a function for.

All too often, new models' codebases are just a dump of code that installs half the universe in dependencies for no reason, etc.

I'd imagine not, since `ruff format` and `ruff check` are separate things too.

But I'll also add another suggestion/ask. I think this could be improved

  $ ruff format --help
  Run the Ruff formatter on the given files or directories
  $ uv format --help
  Format Python code in the project

  $ ruff format --help
  Formats the specified files. Acts as a drop-in replacement for black. 
  $ uv format --help
  Experimental uv formatting. Alias to `ruff format`

For instance, try adding `np.random.shuffle(ref_s[0])` after the line `ref_s = self.voices[voice]`...

EDIT: be careful with your system volume settings if you do this.