I think the last time the startup issue came up on HN we concluded it was because they are mounting a squashfs filesystem image that was created with the compression turned up to 11, and not using the kind of compression algorithm that is only slow on compression. That and the general terribleness of squashfs.
Now keep in mind we can't blame squashfs here. It was developed for use on <16 MiB large NOR flash chips for embedded devices likely connected over SPI - the underlying flash and interface is so unbelievably slow that no amount of compression or terrible kernel code would ever start showing up in some kind of benchmark. Using it on super fast desktop machines with storage that rivals RAM bandwidth and latency is just the opposite of what it was developed for.
The thing that irks me about this is that we have carefully optimized file systems, virtual memory, and dynamic linkers over the years to try to make starting a program scale only in the number of touched pages, and somehow it was considered sane to destroy all of that and store the program compressed on disk, when we know from experience with every other package manager we had been using that he software wasn't what was using up all of our disk space :/. Most of the large assets that come with software (such as images) don't even compress well as they are either difficult to extract entropy from (machine code; we do it but it usually uses more specialized algorithms) or files that are already compressed (such as images)! In contrast, iOS extracts packages to disk when you install them, and Android has developers use zip files that are configured to just not compress (to instead "store") anything the developer might have expected to be able to memory map at runtime and then has them run the file through zipalign to add padding that ensures all the stored files begin at memory mappable page offsets in the file, allowing them to not extract the file (keeping it as a single unit as signed by the developer)--though they do still extract the code to prelink and now even precompile it!--while not compromising on startup performance.
From my experience squashfs can give quite okay perf if you use the right params and compression. I'd expect one of the largest linux distos to tune those params correctly.
Is “quite okay perf” really the goal post though? Using applications is what makes a computer useful. It’s bad enough that applications these days are terribly slow and inefficient to begin with, I don’t want even more unnecessary slow downs on top of that.
I'm not advocating this usecase for squashfs, just saying that the perf I'm seeing in snaps is not necessarily what properly configured squashfs would give.
Now keep in mind we can't blame squashfs here. It was developed for use on <16 MiB large NOR flash chips for embedded devices likely connected over SPI - the underlying flash and interface is so unbelievably slow that no amount of compression or terrible kernel code would ever start showing up in some kind of benchmark. Using it on super fast desktop machines with storage that rivals RAM bandwidth and latency is just the opposite of what it was developed for.