Use zstd compression for sfs

[ValdikSS]

Current version of Fossapup64 uses xz compression for sfs (squashfs) files, which is efficient size-wise but pretty slow to decompress.
Using zstd (zstandard) for squashfs compression produces files a but more in size, but the decompression speed is 5-8 times faster, which results in faster OS and program startup and better overall experience.

I'd propose building sfs files with -comp zstd mksquashfs option.

[ThruHammer]

Yo, Mate!

Any Pup improvement proposals belong here.

But then, I know you were only testing us, right? Hahaha...

[taersh]

Yo, Mate!

Any Pup improvement proposals belong here.

But then, I know you were only testing us, right? Hahaha...

I don't get the point, why the opening post should be in Programming section of the forum instead of FossaPup section.

[mistfire]

Current version of Fossapup64 uses xz compression for sfs (squashfs) files, which is efficient size-wise but pretty slow to decompress.
Using zstd (zstandard) for squashfs compression produces files a but more in size, but the decompression speed is 5-8 times faster, which results in faster OS and program startup and better overall experience.

I'd propose building sfs files with -comp zstd mksquashfs option.

However you're unable to swap older kernels. Only the kernels with built-in zstd compression.

[bigpup]

This is something you need to bring up on Woof-CE
https://github.com/puppylinux-woof-CE/woof-CE
Make an issues topic about it.

[dimkr]

Slacko 8.0 already uses zstd.

https://github.com/puppylinux-woof-CE/w ... ssues/2029
https://github.com/puppylinux-woof-CE/w ... 2d4e460e6b

I've been using zstd on my ARM laptop (https://github.com/puppylinux-woof-CE/woof-CE/pull/1948) for quite a while, and I can definitely see that applications start faster.

[LateAdopter]

The underlying problem with Fossapup64 is that it is squashed with -b 512K and my Intel Braswell N3150 can't handle that block size efficiently, probably because the cpu cache is too small.

Having seen dimkr's issue that he raised on github, I already resquashed Fossapup64 and adrv with -comp zstd -b 256K and the problems went away.

The squashfs-tools implementation of zstd is only partial. Compression levels above 19 don't do anything differently from 19. The file sizes are about half way between XZ and GZIP. You need to update squashfs-tools to the version currently on the Ubuntu Focal repo if you want to use zstd.

[Clarity]

This post from me is NOT objecting to this compresson technology: But it intends to have us look to how we got to where we are.

Puppy has an artificial size limit that was imposed as a target back in the days of dial-up where that concern was about the size of the bootable payload (ISOs) over 9.6Kb link. Over the years PUP has found crafty ways to add more stuff in the payload by changing the compression technology for the shipped payload(s). Today we are NOT limited by that awful communications transfer speed; rather for some cell phone users (and some ISPs) of today, we are limited in a monthly allocation for data transfers. As such PUPs attempts at its compressors is mainly aimed to satisfy that mere end, while insuring it is bootable via today's BIOS-EFI PCs.

For me, I am not concerned about the payload, as with ISO booting and save-folders, I am not bothered by the compression technology. MY BIG ISSUE is performance! NOT compression!

If the CPU load is increased by a constant compress-deompress use, this impacts the load, electric demand/battery and heat on any system.

So whatever or which is decided for selection, it should accompany its operation and most importantly its performance on RAM based users,

Hope this helps.

[amethyst]

Gzip compression is a good middle-road. Compression ratio - okay; Compression and Decompression speed - Good; Resource friendly. Your Puppy needs to be very large to see a remarkable difference in performance between different compressors. If the puppy files are copied to RAM at bootup the difference is even less noticable and of course once stuff are loaded into memory as you work, there are no difference. Interesting thing is that a higher compression level takes longer to compress but quicker to decompress. Gzip level 4 or 5 is a good option for compressing and decompressing..

[ValdikSS]

However you're unable to swap older kernels. Only the kernels with built-in zstd compression.

Current Fossapup64 kernel is compiled with zstd in squashfs support, I've checked it.

[ValdikSS]

Puppy has an artificial size limit…

If it's still the case, you can use current xz compression and recompress sfs files to zstd (or even lz4, it's 4 times faster) upon installation. This way you'll have very fast installed system while maintaining small iso file size.

[ValdikSS]

Your Puppy needs to be very large to see a remarkable difference in performance between different compressors. If the puppy files are copied to RAM at bootup the difference is even less noticable and of course once stuff are loaded into memory as you work, there are no difference.

I'm not a Puppy user. I was testing Puppy for very old PCs (11-13 years old) as a replacement for Windows XP, as one of the most lightweight distro with good repository support (from Ubuntu). So my use case is underpowered PC with old CPU and <= 1 GB RAM, where you can't use in-RAM loading.

On a virtual machine, XZ decompress speed is about 60 MB/s, while zstd is somewhere about 500 MB/s. I've recompressed all sfs files manually, using mksquashfs from Ubuntu repository, and got a visible improvement in OS and software startup speeds. LibreOffice, for example, starts twice faster.

[amethyst]

Maybe you should do a standalone installation of Puppy instead of running on virtual machine to get more reliable results. If you are really very badly RAM challenged a full install would probably be a good choice (so no sfs files). Having said that I still think gzip is a good compression for sfs files as it's light on resources. The old Puppys used gzip compression.

[dimkr]

According to benchmarks zstd is so much faster than xz and gzip, but the compression ratio is much closer to that of xz. IMHO, if today's Puppy is large anyway, the huge increase in decompression speed and reduction in RAM used during decompression, are worth the modest 20-30 MB increase in size.

[amethyst]

According to benchmarks zstd is so much faster than xz and gzip, but the compression ratio is much closer to that of xz. IMHO, if today's Puppy is large anyway, the huge increase in decompression speed and reduction in RAM used during decompression, are worth the modest 20-30 MB increase in size.

If you have low specs I would go with gzip. Another consideration is that older kernels do not support zstd compression as far as I know. On an old machine a kernel swop to use an older kernel with a new Puppy may be necessary. Gzip works with all kernels.

[dimkr]

If zstd is faster to decompress, and the compression ratio is better, why choose gzip?

https://quixdb.github.io/squash-benchmark/
https://facebook.github.io/zstd/

[amethyst]

If zstd is faster to decompress, and the compression ratio is better, why choose gzip?

https://quixdb.github.io/squash-benchmark/
https://facebook.github.io/zstd/

I just told you. viewtopic.php?p=19272#p19272

[amethyst]

Also - Disk space is hardly an issue these days so one may as well use the compressor with the fastest compression/decompression speed. That seems to be lz4 (if your kernel supports it). I use gzip with block size 4K, compression rate not all that great but quite fast in compressing and decompressing. Also compatible with all kernels, very important if you run a lot of Puppys and share the same extra sfs files between them.

[Grey]

Also - Disk space is hardly an issue these days so one may as well use the compressor with the fastest compression/decompression speed. That seems to be lz4 (if your kernel supports it). I use gzip with block size 4K, compression rate not all that great but quite fast in compressing and decompressing. Also compatible with all kernels, very important if you run a lot of Puppys and share the same extra sfs files between them.

Hello. Why did you choose 4K block size? I want to repack Fossapup from xz to gzip so that it loads a little faster in QEMU.

[amethyst]

Also - Disk space is hardly an issue these days so one may as well use the compressor with the fastest compression/decompression speed. That seems to be lz4 (if your kernel supports it). I use gzip with block size 4K, compression rate not all that great but quite fast in compressing and decompressing. Also compatible with all kernels, very important if you run a lot of Puppys and share the same extra sfs files between them.

Hello. Why did you choose 4K block size? I want to repack Fossapup from xz to gzip so that it loads a little faster in QEMU.

Fastest for compressing in gzip (not so sure about decompressing). Gzip will be faster than xz no matter the quality setting. Note however - unless the file is very big, decompressing time is not really all that noticeable.

[Grey]

Fastest for compressing in gzip (not so sure about decompressing). Gzip will be faster than xz no matter the quality setting. Note however - unless the file is very big, decompressing time is not really all that noticeable.

Took note. Thanks.