Queston about Puppy's in-memory filesystem and kernel disk cache

[m-cuda]

@mikeslr wrote:

If the Key/CD a Puppy has booted from has been removed, no hard-drive is mounted and no swap file/partition exists, than everything which happens has to be happening in RAM; what distinguishes "Actual Used RAM" from "cache & buffers"?

On a full install where the filesystem lives on a hard disk when the kernel reads a file it copies a block from the disk into a kernel disk cache. On a frugal install, it seems to me that this is unnecessary as the filesystem already lives in RAM. Does the kernel directly use the block from the in-memory filesystem or does it make a copy into its disk cache?

vmstat on my frugal install reports a cache about 1GB. Are the pages in this cache duplicates of pages in the in-memory filesystem?

[user1111]

Think of a squashed filesystem (SFS) much the same as a zip file, compressed file contents. Modified to be a filesystem. Indeed there is even a alternative zip based filesystem version that's like SFS that also caters for saving changes back to the zip file. When a file is read from that the compressed version of just a single file is read and decompressed. With that decompressed copy of a file in ram its then used in whatever way intended. The kernel may then cache that files content into its cache buffer so its readily available more directly if accessed again. So you end up with two copies of the file, the space it takes up in compressed form within the sfs (that I'm assuming has been copied to ram), and the decompressed version that may be in ram (being used) - or cached, or dropped from cache in order to free up (ram/cache) space for other things.

If for instance you had a system with 8GB of ram, i.e. 'more than enough', then if there was a 500MB SFS that had been loaded into ram, and fully extracted it took up 1.5GB of space, then conceptually that could result in 2GB of ram/cache space being used to fully accommodate that if each and every file were 'read' during the session. So yes, two copies, the compressed form (as part of the sfs) and a decompressed form, typically where the space the compressed form takes up tends to average around 50% the size of the decompressed form, so as a rough guide 1.5x the space combined. But where in most 'average' usage, not all of the files will be read (decompressed). gparted for instance might not be used during a session, so a decompressed copy of that isn't loaded into cache.

[bigpup]

reports a cache about 1GB

Just because a cache is setup by the kernel memory control system, does not indicate anything is in the cache.
It is just setup to be used when needed.
The memory is still free to use for other purposes, if needed.

To get a much better understanding.
Pup-Sysinfo->Devices->Memory
This will give very detailed info on what memory is doing.
example for mine right now:
Total RAM 2GB
Actual Used RAM: 366 MB Used - (buffers + cached)
Actual Free RAM: 1542 MB Free + (buffers + cached)
Buffers: 80 MB
Cached: 1236 MB
The rest of memory is being used by the graphics controller, which uses system memory.

What is actually going on with Puppy OS SFS's is better discussed here:
How Puppy works:
http://bkhome.org/archive/puppylinux/de ... works.html

[mikeslr]

Sorry, bigpup. Barry's blog entry doesn't answer the question posed. It explains how a Puppy functions when a Storage medium is available:
"What you will have in a RAM-challenged PC is the kernel, initrd.gz (uncompressed) and the "working files" in RAM. Those "working files" are only new and changed files, so there will hardly be anything in RAM, meaning that we are going to find our Pup running on very minimal systems ....we have yet to find out how little RAM will work.

Note though, the set of "working files" will grow during a session. Any new or modified file will get added to it. However, there is a daemon (a background program) that will warn if RAM space is running low, then there is a simple fix -- flush the "working files" to the persistent storage..."

Files aren't disembodied ideas popping into and out-of existence. They are things. In the real world every "thing" occupies space. You can compress it so that it occupies less space*. But it will still be occupying some space. You can refer to it so that only the reference** occupies space: but to use or modify it, a reference isn't sufficient. You have to move it into a space you control.

-=-=-=-
* At least from our 'this world' prospective, even a 'Black Hole' occupies some space in our world. Whether "space" has any meaning from within the prospective of a Black Hole is beyond the scope of this thread. And just thinking about that gives me a head-ache.

** That was what I thought inodes were doing. "An inode is a data structure on a filesystem on Linux and other Unix-like operating systems that stores all the information about a file except its name and its actual data. A data structure is a way of storing data so that it can be used efficiently." I thought on bootup or later opening of an application or file, an inode --each of which occupies little space-- was created in RAM. By following an inode as and when needed the actual data could quickly be obtained from its location on storage. But if storage (=hard-or-USB Drive/CD/DVD) has been removed inodes can't be followed to actual files on storage. Inodes which continue to function must be referencing some location still available to the operating system.

AFAIK, video cards may have a certain amount of RAM adding to the total RAM otherwise only provided by Memory Cards. And I think all recent CPUs are manufactured with a small amount of RAM, used as cache to expedite frequently occurring processes. Those physical entities do not, however, explain how RAM can hold data which decompressed (and thus usable/modifiable) exceeds the amount of RAM.

rufwoof's post explains how. But it reveals that the read-out of pupsys-info can't be taken at face value. For example,
Memory Allocation:
Total RAM: 7853 MB
Used RAM: 3052 MB
Free RAM: 4801 MB
Buffers: 169 MB
Cached: 2088 MB
Actual Used RAM: 795 MB Used - (buffers + cached)
Actual Free RAM: 7058 MB Free + (buffers + cached). Colored for emphasis.

I think the operative word "potential" has to be read into the the statement and the mathematical symbols ( + and - ) have to be reversed. In other words, what the last two lines above are saying is that currently my system is using 795 MB of RAM, plus RAM used by buffers and cached leaving a potential of 7058 MBs available for usage diminished, however, by the such amount of RAM as may be needed for buffers and cached.

FWIW: Menu>System>Htop seems to be indicating that of 7.67 Gbs of RAM, 1.59 Gbs are in use.

Far more informative than either is the information provided by Menu>System>Hardinfo Hardware Information, select the Memory Tab. At this moment, it reads:
-Memory-
MemTotal Total Memory 8042192 KiB
MemFree Free Memory 4791920 KiB
MemAvailable 5913696 KiB
Buffers 187108 KiB
Cached 2220480 KiB
SwapCached Cached Swap 0 KiB
Active 1240708 KiB
Inactive 1667816 KiB
Active(anon) 783948 KiB
Inactive(anon) 630512 KiB
Active(file) 456760 KiB
Inactive(file) 1037304 KiB
Unevictable 175452 KiB
Mlocked 0 KiB
SwapTotal Virtual Memory 0 KiB
SwapFree Free Virtual Memory 0 KiB
Dirty 0 KiB
Writeback 0 KiB
AnonPages 484568 KiB
Mapped 263732 KiB
Shmem 913532 KiB
KReclaimable 66288 KiB
Slab 117872 KiB
SReclaimable 66288 KiB
SUnreclaim 51584 KiB
KernelStack 3248 KiB
PageTables 9856 KiB
NFS_Unstable 0 KiB
Bounce 0 KiB
WritebackTmp 0 KiB
CommitLimit 4021096 KiB
Committed_AS 2110560 KiB
VmallocTotal -1 KiB
VmallocUsed 7816 KiB
VmallocChunk 0 KiB
Percpu 1024 KiB
AnonHugePages 147456 KiB
ShmemHugePages 0 KiB
ShmemPmdMapped 0 KiB
DirectMap4k 86528 KiB
DirectMap2M 8187904 KiB

But I neither understand what most of that information means, nor --more importantly-- what its tells me about the capabilities and limitations of my system, nor how I could best make use of such knowledge.

[bigpup]

You have gone down into the black hole of memory control by the Linux kernel!

This is all you really need to know:
Actual Used RAM: 795 MB
Actual Free RAM: 7058 MB
And it is automatically being controlled, as needed, based on whatever is going on.
Stuff is constantly going into and out of memory as needed.
These readings are going to constantly change.
You will only get a head ache trying to keep up with memory control activity.

[m-cuda]

@rufwoof wrote

So you end up with two copies of the file, the space it takes up in compressed form within the sfs (that I'm assuming has been copied to ram), and the decompressed version that may be in ram

Thanks for this - I did not know the the *.sfs filesystems were left compressed when loaded into RAM but that makes a lot of sense because:

• Uncompressing the .sfs file would significant slow the startup of Puppy.

• It would be a significant waste of RAM and CPU cycles to uncompress files that may never be used.

It also partially explains why the baseline memory usage as reported by htop is much smaller than I would expect.

@bigpup wrote

Just because a cache is setup by the kernel memory control system, does not indicate anything is in the cache.
It is just setup to be used when needed.

@williams2 wrote

Ram in a tmpfs file system shows up as ram in the cache area. So files in /tmp/ and if the save layer is in ram (pfix=ram, or usbflash or ataflash), these files show as taking space in the cache area.

I think that explains why the cache is so huge. From this and the information in rufwoof's post for files that are in use at least for some of the blocks there are two copies - the compressed blocks and the uncompressed blocks - living in the cache, so not surprisingly the cache is huge. It also means that the 'used' reported by top is very optimistic as it totally discounts cache. Incidentally, the postviewtopic.php?f=2&p=7313#p7313 from which I got this quote has a lot more very interesting information about how Puppy uses RAM.

a cache is setup

Are you sure the kernel works this way? I think it just dynamically transfers blocks from its free memory list to the cache as needed - there is no need for a setup phase - all blocks in the cache are in use, there is no need for pre-allocation.

I thank everyone for responding to this question. I understand a lot more but I am afraid I am still not quite sure I understand totally how Puppy is using RAM - need more time to think and do more research.

@mikeslr wrote

--more importantly-- what its tells me about the capabilities and limitations of my system, nor how I could best make use of such knowledge.

That is where I want to reach - seems rather difficult.

ADDENDUM

@rufwoof wrote

So you end up w ith two copies of the file, the space it takes up in compressed form within the sfs (that I'm assuming has been copied to ram), and the decompressed version that may be in ram

I have been thinking about this and it seems to me that once Puppy has decompressed a .sfs file it no longer needs the compressed copy so it could return the blocks of the compressed file to the free list, i.e., there only needs to be exactly one copy of a file - either the compressed file or the uncompressed file - in the cache at any one time.

@williams2 wrote

Ram in a tmpfs file system shows up as ram in the cache area. So files in /tmp/ and if the save layer is in ram (pfix=ram, or usbflash or ataflash), these files show as taking space in the cache area.

Does this mean on a frugal install for an executing binary executable living in a .sfs file there may be three copies of the code section in RAM - one in the compressed .sfs file living in the cache, one in the uncompressed file also living in the cache and one living in executable memory. There really is a need for only one copy. How does Puppy handle this?

[user1111]

Idle ram/cache/CPU is wasted ram/cache/CPU .. might as well have a smaller system where the resources are better utilised. Fatdog for instance fully uncompressed (base sfs) is around 1.3GB, reduced to 400MB when compressed using xz high compression. So 2GB and the entire compressed and decompressed 'files' could all fit. Add on what Linux uses and 2.5GB perhaps is 'comfortable'. Many 'modern' systems more than accommodate that.

Running older Puppies that utilise far less space in more modern systems and that could be considered as being like having a large garage built where your car can be lost in a corner, loads of wasted/idle space.

At the other end, where ram challenged, then not copying the main sfs into ram, but reading files from it on a as/when needed basis and keeping as much of the (decompressed) copy of those files in cache, will still tend to perform relatively well, albeit with a delay (I/O) the first time files are read (or re-read if they had to be ejected out of cache).

Conceptually, not copying the main sfs into ram will boot quicker, but then take longer to run programs the first time (as the files are read from disk and decompressed into ram), but then once cached subsequent runs of the program will be quick (assuming cached). Copying the main sfs into ram will be slower to boot, but see faster access when pulling out a file. Run long enough - where all files have been decompressed/cached, and operational speed will be much the same, but where not having loaded the main sfs into ram will have used less total ram/cache. Not copying the main sfs into ram however means that the sfs has to remain available, such as being contained on HDD and having that partition mounted. In contrast having copied the main sfs into ram means that the HDD/partition can be unmounted after copying the sfs into ram.

Presently running 'free -m' on my laptop indicates 24MB used, 12MB shared, 12MB buff/cache, 3149MB available, 3400MB total (its a 4GB ram system where some is reserved for 'graphics'). Has a 1TB HDD that could be completely empty, boots from a 14GB USB that df -h indicates a use% of 0%. Running 'top' (not htop), presents the exact same figures. Very much a case of too big of a garage - but at times I do 'better utilise' those resources. Basically that's a minimal boot, no 'external' modules, only the modules I actually use built into the 5MB vmlinuz (kernel); cli framebuffer with framebuffer vnc into my desktop system (that's running Fatdog booted from a DVD) via wifi, but where the desktop system has ethernet net connection (so downloading stuff is around 10x quicker). Screen refresh rates are quick enough via (compressed) ssh tunnelling of vnc and where after a hour+ of general browsing around and writing this post the vnc server has transmitted around 125MB of data (i.e. laptop vnc client has downloaded around that amount of 'screen' data). Nice in a similar context to tmux, where you can disconnect from one device, and reattach again maybe from another device, and the desktop is as-how-you-left-it.

screenshot.jpg (212.97 KiB) Viewed 708 times

[user1111]

@rufwoof wrote

So you end up with two copies of the file, the space it takes up in compressed form within the sfs (that I'm assuming has been copied to ram), and the decompressed version that may be in ram

I have been thing about this and it seems to me that once Puppy has decompressed a .sfs file it no longer needs the compressed copy so it could return the blocks of the compressed file to the free list, i.e., there only needs to be exactly one copy of a file - either the compressed file or the uncompressed file - in the cache at any one time.

I sometimes PXE boot ... using the network card. My early boot initrd has early wifi net connect and ssh/sshfs support, so the main system sfs could be on a remote system where that systems folder is sshfs mounted as part of bootup. There are all sorts of 'tricks' that could/can be performed. Not sure about releasing sfs blocks however. Yes you can read the sfs's header to see what compression its using and follow the inodes to the compressed data blocks (even 'replace' the content of individual files if you were a dark-hat), but that's a bit of a can of worms, as if the system decides to drop blocks to make space for say a large video edit, then you'd have to reinstate the freed blocks contents and update the sfs's inodes/pointers. A lot of effort in a era where disk and ram is relatively inexpensive (and growing). A lot of complexity/risk for broadly little reward.

[Clarity]

A lot of effort in a era where disk and ram is relatively inexpensive (and growing). A lot of complexity/risk for broadly little reward.

This is not just profound, its accurate!

Best when we consider how to improve performance with additional functionality that the hardware (local PC) and our environment (LANs and WANs) affords us; versus counting bits and bytes. This would mean the prize is in improved system behavior rather than something smaller. If focused on improved behavior results in something smaller that is all fine. But the behavior far outweighs the size or RAM use.

This knowledge in this thread is helpful, though. Thanks everyone.

[williams2]

Does this mean ... there may be three copies of the code section in RAM ... There really is a need for only one copy. How does Puppy handle this?

An executable file of binary instructions for the cpu needs to be in ram for the cpu to be able to execute the instructions.
The cpu can not execute the instructions in a file directly.
The instructions must be copied to ram first.
When the program has stopped executing, the ram memory is available to the system again.

Input/output to a file system almost always uses a buffer.
A copy of a file in a buffer may be written over almost immediately.
In any case, the buffer is always available for the next read or write to the file system.

Caching is a very practical way to speed up input/output operations.
If there is any free and unused ram, using it for caching can speed up input/output a lot.
Ram in the cache is instantly available if needed.

It's probably better to think of tmpfs file systems as taking up space in ram as opposed to living in the cache. But space used by tmpfs still is shown as space used in the cache area.

So all of those copies are useful to have.

Puppy handles it's ram the same way other Linuxes do, using the same kernel and drivers.
Usually other Linuxes have a single partition mounted as the main part of their virtual file system, and Puppy usually has a layered file system, but they all work about the same way.

[williams2]

I think that explains why the cache is so huge

The cache is basically all of the ram that is not being used for other purposes, like executables and buffers, and the tmpfs file system.

If you run Puppy on a computer with 64 GiB of ram, most of that ram would be available for caching.

[some1]

https://landley.net/writing/memory-faq.txt

Ulrich Drepper (the ex-glibc maintainer) wrote an article series called
"What every programmer should know about memory":

Part 1: http://lwn.net/Articles/250967/
Part 2: http://lwn.net/Articles/252125/
Part 3: http://lwn.net/Articles/253361/
Part 4: http://lwn.net/Articles/254445/
Part 5: http://lwn.net/Articles/255364/
Part 6: http://lwn.net/Articles/256433/
Part 7: http://lwn.net/Articles/257209/
Part 8: http://lwn.net/Articles/258154/
Part 9: http://lwn.net/Articles/258188/

Mel Gorman's book "Understanding the Linux Virtual Memory Manager" is
available online:

http://kernel.org/doc/gorman/