Thoughts about keeping WDL initrd small and putting modules externally

[miltonx]

I know that rockedge has developed a huge puppy kernel, which keeps initrd in a skeleton state and 00modules & 01fm can be put outside the intrd. But when I booted with it, my screen display showed improper resolution. This lead me to think that puppy's kernel + modules may not work on some hardware, especially when you try to weedog-boot new or weird distros.

In this case, I think it's probably better to use that distro's built-in vmlinuz and modules, but the process of decompressing/recompressing weedog initrd is kind of annoying. So I have been thinking, is it possible to use that distro's own vmlinuz and modules folder, while at the same time keeping the modules outside of initrd?

I came to re-read wiak's initrd init and w_init scripts.
I noticed that the modprobe procedure (many modules!) is done quite early in the init script (except another modprobe later in w_init to get overlayfs module). Trying to load so many modules in an early stage means you have to squeeze a big module folder in initrd (with all the decompressing/recompressing rigamarole), since we are talking about using distro's vmlinuz instead of a huge puppy vmlinuz now.

So, is it possible to separate the modprobe procedure into two parts? First part is only to add those absolutely needed modules for reading and mounting and overlaying file systems. Second stage happens after all the layers are merged, so that an external full modules folder can be probed to drive other hardware parts, such as graphics, audio, etc.

The purpose is to let the initrd contain a minimal modules folder, only enough for mounting and overlaying files. Other modules can simply reside externally in the 00-modules folder. And the initrd can stay unchanged from distro to distro. For each new distro, we only have to copy the vmlinuz and modules to the weedog-xxx directory.

This means the init and w_init scripts need to be re-written to split modprobe into an early stage and a later stage, which I think is viable. But I'm not sure whether it is practical to build a minimal module folder into initrd which would work with all distros' vmlinuz.

[wiak]

Actually all Puppy's use a huge type kernel as far as I am aware. So you could use any of these externally with WDL skeleton initrd. In reality, it is usually the case that the WDL initrd, despite the huge kernel containing device driver modules, does still need access to one module. That is the overlayfs module since Puppy huge kernel by default does not provide that (since they use aufs), though it could also be compiled in to the huge kernel of course (but generally isn't as yet). Anyway, it was because of that 'missing' overlayfs module that I added a small special piece of code inside WDL initrd to temporarily load a 00modules addon (as either an sfs or an uncompressed directory per usual WDL design practice), simply to use the overlayfs module as it thus happens... Otherwise initrd skeleton would need no modules at all for huge kernel booting.

If kernel is not of that 'huge' type then initrd needs to have media driver modules included - it cannot access them from external media since, not being a huge kernel means it can't read disk partitions unless modules actually included inside the initrd itself.

Rockedge's rt40 huge kernel is just a special compile of his own to provide real time kernel facilities I believe, but a normal FossaPup (or similar) huge kernel can also work with the WDL initrd since it was specially designed to allow that external module addon (for achieving access to overlayfs module). I note that the amount of firmware included with rockedge's huge kernel is quite minimal and that may be the reason for your screen resolution issues, though I don't know).

Firmware isn't needed at initrd stage and can simply be provided to the main firstrib_rootfs via any suitable overlayfs layer (or as an alternative included inside the firstrib_rootfs itself).

Being able to use uncompressed filesystems (simple directories) as an alternative to sfs form of the same is a key part of WDL initrd design, which proves very useful in development/hacking (and more so since w_init can be stored externally and simply edited at any time with geany or similar). The key WDL initrd function is its special _addlayer loop, which is called several times allowing you to store addon layers in various alternative locations, including inside the initrd itself (for a huge initrd arrangement) or on alternative partitions than the one being booted from - the result is that there is a great deal of flexibility in how the overall WDL filesystem can be arranged, and with up to 100 numbered/ordered layers (which itself can also be easily expanded within Linux loop limitations).

I do like using huge kernel variations though since so easy to switch kernel/modules/firmware when desired. But non-huge-kernel combinations aren't too difficult to arrange either really and sometimes it is better to stick to the officially provided kernel/modules combinations I feel.

[wiak]

In answer to your question about splitting modules into two. Yes, you could do that (but not in the initrd itself - see main reason at foot of this post). But the list of modules needed (when not provided inside a huge kernel) needs to be quite large simply because there are so many different boot media types that need to be catered for if you want a general purpose boot on any computer initrd. You do not need all the non-media related modules at that boot initrd stage and do not need to include them inside the initrd at all. Unfortunately it is not easy at all to separate out the needed from the not needed and in practice it is simpler just to include them all. Yes, that makes a bigger initrd, but if you check WeeDog the HowTo Cheatsheet (viewtopic.php?p=36426#p36426), fredx181 of DebianDog provided a slim-modules script (cr_initrd) mechanism whose purpose is in fact to select only boot media modules for inclusion inside the WDL initrd: viewtopic.php?p=36345#p36345 and you will find some other posts about using that also linked to from the Cheatsheet.

You certainly can't keep all modules out of the initrd if not using a specially compiled huge kernel (since that is the whole point of compiling a huge kernel in fact).

In terms of simplifying the procedure for including the 'needed' modules inside the initrd, that's what fredx181 was suggesting in terms of including his cr_initrd functionality in modify_initrd. In other words a special helper script should be written to simplify that task (the initrd itself is best left to do its own job without needing that extra code functionality built into it since once completed that helper script code would be entirely redundant inside the initrd itself, so best not there at all, and wouldn't work anyway - see below...). I haven't myself written that helper script that would either incorporate or call up fred's cr_initrd script, but I'd be happy for anyone interested to do so - I may never get round to that... Note that the initrd itself could not be used to pull in the needed modules, since it isn't itself able to see the boot partition until the modules are already loaded into it - that's why you need a pre-first-use helper script to do that job for you.

Note that one prime reason I designed the initrd/init to be in two parts (init and w_init) was to allow w_init to be external and thus able to be easily user-customised. The idea was to also allow users to contribute their own w_init variants (i.e. external w_init was intended to be considered a user-plugin to allow user contributed variants in the same way build_firstrib_rootfs script also has a user-contributed plugin approach, which rockedge, for example, has well-utilised to provide WDL_Void Linux variants particularly for Zoneminder uses). Basically I wanted a simpler forum-community way for user-involvement/contribution rather than some other alternative woof-CE-style complex build system (but also designed to adopt various useful Puppy components and in particular its huge kernel/module combinations and some portable sfs addons) - in WDL, build script system simplicity has been behind every design decision, along with a careful eye on overall flexibility, strong modularity, and superior multi-layered functionality (in significantly smaller initrd/init code size), which is really why WDL components are so easy to manipulate for so many distro uses (from tiny to huge) - the very generic WDL initrd design being key along with the principle of using official upstream package managers for true scalability and reliable upstream repo use.

In practice, I may not adopt every user-contributed implementation or idea into the core initrd/init design, which is for a particular family business-related use of my own (and I'm too busy to add/change much in the core WDL scripts at the moment anyway), but all working plugins will certainly be shared and published, so all such contributions are welcome and encouraged. That's the point of a plugin-based build system anyway - allows a tight and consistent core design without limiting flexibility and overall development/scale.

w_init also allows users to nicely provide quick bug-fix discoveries, such as you did, miltonx, for the nosuid,nodev inram issue (which, funnily enough, I originally missed because chose to use huge kernel rather than the official Ubuntu Kylin kernel/modules...).

[miltonx]

Thanks, wiak, this answers my question. I suspected that it might not be practical to include basic file system modules which are minimal and versatile for all kernels at the same time. But I will read the posts you mentioned. And I will try to understand more of the init script, in order to keep on with my experiments.

[wiak]

Thanks, wiak, this answers my question. I suspected that it might not be practical to include basic file system modules which are minimal and versatile for all kernels at the same time. But I will read the posts you mentioned. And I will try to understand more of the init script, in order to keep on with my experiments.

Yes, aside from rockedge and myself, there may not be many who have familiarised themselves with the initrd/init or with the optional build_firstrib_rootfs plugin system - but there is a lot of scope there (way beyond what I would ever have time myself to do). Providing a frugal install capability for big distros like Ubuntu Kylin and Manjaro, for example, is certainly one big use of WDL initrd, but the build_firstrib_rootfs facility has the capability to build the 'firstrib' root filesystem component in any shape and form desired so would be great to see someone create a really tiny, yet expandable distro with that - and some would like to see an iso, which is something that hasn't really been done very much at all. One way would be to strip down a current Puppy iso and replace the main components with a WDL system - that way it could benefit from the EFI boot capabilities already catered for in recent Pups (including the ability to boot direct from iso) - better than re-inventing that part of the wheel I feel. I just don't have time myself at the moment though, but good to see WDL initrd being used quite a bit recently starting with Manjaro XFCE efforts.

[rockedge]

and some would like to see an iso, which is something that hasn't really been done very much at all

True! Although I have produced a burnable ISO of WDL32_Void, which is the basic WDL_Void with a JWM - ROX desktop but in a 32 bit version, that I have played around with in VirtualBox a lot. Adding EFI boot was easy as I basically borrowed the Puppy Linux components and methods and stuck in the firstrib_rootfs.sfs and adjusted stuff until it worked.

I can provide a link to this ISO but I have not really refined the system completely and is more a proof of concept creation. Also I never tested the EFI boot portion and only have used GRUB4DOS with this distro. There are several details to clarify about the installation method though, but for the first generation I will try to stick to the idea of an ISO image that can be "burned" to CD(DVD) or to USB drives.

Which distro to use? How complete should it be? I really like the WDL_Void type with WDL_Arch a close 2nd. I suppose to start out it's best to use a system without all the bells and whistles, to get some experience and I will be most likely lean on wiak, fredx181 and the woof-CE guys and peebee at some point to gain the insight needed to achieve a successful ISO version

One way would be to strip down a current Puppy iso and replace the main components with a WDL system

This method works and is what I did to produce the WeeDog32-Void ISO's. That experiment was to provide a 32 bit OS that can run the very latest Firefox and watch YouTube.

First things first....getting WDL-kylin to boot and run with it's own repo provided kernel

[miltonx]

...

Hi, wiak, sorry for another nitpick. When I was reading w_init, I noticed this:

Code: Select all

for fm in *;do
NN=${fm:0:2}
if [ "$NN" = "00" ];then
	if [ "${fm##*.}" = "sfs" ];then
		fw_modules=":00firmware_modules"
		mkdir -p ${layers_base}/00firmware_modules /usr/lib/modules
		mount "${mountfrom}"/${fm} ${layers_base}/00firmware_modules		
		sleep 1  # may not be required
		if [ "fwmod" = "usrlib" ];then
			mount --bind ${layers_base}/00firmware_modules/usr/lib/modules /usr/lib/modules  # needed for overlayfs module
		else
			mount --bind ${layers_base}/00firmware_modules/lib/modules /usr/lib/modules  # default (as in debian and most pups)
		fi
	elif [ -d "$fm" ]; then
		fw_modules=":00firmware_modules"
		mkdir -p ${layers_base}/00firmware_modules /usr/lib/modules
		mount --bind "${mountfrom}"/${fm} ${layers_base}/00firmware_modules		
		sleep 1  # may not be required
		if [ "$fwmod" = "usrlib" ];then
			mount --bind ${layers_base}/00firmware_modules/usr/lib/modules /usr/lib/modules  # needed for overlayfs module
		else
			mount --bind ${layers_base}/00firmware_modules/lib/modules /usr/lib/modules  # default (as in debian and most pups)
		fi
	fi
fi

There is a typo, where fwmod missed "$". Besides, $fwmod is not defined anywhere in the script.

I think the "if...elif" part can be simplified as:

Code: Select all

		...
		if [ -d ${layers_base}/00firmware_modules/usr/lib/modules ];then
			mount --bind ${layers_base}/00firmware_modules/usr/lib/modules /usr/lib/modules  # needed for overlayfs module
		else
			mount --bind ${layers_base}/00firmware_modules/lib/modules /usr/lib/modules  # default (as in debian and most pups)
		fi
		...

[wiak]

Hi miltonx,

Code: Select all

		sleep 1  # may not be required
		if [ "fwmod" = "usrlib" ];then

Yes, that was last part of the code changed, which was quire recently, and you are correct, the above should of course be "$fwmod" (as in the elif section below it), so that omission would result in a bug if trying to use modules that were in an sfs arranged in directory hierarchy for /usr/lib/modules. Thanks for reporting that (wouldn't have cropped up as a bug thus far since no-one yet using modules in an sfs of that form and no-one probably (except myself) would realise there was a variable fwmod I purposively made provision for.

By the way, there is a WeeDog/BugReports forum section, where I will likely move this post since that gives me one place to look for list of such reports (but no complaint from me, thanks for pointing out the error anyway).

As for simplifying the code since fwmod doesn't appear anywhere. Sorry, no, fwmod is a grub kernel line variable for precisely the use the code indicated: if someone has an 00 numbered sfs (or uncompressed) modules directory in form /usr/lib/modules (rather than the traditional Puppy zdrv arrangement, which is /lib/modules) then that /usr/lib/modules form will be correctly mounted and used as long as user puts fwmod=usrlib on their grub kernel line. The default is /lib/modules (Puppy zdrv) form so in that case no fwmod grub kernel line is needed at all. In other words, I purposively do not declare or set fwmod earlier or elsewhere in the init because that is up to grub kernel line to include or not as user requires, which is not an error but intentional on my part.

By the way, it is my intention to change that variable name since it is a legacy of an earlier WeeDog implementation (2 years ago) where I intended putting both firmware and modules in same external sfs, but didn't find that useful in later tests. So reference to firmware_modules more generally will later be amended since that section is actually dealing with modules only (but aside from my renaming the associated variables the code operation will likely remain the same except I suspect Puppy zdrv will move to the /usr/lib/module form in future Pups since I believe Debian/Ubuntu is going to adopt that arrangement in future (which Arch Linux and Void Linux already do) - in that case /usr/lib will end up being my default so it will remain the case that no grub kernel line fwmod (or whatever) variable will normally required.

EDIT: Ah, sorry miltonx, now that I look at it again, seems to me I don't indeed need fwmod, should be enough to just check the mounted 00 filesystem (be it an sfs or an uncompressed dir) and do the bind mount based on what is found inside it - either the /usr/lib case or the /lib case. Thanks, nice simplification. I'll double check but make that mod once I remind myself if I had some other idea in mind (was doing a lot of w_changes mods at the time so sometimes have plans I later forget, but does look like I just chose an inefficient selection method on this occasion - the 00 addon didn't have a lot of significance to me at the time so I put little thought into it, but it is proving useful again now that I'm also sometimes using rockedge's kernel/modules combinations). I will have to change the 'simplification' slightly because I don't want to mount the 00 filesystem if it isn't an sfs or a directory, but that's a small matter (and I could even allow the assumption that is is one of these since it shouldn't be named 00xxxxx otherwise...).

wiak

[miltonx]

Since fwmod does not start with "w_", I did not realize it was intended to be a grub parameter.

[wiak]

Since fwmod does not start with "w_", I did not realize it was intended to be a grub parameter.

Yes, I kind of blundered totally on the naming of that one. But as I've added in my EDIT above, I agree I likely have no use for it anyway.