WDL user contributions

[wiak]

Attached are contributions from the filenamed contributors I received long time back (many thanks).

Note that per all current (25Aug2021) WDL build_system/utilities related code I have assigned these a licence of MIT (rather than leaving them unlicensed, which would be potentially problematic for their further development by others).

From a glance at the scripts they all seem to remain relevant to current WDL builds, and very useful.

Currently wd_rollback_changes_sfs_rufwoof, wd_merge_changes_sfs_rufwoof, and wd_remaster_fredx181.

Simply remove dummy tar and make executable with chmod +x

I had mislaid these small utilities but exploring them now. I cannot guarantee any of them (just about to further test), but all look good, nice and simple to understand, and very useful to me.

wiak

EDIT1. NOTE1 IMPORTANT: That old merge script from @rufwoof looks promising at first sight, but I think there is an issue - the char device files (which are used to indicate similar to aufs "whiteout" files) will 'vanish' in the merge. That's not correct. Unfortunately we have to find these char device files and handle them appropriately, so that merge script is attached for interest only, but will work for case of no "whiteout" marker files present in the sfs files being merged I think... (but often there will be whiteout char device files that need dealt with...). In my w_changes=RAM1 mode, I save back from upper_changes in RAM using rsync to copy back into the rw bootfrom upper_changes directory. I think it might be possible to simply mount the numbered upper_changes files and then rsync them back one by one to an empty directory in a last one wins situation - I imagine the char device files would then be appropriately preserved without my having to seek them out more manually; I will have to try this idea out though to confirm it works - no extra overlayfs used in that method.

EDIT2: hmmm... the above idea seemed to work... I started with two sfs files 50.sfs and 51.sfs. These were actual WDL upper_changes rollback files and I knew there were 'whiteout char device files' in a few places in 50.sfs. After the merge, all seemed fine; the char device whiteout files remained as I needed them to. The code I used was as follows (running from terminal at the directory where the sfs files are); to complete this I just need to put it in a script and in more general purpose form:

Code: Select all

sudo mkdir -p 51 50 merged_changes
sudo mount 51.sfs 51/
sudo mount 50.sfs 50/
sudo rsync -av --delete 51/ merged_changes
mksquashfs 51_50_changes.sfs merged_changes/

Same idea should work fine for a WDL_save_from_RAM (like snap merge) routine without needing to seek out where the char device whiteout files reside (i.e. no 'find' command seekout whiteouts seem required). I may be wrong... we will see. I will try it with w_changes=RAM2 (where media upper_changes is mounted read only at own mount point) - I will probably break my system, but that's okay - I will keep a backup!

EDIT3: Well, seems to simple, but above idea worked for me also for save back w_changes=RAM2 mode back to on media upper_changes. To explain: In RAM2 w_changes mode the on media upper_changes folder is mounted readonly as second to topmost layer, the actual topmost layer is in RAM at /mnt/layers/RAM/upper_changes. My simple save from that RAM upper_changes back to on media upper_changes is basically as simple as:

EDIT: I did have "rsync -av --delete" below but removed the --delete since that is almost certainly not correct for this purpose - might be in the above merge - I haven't checked or thought further about that as yet.

Code: Select all

sudo rsync -av /mnt/layers/RAM/upper_changes/ /mnt/sda4/WDL_arch64/upper_changes

above being for the case where I am booting from directory /mnt/sda4/WDL_arch64, so to make this method more general all I need to do is auto insert the actual boot directory used (which is a simple job indeed since /proc/cmdline contains that info after a boot). As I say, seems to simple... that single line becoming basically my save from RAM to media script... Of course I can also use --exclude along with rsync to exclude any files or dirs I don't need saved (such as various cache or maybe log files or whatever proves useful to exclude). Anywhere, surely I have missed some important detail that will render this totally simple method incorrect - someone may enlighten me, or time will tell... I have checked snapmerge script of dd-bullseye (since that using overlayfs) by @fredx181 and note it makes a list (using find) of whiteout files and uses that in the save/rsync procedure - makes me think my method must have flaws, or I may be lucky... certainly my quick test is not very rigorous so will certainly not surprise me if I have failed to take account of something regarding these whiteout (char device) files. Also I have no idea how dd-bullseye/Porteus initrd/init arranges its overlay structure so may be very different anyway.

NOTE2: I'm still working on getting WDL github public download site up to latest. Been some delays since was busy testing some details. Main additions in these later build scripts is the additional w_changes in RAM modes (which also don't have but need new user utilities to utilise their functionality - i.e. similar utilities to save2flash/snapmerge_whatever - I'm not myself planning to write these utilities in near future though but believe those used in DebianDog's dd-bullseye might work with suitable modifications), and the split init in initrd, the main part of which w_init can be stored externally as a normal (bash code extract) and thus directly edited for quick init boot experiments (so becomes yet another WDL boot plugin addition). Will get that all back running one day soonish

[rockedge]

I like the merge script idea. I also have tried out rufwoof's script which I think I modified at some point but it has been awhile since I've worked with merging the sfs's.
I've downloaded the scripts and after the build of the latest WeeDog64-Void goes well ( works!) I will attempt a Zoneminder build and install then try out some trickery.

Good stuff wiak!

[wiak]

I like the merge script idea. I also have tried out rufwoof's script which I think I modified at some point but it has been awhile since I've worked with merging the sfs's.
I've downloaded the scripts and after the build of the latest WeeDog64-Void goes well ( works!) I will attempt a Zoneminder build and install then try out some trickery.

Good stuff wiak!

Be careful with rufwoof's merge script. I also though using an overlay as he does would do the job, but discovered, as I report above, that it doesn't work because doesn't account for the need to keep 'whiteout' (char device) files. But the simple method I describe in above post do seem to work and even as a one-liner save back from RAM mode script. Sorry, I still haven't pushed the latest build scripts up to github dev site - the ones with w_changes=RAMx modes and split initrd/init into init and external w_init. I'll do that within a few days now I hope - main issue was that there have been many variants developed over past year and I have to sort out the right ones and reorganise the git site with these (thought I had sent latest months ago but turned out I accidentally pushed up the old v208 -rc7 build scripts by mistake).

[rockedge]

I seem to remember that I ended up using the simplified method which worked. rufwoof's script did not quite work as expected and I modified it but seem to think I used something more along the lines of the "simple method"

[fredx181]

I have checked snapmerge script of dd-bullseye (since that using overlayfs) by @fredx181 and note it makes a list (using find) of whiteout files and uses that in the save/rsync procedure - makes me think my method must have flaws, or I may be lucky

First version was much more simple and worked ok (as seemed to be) but there was a thing I didn't like, so extended the script, can't remember well, something with leftover char files from deleted files unneeded and could lead to complications (in some cases), a little vague I know

[wiak]

I have checked snapmerge script of dd-bullseye (since that using overlayfs) by @fredx181 and note it makes a list (using find) of whiteout files and uses that in the save/rsync procedure - makes me think my method must have flaws, or I may be lucky

First version was much more simple and worked ok (as seemed to be) but there was a thing I didn't like, so extended the script, can't remember well, something with leftover char files from deleted files unneeded and could lead to complications (in some cases), a little vague I know

Thanks fred, I'll look into it further. That was just a first stab by me to see that the char files got through okay. I doubt the --delete part of rsync is correct for this particular job but haven't thought of all implications as yet and may well find issues such as you had at the end of the day...

[wiak]

EDIT3: Well, seems to simple, but above idea worked for me also for save back w_changes=RAM2 mode back to on media upper_changes. To explain: In RAM2 w_changes mode the on media upper_changes folder is mounted readonly as second to topmost layer, the actual topmost layer is in RAM at /mnt/layers/RAM/upper_changes. My simple save from that RAM upper_changes back to on media upper_changes is basically as simple as:

Code: Select all

sudo rsync -av /mnt/layers/RAM/upper_changes/ /mnt/sda4/WDL_arch64/upper_changes

above being for the case where I am booting from directory /mnt/sda4/WDL_arch64, so to make this method more general all I need to do is auto insert the actual boot directory used

Well, assuming the bove commandline does prove sufficient, which is certainly very untested (and particularly if --delete is correct here) then nevertheless note well the importance of having that final slash (/) at the end of /mnt/layers/RAM/upper_changes/ and having NO final slash at the end of /mnt/sda4/WDL_arch64/upper_changes (otherwise rsync will create a new directory upper_changes inside upper_changes, which is not what you expected or want!!! (I know it happened to me until I sorted that much out...).

I do expect --delete shouldn't be thre. Also be careful with rsync more generally - I haven't checked but I think it relies on modification times so if a rollback upper_changes file has been made when system clock wrong I imagine that could mess the whole future up... That's why I tend myself not to use RAM modes for business computer use - I simply don't entirely trust it no matter how carefully the save-back mechanisms have been written/tested, but great for my own messing around.

EDIT: I did have "rsync -av --delete" in the RAM mode back to media save, but have now removed the --delete since that is almost certainly not correct for this purpose - might be in the above merge - I haven't checked or thought further about that as yet.

Note that from man pages for rsync, rsync -a is equivalent to rsync -rlptgoD (no -H,-A,-X) so worth checking meaning of these individual options (e.g. maybe D handles overlayfs whitelist char files, and correctly - I'm not sure).

So I now took a very quick look at Puppy's /usr/sbin/snapmergepuppy (in Fossapup). My goodness, does aufs really need all that? I can understand the additional care to make sure you have enough folder space left (good idea) but over 200 lines of code just to save back changes? - hopefully not - I'd abandon all attempts if really that complicated with overlayfs at least. I really hope that I am correct that rsync can do most of the work in one line (but, yeah, seems too simple compared to what Puppy does). But the one-line method does enough for me thus far - maybe since I do very little changes once my system is as I want it and I make the read-only upper_changes rollback files?

[wiak]

First version was much more simple and worked ok (as seemed to be) but there was a thing I didn't like, so extended the script, can't remember well, something with leftover char files from deleted files unneeded and could lead to complications (in some cases), a little vague I know

@fredx181

Yes, I've been experimenting with simple:

sudo rsync -av /mnt/layers/RAM/upperchanges/ /mnt/sda4/WDL_arch64/upperchanges

actually, I think I should be using

Code: Select all

sudo rsync -av /mnt/layers/RAM/upperchanges//mnt/layers/uc_ro

since I have the external upper_changes mounted at uc_ro (and I think I named it wrong - probably better as uc_rw). EDIT: yes, I remember now; I used mount --bind uc_ro to external upper_changes, so really is rw, but I decided to call it uc_ro from the point of view of its overlayfs position, which is a read-only layer so I'll leave the name as it is...

and seems to me the char device whiteouts do get synced down to changes folder okay, but yes, indeed, if a file is removed that is truly only in the top layer anyway then an unnecessary char whiteout file is left behind. So I'd agree that it is probably better to remove these ones as you do: # send files to a list to be removed later

Having said that, I'm not sure what if any damage they do if not removed at end of the rsync (except there could potentially be lots of them... which is very imperfect/inaccurate and isn't good of course). I tried installing and removing and then re-installing an app (beaver editor as it happens) and the unnecessary char device whiteout did appear after the removal (unnecessary since beaver binary had only been installed in top layer so should be gone anyway once installed), but all was fine on re-install (was worried that char whiteout might make pacman refuse to re-install, but didn't). So, yes, to determine which of the whiteout char files it is okay to remove (those related to top layer file/dirs only) it would seem necessary to loop through all the lower layer NN sfs files (as you do) since any char whiteouts in these lower layers need to be left.

So doesn't matter the "a little vague" - I see what you mean anyway since I observe same issue in my own experiments. I'll leave it as utterly simple for WeeDog at present (to confirm how 'bad' the issue becomes) but your extended script does seem better for final use (though not sure you need the additonal '# send "mask" files to list to be copied later', but maybe I miss something about the whys and wherefores of that part (my impression is that rsync -a sends all of them okay so it is only the removal list that is necessary, but no big deal to do both for safety I suppose (or for reasons I don't realise...). Alas I can imagine it takes quite a while filtering throughout all the lower layers looking for char device files - especially if using 100 layers or more potentially!!! - maybe unusual to use so many layers, but it seems that could be a drawback situation if using changes in RAM in this manner (admittedly I don't know since haven't tested any such worst case scenarios): I'm inclined to think keeping 'simple' and not removing unnecessary whiteout char files will be better in multi-layer sfs modules case (though a bit disturbing to allow these unnecessary files to remain and take up disk space - 'might' be slightly 'safer' though in case the (alternative) char whiteout filter ever gets something (unexpectedly/unthought-of) wrong. Yes, I don't entirely 'trust' changes in RAM tricks, even though I usually use them myself too - but for business computer use I'm not so sure... though I do 'trust' changes in RAM if not needing to rsync anything back, i.e. when not really using a rw upper_changes at all but instead just a pre-created upper_changes mount bound and with a tempfs RAM upper_changes that is for the session only and not intended to write back (usually).

FossaPup's snapmergepuppy does look crazy complicated to me though. Whilst I have no idea what it is doing, I guess it is similar (albeit for aufs whiteouts and so on); maybe it is all simply easier with overlayfs because easy to 'test' for a char device file (-c) as you do in dd-bullseye. Or maybe Puppy is filtering out a lot more files/dirs to save the space occupied in the save persistence folder? - thus far I can't see any additional filtering I'd want (only actual changes are appearing in /mnt/layers/RAM/upper_changes, and I'm fine just to rsync all of these at present - except I could use a few simple rsync --exclude entries to not include some unwanted cache's, such as some of the internet browser, and logs I suppose).