Page 1 of 1

Draft: Minimum Puppy Upgrade via Firstrib WLGO rootfs

Posted: Fri Jan 29, 2021 5:30 pm
by s243a

Motivation

When people try to keep an old puppy up to date eventuality at some point they might decide to upgrade a core package (e.g. glibc). Perhaps this is sufficient. However, one can upgrade more than this without a significant impacting the size of the system.

Options
Rather than just upgtading glibc I could take a minimal system that is approximatly less than 50MB and replace the core libs with those from the minimal system.

Some possible minimal systems are tinycore, Slatz, Bulldog (part of Fatdog64) and Wee Dog L_GO. Weedog L_GO rec.

The Fosapup version of Weedog L_GO caught my intrest here because of the apt/dpkg integration and its binary compatability with Fosapup.

I was also curious what wiak was doing with the weedog / firstrib type development and I thought that looking at the rootfs for this project might help me learn a bit about the development that wiak is doing.

The particular version of WDLGO i'm using is WDLGO_UbuntuFocal64. Here is what wiak says about it:

This distro a full-multiuser capable (including PAM) but an absolutely minimum dpkg/apt package manager UbuntuFocal64 compatible system. Except can be made much smaller using slimmed down firmware/modules (fwmods sfs) and deleting docs/man/locale pages etc.

However... being "an absolutely minimum dpkg/apt install" it is primarily provided for development/educational/experimental purposes. DISCLAIMER: USE ENTIRELY AT YOUR OWN RISK!

https://weedoglinux.rockedge.org/viewto ... ?p=332&i=1

The apt/dpkg integration is discussed in the following thread:

Released: WDLGO_UbuntuFocal64 mini-apt/dpkg experimental system

wiak discusses the WDLGO system here:

Under pre-release testing: new WDLGO systems

The actual core WDLGO system is only 27MB so adding UbuntuFocal64 apt/dpkg seems to approximately double the size. Therefore if I wanted a smaller upgrade I could stick to the core WDLGO system.

However, at some point there is deminishing returns when trimming systems. For instance the latest portable ungoogled chromium is 100MB and most modern puppies are over 300MB so even with apt/dpk included the overall size is relatively small.

That doesn't mean that I'm not interested in the core WDLGO system. However, as noted above there are a lot of options for minimal systems. So as an example In my next experiment I might try the same techniques with bulldog in order to provide FATDOG64 compatibility.

First experient Puli/Xenialand further motivation

In my first experient I am using the roots of
WDLGO_UbuntuFocal64 to upgrade core componets of PULI 7.3 (Xenial64). At this point my target system is a chroot or container guest type system and part of my motivation here is to experiment with running browsers in chroot and container type systems. The purpose of the upgrade is to be able to run ungoogled chromium in said system

So how to do the Upgrade

At a high level one upgrades by replacing packages with newer versions. Perhaps the info to do this is easily available (not sure) but I chose to start in a more adhock fashion which in the past has let me recover from accidently a second version of glibc.

Having multiple versions of glibc typically results in a segmentation fault unless one is careful to secret the incomparable versions.

Certainly in older versions of puppy such as Xenial64 the core libs such as glibc can be found in the /lib folder. What I did first was to manually move items in this folder to /lib/glib23 which duplicated the libs in WDLGO_UbuntuFocal64.

The most important ones to move were those associated with glibc 2.23.These libs are easy to identify because they have 2.23 in their name.That said for good measure I moved all libs in this folder which are part of WDLGO_UbuntuFocal64 so as to keep the updated libs. The code to do this has tje following form:

Code: Select all

while read line; do
  mv $HARDLINKED_SFS/cont/lib/$line $HARDLINKED_SFS/cont/lib/glib23/$line
done <<EOF
ld-2.23.so
ld-linux-x86-64.so.2
...
EOF

The above process of moving these essentially removes them from places where they will likely be used by an application.These libs which we've moved are the ones that are most likely to break the system due to conflicting versions. These libs were delt with seperatly because there name differs by the version number.

Now that most of the problematic libs are out of the way the next step is to replace files in our modified puli/Xenial with thise of WDLGO_UbuntuFocal64. However, the replacement is not done for the /etc folder or the /root folder so as to not overwrite puppy configuration files.

In my case I'm doing the replacement by moving the old file to a new folder for reference and then hardlinking the file from WDLGO_UbuntuFocal64. The space of the moved Puli Xenial file is minimal because it is harlinked to a prestine puli/xenial system. In effect what I'm doing here is creating three systems here to of which are prestine and one which is hybird. Hardlinks are used to save space.

The code to do this is as follows:

Code: Select all

while read a_folder; do
    mkdir -p "$MV_LOC/${a_folder}"
  for a_file in $(ls -1a $FR_SOURCE/"$a_folder");  do
 source_file="$FR_SOURCE/$a_folder/$a_file"
    target_file="$TARGET/$a_folder/$a_file"
    if [ -d "$source_file" ]; then 
      ( cd "$(dirname "$source_file")"; 
        echo "$(basename "$source_file")" | \
          cpio -pdu "$(dirname "$target_file")" )
    else
      if [ ! -e "$target_file" ] || [ $(stat -c %s "$source_file") -gt  $(stat -c %s "$target_file") ] || ! cmp --silent "$source_file" "$target_file"; then
        if [ -e "$TARGET/$a_folder/$a_file" ]; then
          mv "$TARGET/$a_folder/$a_file" "$MV_LOC/${a_folder}/$a_file"
        fi
        ln "$FR_SOURCE/$a_folder/$a_file" "$TARGET/$a_folder/$a_file"
      fi
      #  fi
      #done
    fi
  done
  
done <<EOF
lib64
var
var/log
...
EOF

Now for older versions of puppy such as Xenial the default version of some utilities is replaced by a script and the full version is renames to have the -FUL suffix. The next piece of code renames any full version of these utils supplied by WDLGO_UbuntuFocal64 with the -FULL suffix and then restores the script.

Code: Select all

while read a_file_rel_path; do
  if [[ $(file "$TARGET/${a_file_rel_path}") != *ASCII* ]]; then
    mv "$TARGET/${a_file_rel_path}-FULL" "$MV_LOC/${a_file_rel_path}-FULL"
    mv "$TARGET/${a_file_rel_path}" "$TARGET/${a_file_rel_path}-FULL"
    mv "$MV_LOC/${a_file_rel_path}" "$TARGET/${a_file_rel_path}" 
  fi
done <<EOF
sbin/losetup
bin/df
bin/umount
bin/mount
bin/ps
EOF
fi

Finally for older puppies such as Xenial the class loader needs to be located in the /lib folder. I accomplish this via symbolic links

Code: Select all

ln -s ./x86_64-linux-gnu/ld-linux-x86-64.so.2 ld-linux-x86-64.so.2
ln -s ./x86_64-linux-gnu/ld-linux-x86-64.so.2 ld-linux-x86-64.so.1
 

but likely only one of these li ks is necessary.

The current status is that I'm able to chroot into this hybird system and the core utiloties apear to work. I will test other functionality later. Here is the code that I used to create the hybird system:


Re: Draft: Minimum Puppy Upgrade via Firstrib WLGO rootfs

Posted: Fri Jan 29, 2021 7:14 pm
by s243a

It looks like I need to keep the following files from puppy/Xenial

libtinfo.so.5
libtinfo.so.5.9

bash needs these and they either aren't supplied by WDLGO_UbuntuFocal64 or alternatively my stem isn't finding them.


Re: Draft: Minimum Puppy Upgrade via Firstrib WLGO rootfs

Posted: Fri Jan 29, 2021 9:39 pm
by Keef

Often you can get away with a symlink to ncurses5.


Re: Draft: Minimum Puppy Upgrade via Firstrib WLGO rootfs

Posted: Sat Jan 30, 2021 3:43 am
by wiak
s243a wrote: Fri Jan 29, 2021 7:14 pm

It looks like I need to keep the following files from puppy/Xenial

libtinfo.so.5
libtinfo.so.5.9

bash needs these and they either aren't supplied by WDLGO_UbuntuFocal64 or alternatively my stem isn't finding them.

WDLGO_Focal64 firstrib_rootfs contains libtinfo6 (so /lib/x86_64-linux-gnu/libtinfo.so.6 which is a symlink to libtinfo.so.6.2). The WDLGO_Bionic variant I've made, but currently under test/further-developement, uses libtinfo.so.5 (EDIT: However, if I provide a somewhat minimised Puppy dpkg/apt addon that won't include libtinfo since that is only needed in WDLGO itself when providing wpasupplicant, which is not required for Puppy addon since it already has its own wpasupplicant and so on. Nevertheless I'm planning to leave libtinfo in the Puppy addon since may well be a dependency of apps that may later be installed).


Re: Draft: Minimum Puppy Upgrade via Firstrib WLGO rootfs

Posted: Sat Jan 30, 2021 3:00 pm
by s243a

There are some issues that I have to adress related to the multi-arch-hack symlink that exists in older puppies (e.g. Bionic, Xenial and older) but not in newer puppies such as FossaPup. The basic plan is:
1. If dpkg is desired then move puppy files into the multi-arch directories and patch the ppm so as to use these folders rather than recreate the symlink or alternatively upgrade the ppm.
2. If dpkg support is not required then keep the multi-arch symlink and merge the WDLGO_Focal64 firstrib_rootfs in a way to preserve this symlink.

Option #2 might cause issues with pam support (see post). However, I think that multi-user-pam support can work with the multi-arch symlink because if I recall correctly SliTaz has a simmilar folder structure as older versions of puppy. SliTaz doesn't use the Multi-Arch symlink because SliTaz compiles it's own packages. Instead SliTaz lets alien package be installed in whatever folder they were designed for in the package and SliTaz adds an item to LD_LIBRARY__PATH so that the libs can be found.

Finally I'll note that WDLGO_Focal64 firstrib_rootfs isn't the only option for my upgrade idea as I note above, it is just the first one I'm trying.


Re: Draft: Minimum Puppy Upgrade via Firstrib WLGO rootfs

Posted: Sat Jan 30, 2021 3:27 pm
by s243a

Related to the multi-arch issue (see above post). I'm posting the following for reference:

Code: Select all

               # woofce: NO_MULTIARCH_SYMLINK=1 (DISTRO_SPECS)
               if [ -z "$NO_MULTIARCH_SYMLINK" ] ; then
            if [ -f $HOME/.packages/${PKGNAME}.files ];then
              pkg_has_archdir="$(grep -m1 "$DISTRO_ARCHDIR" "$HOME/.packages/${PKGNAME}.files")"
            fi

                  # Workaround to avoid overwriting the $DISTRO_ARCHDIR symlink.
                  if [ "$DISTRO_ARCHDIR" != "" -a -f $HOME/.packages/${PKGNAME}.files -a "$pkg_has_archdir" != "" ]; then
                    mkdir -p /tmp/$PKGNAME
                    rm -rf /tmp/$PKGNAME/*
                    dpkg-deb -x ${CURDIR}/${PKGNAME}.deb /tmp/$PKGNAME/ 2> $TMPDIR/$SELF-cp-errlog
                    for f in $(find /tmp/$PKGNAME \( -type f -o -type l \))
                    do
                       xpath=$(echo "$f" |  cut  -f 4-30 -d "/" | sed "s/$DISTRO_ARCHDIR\///")
                       mkdir -p ${DIRECTSAVEPATH}/$(dirname "$xpath")
                       cp -a "$f" ${DIRECTSAVEPATH}/$(dirname "$xpath")/
                    done
                    rm -rf /tmp/$PKGNAME &>/dev/null
                  else
                    dpkg-deb -x ${CURDIR}/${PKGNAME}.deb ${DIRECTSAVEPATH}/ 2> $TMPDIR/$SELF-cp-errlog
                  fi
                else
                  dpkg-deb -x ${CURDIR}/${PKGNAME}.deb ${DIRECTSAVEPATH}/ 2> $TMPDIR/$SELF-cp-errlog
                fi

http://murga-linux.com/puppy/viewtopic. ... 34#1031018

This looks to be (old?) code from pkg which preserves the multi-arch symlink when installing a package. It isn't the anyoying code that I mentioned elsewhere that will move packages from the multi-arch directories and recreate the multi-arch sylink. Said anoying code was removed in later versions of the ppm and package.

On a (somewhat?) related note JRB, has a slightly modified version of the legacy banch of Woof-CE (see post) which presumably keeps the multi-arch symlink.


Re: Draft: Minimum Puppy Upgrade via Firstrib WLGO rootfs

Posted: Sun Jan 31, 2021 2:27 am
by rockedge

A while go I loaded a WeeDog-Void's firstrib_rootfs.sfs into a Puppy Linux Fossapup64-9.0.5. The firstrib_rootfs.sfs was built using the WeeDog scripts with no X server included.
At that time I think I ran into some snags and while I was putting together a WeeDog-Void ISO, I put that experiment to the side. But now I might revisit the experiment and build a fresh No-X WeeDog-Void really basic and see what happens if I load the firstrib_rootfs into a Fossapup64.

At this time I am using the current Firstrib WLGO rootfs on a fully loaded Fossapup64 booted with the full complement. Went through the same procedure a bit refined and installed the LAMP and Zoneminder with apt successfully. Some manual work was needed but it is rare that a Zoneminder installation doesn't need some manual configuration.

Also to note is Fossapup64 has Hiawatha 10.8.3 built in. Hiawatha is one of the best web servers out there. So on this system I can turn off Apache and run Hiawatha or vise versa and run Zoneminder.


Re: Draft: Minimum Puppy Upgrade via Firstrib WLGO rootfs

Posted: Sun Jan 31, 2021 4:44 am
by wiak
rockedge wrote: Sun Jan 31, 2021 2:27 am

But now I might revisit the experiment and build a fresh No-X WeeDog-Void really basic and see what happens if I load the firstrib_rootfs into a Fossapup64.

I think you'll run into trouble with that idea rockedge. WDL_Void follows Void Linux filesystem structure and FossaPup64 seems to follow Debian filesystem structure so won't match. Earlier Debian-based pups seem to adopt their own filesystem structure so match none of the above... Anyway, for dpkg/apt/PAM addon I'm about to build that one-off special WDLGO_Bionic32 with BionicPup32 filesystem structure to match that - I think that should work thereafter - but only testing will confirm that. Not enough hours in the day to manually hack things to work so may well be my only attempt for that one since other things I need to get on with - main thing is that any such WDLGO addon will have the correct files inside it, and that's the harder part to get right really (in terms of getting dpkg database registering everything correctly - without that any such assembly is pretty much worthless in practice) - after that just a matter of merging in to Puppy the way it needs in terms of symlinks (or no symlinks...) really, so should still be useful. However, I tend to think the one-off addon I'm about to produce may well work out-of-the-box now that I know the issue.

Note that my dpkg/apt addon knows (and cares) nothing about what Puppy itself already has installed. It can't and isn't necessary that it does - the essential ingredient is as I mention above that its dpkg database correctly knows exactly what is installed in the addon itself (including the package dpkg...). It is far from easy to ensure that dpkg database integrity. One way is to start from a minimum debootstrap install of course, but that is far too big for what I wanted for WDLGO (though I do use debootstrap for normal WeeDogLinux Debian-based builds - WDLGO is something special in terms of minimum Debian-base though and built from ground up without debootstrap with a view towards absolute minimum dpkg/apt addon size possible, which it has successfully and I believe uniquely achieved). Anyway, back to the BionicPup32 version to see if my new plan works.


Re: Draft: Minimum Puppy Upgrade via Firstrib WLGO rootfs

Posted: Sun Jan 31, 2021 6:24 am
by s243a
wiak wrote: Sun Jan 31, 2021 4:44 am

WDLGO is something special in terms of minimum Debian-base though and built from ground up without debootstrap with a view towards absolute minimum dpkg/apt addon size possible, which it has successfully and I believe uniquely achieved). Anyway, back to the BionicPup32 version to see if my new plan works.

I think this is useful in its own right. By merging the rootfs from "WDLGO_UbuntuFocal64" into Puli/Xenial I was able to get ungoogled-chromium working within the resulting chroot environment. It's working currently as root (--no-sandbox) but I'll also try to get it working with spot.

Given that WDLGO is in the 27-47MB range in my mind this is quite a small upgrade in order to get the browser working.

Since Xenial is an older puppy it has a symlink for the multiarch directory. I took the approach of replacing the symlink with an actual directory and then moving any subdirectories within these lib directories into the corresponding multi-arch directory except where WDLGO didn't have the directory as a sub directory of the multi-arch directory. I also created a symlink from the original location to the new location within the multi-arch directory.

Here is the code section where I do this:

Code: Select all

       rel_path="$(dirname "$arch_dir_rel")/$b_name"
       case "$rel_path" in
       lib/lsb|lib/systemd|lib/terminfo) continue; ;;
       usr/lib/apt|usr/lib/debootstrap|usr/lib/dpkg) continue; ;;    
       usr/lib/locale|usr/lib/mime|usr/lib/NetworkManager) continue; ;;
       usr/lib/pm-utils|usr/lib/ssl|usr/lib/tmpfiles.d) continue; ;;
       esac
       [[ "$target_arch_parent/$b_name" = "$target_arch_dir" ]] && continue
       mv "$target_arch_parent/$b_name" "$target_arch_dir/$b_name"
       ( cd "$target_arch_parent"; ln -s "$arch_dir/$b_name" "$b_name" )

The complete code is:

Regarding pam support, I don't understand it enough to know if I need it. I am interested in dpkg support but getting the metadata right might require some care.

Also note, that I created a system only to run a browser at this point. The ppm will break my system until I patch or upgrade it.


Re: Draft: Minimum Puppy Upgrade via Firstrib WLGO rootfs

Posted: Sun Jan 31, 2021 8:53 am
by wiak

My BionicPup32 addon(s) sfs are now working by the way (keeping the standard Pup symlink libs in a one-off scripted build). Not sure if these are of any help to your Xenial upgrade for Chromium and so on. Still a few days before I publish the addon sfs files per post below:

viewtopic.php?p=16434#p16434


Re: Draft: Minimum Puppy Upgrade via Firstrib WLGO rootfs

Posted: Sun Jan 31, 2021 3:19 pm
by s243a
wiak wrote: Sun Jan 31, 2021 8:53 am

My BionicPup32 addon(s) sfs are now working by the way (keeping the standard Pup symlink libs in a one-off scripted build). Not sure if these are of any help to your Xenial upgrade for Chromium and so on. Still a few days before I publish the addon sfs files per post below:

viewtopic.php?p=16434#p16434

I might try upgrading the Wary Version of Puppy Arcade with your BionicPup32 add-on. First I need to finish my Puli/Xenial64 experiment.


Re: Draft: Minimum Puppy Upgrade via Firstrib WLGO rootfs

Posted: Sun Jan 31, 2021 5:43 pm
by rockedge
wiak wrote:

I think you'll run into trouble with that idea rockedge. WDL_Void follows Void Linux filesystem structure and FossaPup64 seems to follow Debian filesystem structure so won't match.

I know. The experiment was more to see what damage it does and identify how stripped down I would need the Fossapup64 to be to actually integrate xbps and have it work at all. I did manage to get xbps to download the Ubuntu repos though...but my notes are thin and I'm not sure how I managed that at the moment. More or less went deep down the rabbit hole and met up with the Cheshire Cat and the Hookah-Smoking Caterpillar.

I am excited to test out the Bionic32 plugin! So I am setting up the Bionic32's in 2 places to begin the tests.


Re: Draft: Minimum Puppy Upgrade via Firstrib WLGO rootfs

Posted: Sun Jan 31, 2021 11:55 pm
by wiak
rockedge wrote: Sun Jan 31, 2021 5:43 pm
wiak wrote:

I think you'll run into trouble with that idea rockedge. WDL_Void follows Void Linux filesystem structure and FossaPup64 seems to follow Debian filesystem structure so won't match.

I know. The experiment was more to see what damage it does and identify how stripped down I would need the Fossapup64 to be to actually integrate xbps and have it work at all. I did manage to get xbps to download the Ubuntu repos though...but my notes are thin and I'm not sure how I managed that at the moment. More or less went deep down the rabbit hole and met up with the Cheshire Cat and the Hookah-Smoking Caterpillar.

I'm not saying it is impossible - a special WDL_Void that adopted an underlying FossaPup64 filesystem structure 'might' work (to a greater or lesser extent) using its firstrib_rootfs to provide xbps facility to FossaPup (assuming similar modifications I made for BionicPup32 dpkg_apt addon work in practical tests). The method is to change the build_firstrib_rootfs script (for that one-of special WDL_Void creation) to use the FossaPup (debian) filesystem arrangement (meaning not using symlinks for so many directories like normal Void does). The special WDL_Void creation itself would not likely work standalone without further tinkering to provide the system with new exec and lib PATH info to compensate for such files no longer being reachable via in-PATH symlinks. However, its firstrib_rootfs would likely work in FossaPup in terms of the latter finding everything since it already has its exec and lib PATHS set up for that official Debian-style filesystem arrangement. Having said that, maybe the experiment would also meet up with the Cheshire Cat and so on ;-)


Re: Draft: Minimum Puppy Upgrade via Firstrib WLGO rootfs

Posted: Mon Feb 01, 2021 6:03 am
by wiak
rockedge wrote: Sun Jan 31, 2021 2:27 am

A while go I loaded a WeeDog-Void's firstrib_rootfs.sfs into a Puppy Linux Fossapup64-9.0.5. The firstrib_rootfs.sfs was built using the WeeDog scripts with no X server included.
At that time I think I ran into some snags and while I was putting together a WeeDog-Void ISO, I put that experiment to the side. But now I might revisit the experiment and build a fresh No-X WeeDog-Void really basic and see what happens if I load the firstrib_rootfs into a Fossapup64

But I think the easiest and likeliest way to use xbps with Puppy is simply to compile a version into Puppy (is that not what I noticed you trying before?). Easier still, and possibly the approach I would adopt is to use the static version of xbps that is used in WeeDogLinux Void flavour builds.

The other thing that can always be useful is simply to use pretty much any 01firstrib_rootfs.sfs unsquashed and renamed as simply firstrib_rootfs and then can be used from Puppy or pretty much any Linux OS in a chroot. You don't even need to know anything much about making chroot systems, just use that firstrib_rootfs uncompressed directory with WeeDogLinux-provided ./mount_chroot-latest.sh script. I often experiment with firstrib_rootfs that I am building via that process since often in different host Linux OS whilst working. If doing so you need of course to remember that, when finished with the chroot, you run ./umount_chroot-latest.sh to remove the mounts you no longer need...

To run X applications from the chroot on the host system display you do generally need to use various techniques to pass the DISPLAY value and X authenticity to the chroot (or use vnc or similar thin-client solutions). There have been various posts about such 'tricks' (some by me over the years; I've seen others by rufwoof) so you'd need to search for these, and make appropriate HowTo or, even better, script additions if you succeed ;-)


Re: Draft: Minimum Puppy Upgrade via Firstrib WLGO rootfs

Posted: Mon Feb 01, 2021 3:42 pm
by rockedge

Yes I compiled XBPS successfully in Fossapup64-9.0.5

That is the version I've been experimenting with. All the components are installed and work. I can even perform

Code: Select all

xbps-install -Suy

But I promptly broke that system by observing what happens when attempting to install xterm from the Void repos. went almost all the way but the Void glibc replacing the Ubuntu version wrecks it. Locked up..hard stop needed to reboot. But never expected it to work all the way.
So I restored the system (Puppy is so easy to do this type of work with!) and tinkered with it until XBPS started to download the Ubuntu repo data. Although as one can imagine that only worked up to that point. For the coup de grâce I tried then to install xterm to watch what would happen. Of course it crashed and did not go further than a blinking cursor.
I again restored the system and compiled xbps again. (Goes really smoothly now). I have included the firstrib_rootfs.sfs I've been using again and have apt fully working. I run into some errors lately using Pkg in Fosssapup64 updating the repos. So on a fully loaded Fossapup64 I've compiled XBPS which works somewhat, APT which totally works and Pkg and PPM also functioning (Pkg having some grep issue updating the repos).

I then installed completely a LAMP from scratch with APT only and then Zoneminder with APT. Went smoothly and both are functioning well and as expected.
Now I can switch web servers between Apache2 and the included in Puppy, Hiawatha v10.8.3 and run the web server and Zoneminder with either one.

XBPS I cloned from github and followed the build instructions, which are basic.
So I can say that XBPS so far will compile and run. I am considering using a xbps compiled on Fossapup64 then using that in a heavily modified Puppy Linux that you described. So far just pretty amazed it compiles and runs.

Sitting in a blizzard right now so looks like I'll be here tinkering.