Puppy4: How to Remove Built-In Packages from an Extracted SFS

[s243a]

I decide to start a new thread for this specific how-to because it might distract from the discussion around the motivation of why one might want to do this. For a discussion of the motivation see:
"Updating Puppy 4, for learning and nostalgia"

It appears that for puppy 4, the list of files that make up the built-in packages aren't included...at least not to my knowledge. So for AttackPup, I created a list of files for the Built-In core/cli components:

cli_builtin_files-puppy4%28attackpup%29.tar.xz (also attached in an unfortunately a less compressed format)

I developed a list of core and CLI type file that either are typically built-into puppy but I also added a few files which aren't (e.g. strace and midnight commander) that I just wanted to make sure they were removed for good measure because I wanted to include these as part of a CLI based system. The list was originally developed by looking at /var/packages/DISTRO_PKGS_SPEC of upupGG+D groovy pup and so the list should be widly applicable if I say wanted to use it to isolate or remove a CLI from a newer puppy but then I looked to see what I missed when trying to use this to remove files from attack pup. The list is:

the package to be removed is on the left and on the right there is a comment (starting with a "#") that gives information about the DISTRO_PKGS_SPEC line I borrowed it from. Most of these comments will start with "#upupGG+D: " but there a few items that start with "#puppy4 that are specific to either puppy 4 or a derivative (e.g. attackpup).

Prior to removing these files I modify puppy4 to have the package infromation in the same place as newer versions of puppy. So /root/.packages is moved to /var/packages.

Code: Select all

if [ -d ./root/.packages ]; then
  #mkdir -p ./var/packages
  mv ./root/.packages ./var/packages
  cd ./root
  ln -s ../var/packages .packages
  cd "$CWD"
fi

and then I patch in the missing list of files for each built-in package:

Code: Select all

[ ! -z "$BUILTIN_FILE_LIST_PATCH" ] && cp -arfv --remove-destination "${BUILTIN_FILE_LIST_PATCH}"/* "$TARGET"

Finally, I remove the list of built-in files:

Code: Select all

CHROOT_DIR="$TARGET" bash -x ./remove_builtin.sh --pkg-list uninstallsfs "$TARGET"

* -x option not necessary, last argument "$TARGET" isn't implemented yet so export CHROOT_DIR instead.

The function to remove the built-in files is here:

Once the desired built-in files are removed one might want to replace them with a different version (e.g. a newer version). To do this one could likely just copy over (or under) said files.

That said I created a more involved script that:
1. takes some care in both /etc and /root of which files to use
2. extracts both the replacement files and the puppy that is having certian built-ins packages removed
3. Sometimes (depending on the specified options) Modifies /etc/profile and /etc/ld.so.conf to use muti-arch subfolders.
4. Sometimes (depending on the specified options) moves files to the multi-arch folders.
5. Moves the ./packages folder to the new location, adds missing metadata for built-in files and calls the remove_builtin.sh function mentioned above.

Here is the complete script:

This last script "mv_extracted_stuff.sh" was first mentioned in the following post,

viewtopic.php?p=16431#p16431

and was used to upgraded Puli/Xenial with files from "WDLGO_UbuntuFocal64.

[s243a]

Not mentioned in my previous post was how I built the missing file lists for each built-in package. What I did was for each file in my remove list, I extracted the related package at:

https://distro.ibiblio.org/puppylinux/pet_packages-4/

and used a command of the form

Code: Select all

find . -name '*' | sed -r 's#^[.]##' 

I then checked to see what files failed to be removed in the /lib directory and modified the file list accordingly. Note that sometimes, there were multiple versions of a package on ibiblio in the puppy 4 repo. In these cases I looked in the file woof-installed-packages to see which version to use as the starting point for my file list.

Puppy 4 likely has the packages used to build the puppy in this folder but newer versions of puppy may not. This is because for newer versions of puppy they tried to use common 32bit, common 64bit repos for each version of puppy, excluding any package that was taken from the compatible binary distribution. The problem is that there are no snapshots of these repos and as a consequence older versions of puppy may fail to build in woof-CE.

Fortunately, though puppies newer than puppy 4 have the missing metadata so one can in theory extract the package from the iso and create legacy repos that one may be able to use to build these puppy via woof-CE.

On another note, I couldn't find all packages I needed to remove in the puppy 4 repo so In these cases I looked at packages for an older version of linux (e.g. warry or jessie) as the starting point of building my list of files for a given builtin package.

[s243a]

On another note, my list of packages to remove, in order to gut the core/cli components of puppy is larger than the number of packages contained in "puppy 4" and derivatives. It might be instructive to see which of these packages are actually in puppy4. In attack pup here is the list or core/cli utilties that I'm gutting for the purpose of modification:

Code: Select all

# ls -A
bdb        e2fsprogs  glib      libstdc++  perl-digest-sha1  squashfs-tools4
bzip2      expat      glibc     libusb     perl-html-parser  tar
coreutils  file       grep      ncurses    procps            util-linux
cpio       findutils  gzip      ntfs-3g    psmisc            wget
curl       fuse       iptables  openssl    readline          wireless-tools
dhcpcd     gawk       keyutils  parted     sed               wpa_supplicant
dialog     gcc        libcap    pcre       sqlite            zlib
diffutils  gettext    libidn    perl       squashfs-tools

this list might be another starting point of which packages that one might want to include for a somewhat minimal cli based system. Of course not all packages are necessary. In many cases the busybox version can be used instead. Puppy seems to be favouring the busybox version of packages rather than the full utilities and in most cases this seems to work for me with the exception of realpath and tar. If your instance you search upupGG+D for '*-FULL' you will see in most cases the full utility has been renamed (at least in the base sfs file) and a symlink to busybox was created to be used instead of the full version.

This I suppose is helpful for testing to see if a script will work with the busybox version of the utilities but in many cases one might want to use the full utilities instead. In my mv_estracted_stuff.sh script (mentioned above) I wrote code to create an override directory that will override the slected packages with the full version.

Code: Select all

mkdir -p "$TARGET"/full-utils
if [ "$USE_NON_FULL_VERSIONS" = yes ]; then
  while read a_file_rel_path; do #EXTRACTED_SFS_FP
    if [[ $(file "$TARGET/${a_file_rel_path}") != *ASCII* ]]; then
      #mv "$TARGET/${a_file_rel_path}-FULL" "$MV_LOC/${a_file_rel_path}-FULL"
      #mv "$TARGET/${a_file_rel_path}" "$TARGET/${a_file_rel_path}-FULL"
      #mv "$MV_LOC/${a_file_rel_path}" "$TARGET/${a_file_rel_path}" 
    
      #mv "$TARGET/${a_file_rel_path}-FULL" "$MV_LOC/${a_file_rel_path}-FULL"
      [[ ! -e "$EXTRACTED_SFS_FP/${a_file_rel_path}-FULL"  ]] && continue
      if [[ $(file "$EXTRACTED_SFS_FP/${a_file_rel_path}") != *ASCII* ]]; then
        if [[ $(readlink "$EXTRACTED_SFS_FP/${a_file_rel_path}") = *busybox* ]] || [[ $(realpath "$EXTRACTED_SFS_FP/${a_file_rel_path}") = *busybox* ]]; then
          [ ! "$USE_BB_FOR_nonFULL" = yes ] && continue
        else
          continue #TODO think of a case where we might not want to continue here. 
        fi
      fi  
      [ -e "$TARGET/${a_file_rel_path}-FULL" ] && rm "$TARGET/${a_file_rel_path}-FULL"
      mv "$TARGET/${a_file_rel_path}" "$TARGET/${a_file_rel_path}-FULL"
      cp -aP "$EXTRACTED_SFS_FP/${a_file_rel_path}" "$TARGET/${a_file_rel_path}" 
      util_name="$(basename "$TARGET/${a_file_rel_path}")"
      for a_util in "${FULL_UTILS[@]}"; do
        if [ "$util_name" = "$a_util" ]; then
          ( cd "${TARGET}/full-utils"; ln -s ../"${a_file_rel_path}" "$a_util" )
        #else
        #  ( cd "${TARGET}/full-utils"; ln -s ../"${a_file_rel_path}-FULL" "${a_util}-NOTUSED" )
        fi
      done
      [ ! -e  "${TARGET}/full-utils/$util_name" ] && \
        [ ! -e  "${TARGET}/full-utils/${util_name}-NOTUSED" ] && \
          ( cd "${TARGET}/full-utils"; ln -s ../"${a_file_rel_path}-FULL" "${util_name}-NOTUSED" )
    fi
done <<EOF
sbin/losetup
sbin/pivot_root
sbin/sulogin
sbin/switch_root
bin/df
bin/echo
bin/false
bin/more
bin/mount
bin/ps
bin/sleep
bin/su
bin/sync
bin/true
bin/umount
usr/bin/basename
usr/bin/env
usr/bin/expr
usr/bin/[
usr/bin/logger
usr/bin/mesg
usr/bin/realpath
usr/bin/sha1sum
usr/bin/sha256sum
usr/bin/tee
usr/bin/tr
usr/bin/unlink
usr/bin/whoami
usr/bin/who
usr/bin/yes
usr/sbin/chroot
usr/sbin/fdformat
usr/sbin/readprofile
usr/sbin/rtcwake
EOF
fi

and so if one wants to over-ride specific busybox version of a linux cli utility with the full utilities then they can append this directory to the front of PATH. My script does overriding this in /etc/profile. Utilities which aren't over-riden are sylinked to the full utility with -FULL appended to the name. This is so that people can easily change which overrides to use by modifying the name of the symlink.

Certainly, it's good that puppy will work primary with busybox utilities but often users (e.g. @musher0 ) prefer the full versions of these utilities and @wiak chose to use the the full versions of these utilities in wdlgo and still produced quite a small system.

Of course there is the question of which full-utilities to include. For instance in modern puppies include bsdutils as part of the util-linux builtin but in puppy4 the busybox versions of these utilities are used instead.