Parted doesn't work with loop devices

[svartkanin]

This isn't an issue for the installer per se but rather for testing.
The README instructions currently refer to creating an image and then setting it up as a loop device. But such device can't be listed by parted and therefore also not pyparted and archinstall.

https://git.savannah.gnu.org/gitweb/?p=parted.git;a=blob;f=libparted/arch/linux.c;h=ccbba8656bc7ef0663aea6431bfb8873deccd0ef;hb=HEAD#l2304

Maybe a question for @Torxed would you be aware of other options to get a test device setup that could be used in tests?

[Torxed]

I'll have a dig at it when I come back home this weekend :)

[codefiles]

I just finished typing this up, took some time to put it all down. Let me know if it was worth it.

What @svartkanin linked is just a function to skip listing them. Partitioning loop devices with parted is possible. Here is an example that includes listing a loop device with lsblk and losetup as well as partitioning it with parted.

# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda    254:0    0   20G  0 disk
├─vda1 254:1    0    1G  0 part
└─vda2 254:2    0   19G  0 part /
# losetup -a
# truncate -s 2G example.img
# losetup -P --show -f example.img
/dev/loop0
# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0    7:0    0    2G  0 loop
vda    254:0    0   20G  0 disk
├─vda1 254:1    0    1G  0 part
└─vda2 254:2    0   19G  0 part /
# losetup -a
/dev/loop0: [65026]:393281 (/root/example.img)
# parted --script /dev/loop0 \
> mklabel gpt \
> unit mib \
> mkpart boot fat32 1 513 \
> mkpart root ext4 513 100% \
> set 1 boot on
# lsblk
NAME      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0       7:0    0    2G  0 loop
├─loop0p1 259:0    0  512M  0 part
└─loop0p2 259:1    0  1.5G  0 part
vda       254:0    0   20G  0 disk
├─vda1    254:1    0    1G  0 part
└─vda2    254:2    0   19G  0 part /
# losetup -a
/dev/loop0: [65026]:393281 (/root/example.img)

It has been a while but I had this working in my testing. Here is the first crude patch I used to add all loop devices to the list of available devices: 5177958. There is more testing I did locally and in a private repo to work on the details. I am still interested in getting this together and was waiting for pull request merges to pick back up before adding pull requests.

What I have discovered is it may not be possible to do within a Docker image like is being used for the other GitHub workflows for the repository. The gist is there does not appear to be the ability to get updated information about the loop device partitions after creating them (no UUID or PARTUUID available I think maybe other stuff or everything, I do not recall the exact details).

The workaround I used was to create a chroot of Arch Linux in a GitHub Ubuntu virtual machine (not a Docker image) workflow. I used Method B since archinstall will generally be run from the LiveCD image and it will already have all of the needed packages.

So with a patch to add loop devices to the list of all devices, and scripting to set up the chroot and prepare the image file, it is possible within a GitHub workflow. From there you can produce an image file of the installation that can be booted with QEMU in a GitHub Ubuntu vm workflow or the same workflow the image was created in. As @Torxed pointed out here #1114, the Arch Linux ci script build_in_archiso_vm.sh is a good way to do this. I used that with some inconsequential modifications in my testing.

Also to get the resulting image to boot with QEMU I had to boot a fallback initramfs (it has "a full range of modules") or configure mkinitcpio to have the additional hook ata_piix and generate the initramfs again. If not the boot loader would not detect the drive. Might just be how QEMU was configured for the drive in my command line arguments?

Maybe Arch Linux DevOps can set up an Arch Linux virtual machine as a runner?

[Torxed]

I'll for sure ask them, I know this would be very possible on the GitLab instance and other things are prepped for a move there at a later time (currently very early discussions because the community here is very awesome.. so no move yet).

They have been asked now :) great work and I'd love a runner with your work in it! The quemu arguments probably matter, but mostly the PCI bus and blockdev setup if everything is set up as 1:1 to match the host in the other regards :)

[svartkanin]

@codefiles I probably didn't describe it properly but yes what I meant is that -l wouldn't list the partitions with parted and therefore archinstall wouldn't be able to list them as options.
But your change seems to address that, even tho I was hoping to be able to avoid such "manually" detection using a library now :( but there doesn't seem to be a way around it

[codefiles]

The quemu arguments probably matter, but mostly the PCI bus and blockdev setup if everything is set up as 1:1 to match the host in the other regards :)

@Torxed I may be mistaking testing I did locally with QEMU where I needed to boot a fallback initramfs with testing I also did running it remote as a workflow. I probably created an install using a physical machine as a host with the image to be booted as a VM in QEMU. In that case mkinitcpio would pick up the host machine modules instead of modules it would pick up, and possible need, if it was running in QEMU. That was probably the issue I am thinking about with the ata_piix module.

@codefiles I probably didn't describe it properly but yes what I meant is that -l wouldn't list the partitions with parted and therefore archinstall wouldn't be able to list them as options.

@svartkanin No you explained it fine. I figured you might have thought that they would need to be in the listing to partition with parted/pyparted.

But your change seems to address that, even tho I was hoping to be able to avoid such "manually" detection using a library now :( but there doesn't seem to be a way around it

What library? It works with losetup which is in the util-linux package. That package is already required and is in the PKGBUILD dependency list. Could also work with lsblk which is in use in the code and is from util-linux too.

[svartkanin]

I meant to use the pyparted library alone and that the getAllDevices will just magically find everything so we don't have to SysCommand things :)

[svartkanin]

@codefiles do you want to raise a PR to get the loop device handler in (the commit you linked)? Maybe with a try-except and only logging any problems but just continuing with the other devices?

[codefiles]

Yea, I will submit it in a bit. Good idea on the try-except and logging, thanks for the suggestion.

[klausenbusk]

What I have discovered is it may not be possible to do within a Docker image like is being used for the other GitHub workflows for the repository. The gist is there does not appear to be the ability to get updated information about the loop device partitions after creating them (no UUID or PARTUUID available I think maybe other stuff or everything, I do not recall the exact details).

Assuming you are hitting this issue moby/moby#27886, try mounting /dev:/dev in the container as a workaround (syntax documented here).

[codefiles]

I am aware of that and tried it, 64e7865. Using that suggestion, it is then possible to see the loop devices but as I stated it is not possible to get new information about partitions that were created. Could you create a simple test example to demonstrate use of the information you provided? In an Arch Linux Docker container create an image preferably with truncate, then set up the image as a loop device with losetup, partition it and format the partition(s). While still in the Docker container print the UUID and PARTUUID for the new partition(s). Finally can you run the mount command (as archinstall will attempt to) inside the Docker container and mount the loop device partition(s)?

[klausenbusk]

It works on my machine ;)

$ sudo docker run --privileged --rm -t -i -v/dev:/dev archlinux bash
[root@8d0551abe484 /]# cd
[root@8d0551abe484 ~]# truncate -s1G foo.img
[root@8d0551abe484 ~]# losetup --find --show --partscan foo.img
/dev/loop0
[root@8d0551abe484 ~]# fdisk /dev/loop0

Welcome to fdisk (util-linux 2.36.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x23b7adff.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): 

Using default response p.
Partition number (1-4, default 1): 
First sector (2048-2097151, default 2048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-2097151, default 2097151): 

Created a new partition 1 of type 'Linux' and of size 1023 MiB.

Command (m for help): w

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

[root@8d0551abe484 ~]# mkfs.ext4 /dev/loop0p1
mke2fs 1.46.2 (28-Feb-2021)
Discarding device blocks: done                            
Creating filesystem with 261888 4k blocks and 65536 inodes
Filesystem UUID: 01fcf907-1b92-48ed-bdb8-31f4d28f6af2
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

[root@8d0551abe484 ~]# lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
loop0           7:0    0     1G  0 loop  
└─loop0p1     259:0    0  1023M  0 part  
[redacted]
[root@8d0551abe484 ~]# blkid
[redacted]
/dev/loop0p1: UUID="01fcf907-1b92-48ed-bdb8-31f4d28f6af2" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="23b7adff-01"
[root@8d0551abe484 ~]# mount /dev/loop0p1 /mnt
[root@8d0551abe484 ~]#

[Torxed]

We we discussed on the Arch devops channel, it's probably down to the way GitHub prohibits or overrides -v?
As previously tested here: b2ad906 which failed.

[codefiles]

Thanks, I could not get it to work in a GitHub workflow. I should have been more clear that I meant demonstrate it working in a GitHub workflow. The chroot method that I had mentioned does work in a GitHub workflow though.

[codefiles]

This should work with the pull request I submitted but the sample configuration file is broken and probably needs to be updated.

truncate -s 5G example.img
loop_device=$(losetup --partscan --show --find exmaple.img)
sed -i "s|/dev/sda|$loop_device|" examples/config-test.json
archinstall --config examples/config-sample.json --creds examples/creds-sample.json --silent

[klausenbusk]

Thanks, I could not get it to work in a GitHub workflow. I should have been more clear that I meant demonstrate it working in a GitHub workflow. The chroot method that I had mentioned does work in a GitHub workflow though.

So apparently github starts the container like docker run archlinux tail -f /dev/null and then uses docker exec to execute the commands. For some reason the partition table is not updated when using exec.

Edit: Maybe the whole thing is just racy and that is the issue🤷