Early experiences with PXE (net-)boot of KVM VMs on Ubuntu for s390x

Canonical

on 20 December 2017

Tags: IBM , kvm , mainframe , s390x , Virtual Machines

This article was last updated 3 years ago.

This article originally appeared on the Ubuntu On Big Iron blog

This article mainly assumes that an Ubuntu Server 17.10 (artful) is installed in LPAR.

Ubuntu 17.10 comes with a qemu-kvm version that is new enough to act as a KVM host that also allows to PXE netboot KVM virtual machines.

Any potentially different or additional steps for Ubuntu Server 16.04(.x) are marked in italics.

PXE boot will only work on Ubuntu 16.04 if the qemu-kvm package from the Pike Ubuntu Cloud Archive (UCA) is used.

Prepare virtualization packages

Note:
On Ubuntu 16.04.x qemu-kvm from the Pike Ubuntu Cloud Archive (UCA) need to be installed.
The Pike Cloud Archive can be enabled like this:

$ sudo add-apt-repository cloud-archive:pike
$ sudo apt update

On Ubuntu 17.10 (or higher) just install the default qemu-kvm and related packages with:

$ sudo apt install -qq -y libvirt-daemon-system qemu-kvm

The just installed packages modify the underlying environment, hence it’s recommended to quickly logout and login again to make the changes take effect.

Prepare s390 netboot ROM

$ sudo apt -y -qq install ubuntu-dev-tools

$ pull-lp-source qemu artful
pull-lp-source: Downloading qemu version 1:2.10+dfsg-0ubuntu3.1
...

$ sudo cp qemu-2.10+dfsg/pc-bios/s390-netboot.img /usr/share/qemu/

Prepare a tftp server

Now a tftp server is needed for PXE netboot, so let’s install and configure tftp:

$ sudo apt -y -qq install tftpd xinetd tftp
...
$ cat << EOF | sudo tee /etc/xinetd.d/tftp
service tftp
{
 protocol        = udp
 port            = 69
 socket_type     = dgram
 wait            = yes
 user            = nobody
 server          = /usr/sbin/in.tftpd
 server_args     = /srv/tftp
 disable         = no
}
EOF

Create the tftp folder, copy over the pxelinux.0 file, adjust the folder options and (re-)start the service:

$ sudo mkdir -p /srv/tftp
$ sudo chmod -R 777 /srv/tftp
$ sudo chown -R nobody: /srv/tftp
$ sudo systemctl restart xinetd

By now you should be able to check if tftp is listening to your local port 69 on the default libvirt networks IP of your host:

$ nc -uvz 192.168.122.1 69
Connection to 10.245.236.13 69 port [udp/tftp] succeeded!

Prepare libvirt network to provide PXE

Since we have slightly different network needs, we will modify the default network a bit,
which was created for you on install. In this step we add the lines in bold:

It's not too bad to do a quick backup of the virtual network configuration at the beginning:

$ virsh net-dumpxml default > net-default.xml

Now do the needed tftp and bootp modifications:

$ virsh net-edit default
<network>
 <name>default</name>
 <uuid>cfa3d2de-4c67-4f8f-91eb-69502e59e64d</uuid>
 <forward mode='nat'/>
 <bridge name='virbr0' stp='on' delay='0'/>
 <mac address='52:54:00:08:6f:33'/>
 <ip address='192.168.122.1' netmask='255.255.255.0'>
 <strong><tftp root='/srv/tftp'/></strong>
  <dhcp>
   <range start='192.168.122.2' end='192.168.122.254'/>
   <strong><bootp file='pxelinux.0' server='192.168.122.1'/></strong>
  </dhcp>
 </ip>
</network>
</pre>

And restart the virtual network again:

$ virsh net-destroy default
Network default destroyed

$ virsh net-start default
Network default started

Prepare netboot image

Before we can perform a PXE netboot, a 'pxelinux.0' netboot image is needed.
We will now create a pxelinux.0 file that is based on the Ubuntu 16.04.3 (xenial) installer.

Now, download the latest xenial installer files (kernel and initrd):

$ wget --quiet http://ports.ubuntu.com/ubuntu-ports/dists/xenial-updates/main/installer-s390x/current/images/generic/kernel.ubuntu http://ports.ubuntu.com/ubuntu-ports/dists/xenial-updates/main/installer-s390x/current/images/generic/initrd.ubuntu

And create a pxelinux.0 file out of these kernel and initrd installer files:For the PXE image creation a separate tool is needed from the s390-tools collection:

$ git clone --quiet https://github.com/ibm-s390-tools/s390-tools.git
...
$ ./s390-tools/netboot/mk-s390image kernel.ubuntu -r initrd.ubuntu pxelinux.0
$ ll pxelinux.0
-rw-rw-r-- 1 ubuntu ubuntu 15346314 Dec  5 05:12 pxelinux.0
$ sudo cp pxelinux.0 /srv/tftp
$ sudo chmod -R 777 /srv/tftp
$ sudo chown -R nobody: /srv/tftp

It's now time for a little download test:

$ cd /tmp/

$ echo 'get pxelinux.0' | tftp 192.168.122.1

tftp> Received 15453177 bytes in 0.3 seconds
 $ cd -

Prepare guest definition

Now proceed with the creation of a KVM virtual machine definition for the PXE netboot VM:

$ cat << EOF > netboot.xml
<domain type='kvm'>
 <name>netboot</name>
 <memory unit='KiB'>2097152</memory>
 <vcpu placement='static'>1</vcpu>
 <os>
  <type arch='s390x' machine='s390-ccw-virtio-xenial'>hvm</type>
  <boot dev='network'/>
 </os>
<devices>
 <emulator>/usr/bin/kvm</emulator>
 <disk type='file' device='disk'>
  <driver name='qemu' type='qcow2'/>
  <source file='/var/lib/libvirt/images/netboot.qcow2'/>
  <target dev='vda' bus='virtio'/>
 </disk>
 <interface type='network'>
  <source network='default'/>
  <model type='virtio'/>
 </interface>
 <console type='pty'>
 <target type='sclp' port='0'/>
  </console>
 </devices>
</domain>
EOF

Let's also quickly create a virtual disk that is already part of the VM definition above:

$ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/netboot.qcow2 5G
Formatting '/var/lib/libvirt/images/netboot.qcow2', fmt=qcow2 size=5368709120 cluster_size=65536 lazy_refcounts=off refcount_bits=16

And now define the virtual machine named 'netboot':

$ virsh define netboot.xml 
Domain netboot defined from netboot.xml

Finally start the netboot guest to prove that PXE netboot is generally working:

$ virsh start netboot --console
Domain netboot started
Connected to domain netboot
Escape character is ^]
  Using MAC address: 52:54:00:fa:01:01
                                        Requesting information via DHCP: done
                                                                               Using IPv4 address: 192.168.122.217
                                    Requesting file "pxelinux.0" via TFTP from 192.168.122.1
              Receiving data:  14986 KBytes
                                             TFTP: Received pxelinux.0 (14986 KBytes)
     Network loading done, starting kernel...
                                             [    0.395926] Initializing cgroup subsys cpuset
[    0.395929] Initializing cgroup subsys cpu
[    0.395930] Initializing cgroup subsys cpuacct
[    0.395933] Linux version 4.4.0-96-generic (buildd@z13-009) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #119-Ubuntu SMP Tue Sep 12 14:59:56 UTC 2017 (Ubuntu 4.4.0-96.119-generic 4.4.83)
[    0.395937] setup: Linux is running under KVM in 64-bit mode
...
[    0.659971] AppArmor: AppArmor sha1 policy hashing enabled
[    0.659975] ima: No TPM chip found, activating TPM-bypass!

  ┌──────────────────┤ [!!] Configure the network device ├──────────────────┐
  │                                                                         │
  │ Please choose the type of your primary network interface that you       │
  │ will need for installing the Debian system (via NFS or HTTP). Only      │
  │ the listed devices are supported.                                       │
  │                                                                         │
  │ Network device type:                                                    │
  │                                                                         │
  │  ctc: Channel to Channel (CTC) or ESCON connection                      │
  │  qeth: OSA-Express in QDIO mode / HiperSockets                          │
  │  iucv: Inter-User Communication Vehicle - available for VM guests       │
  │  virtio: KVM VirtIO                                                     │
  │                                                                         │
  │                                                                │
  │                                                                         │
  └─────────────────────────────────────────────────────────────────────────┘

For now let's leave the VM's console via Ctrl+]

$

The same again but with Preseeds

Prepare preseed

Okay, this first virtual machine was able to netboot into the default installer - but that's just the first step.
The second step is to combine this with a non-interactive preseed installation.

To do so download an example pressed file (like this - rename it to 'preseed.cfg') and store it locally in the tftp directory:

$ wget --quiet "https://wiki.ubuntu.com/S390X/InstallationGuide/AutomatedInstallsWithPreseed?action=AttachFile&do=get⌖=preseed_kvm.cfg" -O /srv/tftp/preseed.cfg

Do another quick check if preseed.cfg is reachable via tftp:

$ cd /tmp
/tmp$ echo 'get preseed.cfg' | tftp 192.168.122.1
tftp> Received 699 bytes in 0.0 seconds
$ cd -

Prepare pxe config to map to preseed config

A pxelinux.cfg/default configuration file need to be created as well - just to be able to pass over the kernel parameters for preseed installation.

$ sudo mkdir -p /srv/tftp/pxelinux.cfg
$ cat << EOF | sudo tee /srv/tftp/pxelinux.cfg/default
kernel kernel.ubuntu 
initrd initrd.ubuntu
append auto=true priority=critical url=tftp://192.168.122.1/preseed.cfg s390-netdevice/choose_networktype=virtio netcfg/disable_dhcp=false netcfg/get_hostname=netboot-preseed netcfg/get_domain=mydomain.com network-console/password=pass4inst network-console/start=true
EOF

Now some additional tools/packages are needed to generate a general pxelinux.0 file:

$ sudo apt install -y -qq kexec-tools gcc make
$ cd s390-tools/netboot
$ make -f Makefile.pxelinux.0
$ mv /srv/tftp/pxelinux.0 /srv/tftp/pxelinux.0.installer
$ ./s390-tools/netboot/mk-s390image ./kernel.ubuntu pxelinux.0

Now we make all this available to PXE boot:

- the new pxelinux.0 that will read the initial config

- the kernel and initrd to be loaded from there and to kexec into

$ cp pxelinux.0 /srv/tftp/pxelinux.0
 $ cd -
$ cp initrd.ubuntu /srv/tftp/
$ cp kernel.ubuntu /srv/tftp/
$ sudo chmod -R 777 /srv/tftp
$ sudo chown -R nobody: /srv/tftp

Create second VM

Now define a second guest named 'netboot-preseed' for the PXE netboot initiated non-interactive preseed installation on KVM s390x.
It will be done based on the previous example, hence we only need to adjust some names:

$ sed -e 's/netboot/netboot-preseed/g' netboot.xml > netboot-preseed.xml

Again create the virtual disk - in this case really needed, because the installation should complete:

$ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/netboot-preseed.qcow2 5G

Now define that second VM:

$ virsh define netboot-preseed.xml
Domain netboot-preseed defined from netboot-preseed.xml

Now start this second VM:

$ virsh start netboot-preseed --console
Domain netboot-preseed started
Connected to domain netboot-preseed
Escape character is ^]
  Using MAC address: 52:54:00:fa:4d:4d
                              Requesting information via DHCP:  010
...
                              Requesting information via DHCP: done
                                                                               Using IPv4 address: 192.168.122.213
                                    Requesting file "pxelinux.0" via TFTP from 192.168.122.1
              Receiving data:  14986 KBytes
                                             TFTP: Received pxelinux.0 (14986 KBytes)
     Network loading done, starting kernel...
                                             [    0.394899] Initializing cgroup subsys cpuset
[    0.394902] Initializing cgroup subsys cpu
[    0.394903] Initializing cgroup subsys cpuacct
[    0.394907] Linux version 4.4.0-96-generic (buildd@z13-009) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #119-Ubuntu SMP Tue Sep 12 14:59:56 UTC 2017 (Ubuntu 4.4.0-96.119-generic 4.4.83)
[    0.394912] setup: Linux is running under KVM in 64-bit mode

...
can't open /dev/tty3: No such device or address
can't open /dev/tty2: No such device or address
can't open /dev/tty4: No such device or address  ⇐ just ignore these lines !
...
 Finishing the installation 
  │                                   95%                                   │
The system is going down NOW!
Sent SIGTERM to all processes device or address
Sent SIGKILL to all processes device or 
$

The above was the post install shutdown, last task is to adjust the boot order (to not install again):

$ virsh edit netboot

and change:

    <boot dev='network'/>

to:

    <boot dev='hd'/>

and to restart - this time from disk:

No do a final restart of the VM and have fun:

$ virsh start netboot-preseed --console
[...]
netboot-preseed login: _

At this stage two KVM virtual machine are in use. The first one ('netboot') was netboot-ed and landed in a regular interactive d-i installation; the second ('netboot-preseed') was again net-booted, but then performed a non-interactive d-i preseed installation.

References:

Ubuntu cloud

Ubuntu offers all the training, software infrastructure, tools, services and support you need for your public and private clouds.