quarta-feira, 21 de fevereiro de 2018

Creating custom ISO images for Fedora, CentOS or RHEL

TL;DR: This post is about creating custom ISO images with custom made RPM packages.

Motivation: My motivation was VERY specific. I had to debug a package that had an issue only on first execution after a fresh clean installation - https://bugzilla.redhat.com/show_bug.cgi?id=1518498. This means I had to create a new package and include it on a new custom ISO image every time I wanted to debug a new line or test a new solution.

Also, during the course of this, I faced error messages like:

09:58:3,721 DEBUG packaging: Member: hypervkvpd.x86_64 0:0-0.32.20161211git.e17 - u

09:58:3,727 ERR packaging: Error populating transaction after 10 anaconda retries: 
failure: Packages/hypervkvpd-0-0.32.20161211git.e17.x86_64.rpm from anaconda: 
[Errno 256] Mo more mirrors to try

09:58:3,727 DEBUG packaging: file:///run/install/repo/Packages/hypervkvpd-0-0.32.
20161211git.e17.x86_64.rpm: [Errno -1] Header is not complete.

The trick is very simple, first create your RPM package - here I assume you already know how to do that. Make sure your package has the same name and same dependencies than the one you want to replace. After that, you're ready to start building the new custom ISO image:

1. Download and mount the ISO image you want to customize:
# mount -t iso9660 -o loop RHEL-7.5-20180206.n.0-Server-x86_64-dvd1.iso temp/

2. Create the root directory for the new ISO image:
# mkdir iso_build

3. Copy everything from the original ISO to your newly created root directory:
# cp -pRf temp* iso_build/

4. Place your newly created RPM inside the iso_build/Packages/ directory;

5. Remove all hash data from repodata/ directory, but KEEP the comps file:
# rm -rfv repodata/*.gz repodata/*.bz2;

6. Create the new repository hash information, using as a reference the old comps file you didn't delete:
# createrepo /root/iso_build/Packages -g /iso_build/repodata/ -o /iso_build/ \
-u file:///run/install/repo/Packages/;

7. From inside iso_build/, generate the new ISO image:
# genisoimage -U -r -v -T -J -joliet-long -V 'RHEL-7.5 Server.x86_64'         \
-volset 'RHEL-7.5 Server.x86_64' -A 'RHEL-7.5 Server.x86_64'                  \
-b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot                   \
-boot-load-size 4 -boot-info-table -eltorito-alt-boot                         \
-e images/efiboot.img -no-emul-boot -o /RHEL-7.5-DEBUG-Server-x86_64-dvd1.iso .

Special thanks to Arx Cruz and Edjunior Machado who helped me on this issue.

Enjoy your debugging.

sexta-feira, 9 de fevereiro de 2018

QEMU Sandboxing for dummies

DevConf is an annual conference that takes place in Brno, Czech Republic. This year I applied for a talk to go over my work that I develop since mid 2012: Security on QEMU/KVM Virtual Machines using SECCOMP. Since then I became the maintainer of this feature on QEMU and released the second and better version not long ago. On this post you'll find the slides and the full video of the presentation.

So here we go, very first experience lecturing in English, what a catastrophe! In my defense the audience was very peculiar, not only the guy that very started libseccomp was there, but my manager and the director of the department as well. Anxiety apart, I think it was an outstanding experience, would do it again in the future. :-)

sexta-feira, 19 de janeiro de 2018

Xen synchronicity between frontend and backend devices

So I bumped into a problem last month and it took me too much time to figure out the big picture of the problem since I didn't find too much documentation about that. The help I could find when trying to figure out this was mostly from good people on the channel #xendevel @ Freenode, mostly maintainers. So if you want to understand a little bit of Xen without pinging people on IRC, that's the place.

The problem is the following: I'm running RHEL on Xen Hypervisor and whenever I try to unload and reload xen_netfront kernel module I see outputs like that on dmesg:
# modprobe -r xen_netfront

# dmesg|tail
[ 105.236836] xen:grant_table: WARNING: g.e. 0x903 still in use!
[ 105.236839] deferring g.e. 0x903 (pfn 0x35805)
[ 105.237156] xen:grant_table: WARNING: g.e. 0x904 still in use!
[ 105.237160] deferring g.e. 0x904 (pfn 0x35804)
[ 105.237163] xen:grant_table: WARNING: g.e. 0x905 still in use!
[ 105.237166] deferring g.e. 0x905 (pfn 0x35803)
[ 105.237545] xen:grant_table: WARNING: g.e. 0x906 still in use!
[ 105.237550] deferring g.e. 0x906 (pfn 0x35802)
[ 105.237553] xen:grant_table: WARNING: g.e. 0x907 still in use!
[ 105.237556] deferring g.e. 0x907 (pfn 0x35801)

Moreover, the interface is not usable as well:

# dmesg|tail
[ 105.237163] xen:grant_table: WARNING: g.e. 0x905 still in use!
[ 105.237166] deferring g.e. 0x905 (pfn 0x35803)
[ 105.237545] xen:grant_table: WARNING: g.e. 0x906 still in use!
[ 105.237550] deferring g.e. 0x906 (pfn 0x35802)
[ 105.237553] xen:grant_table: WARNING: g.e. 0x907 still in use!
[ 105.237556] deferring g.e. 0x907 (pfn 0x35801)
[ 160.050882] xen_netfront: Initialising Xen virtual ethernet driver
[ 160.066937] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 160.067270] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 160.069355] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready

# ifconfig eth0
eth0: flags=4098 mtu 1500
ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

# ifconfig eth0 up
SIOCSIFFLAGS: Cannot assign requested address

The first problem happens because the backend part of the module (xen_netback) is still using some pieces of memory (g.e. which states for grant entries) that are shared between guest and host. The ideal scenario would be to wait for the netback to free those entries and only then unload the netfront module. This was actually a bug on the synchronicity of the netfront and netback parts.

The state of the drivers are kept in separate structs, as defined in include/xen/xenbus.h:69:

/* A xenbus device. */
struct xenbus_device {
    const char *devicetype;
    const char *nodename;
    const char *otherend; 
    int otherend_id;
    struct xenbus_watch otherend_watch; 
    struct device dev;
    enum xenbus_state state;
    struct completion down;
    struct work_struct work; 

And the netfront state can be seen from the hypervisor with the command:

# xenstore-ls -fp
/local/domain/1/device/vif/0/state = "4" (n1,r0)

The number 4 indicates XenbusStateConnected (as defined in include/xen/interface/io/xenbus.h:17). So it means everything is a matter of wait for one end to finish using the memory region and the other to free, this first piece of the puzzle is solved by this patch:

diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index 8b8689c6d887..391432e2725d 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -87,6 +87,8 @@ struct netfront_cb {
 /* IRQ name is queue name with "-tx" or "-rx" appended */
+static DECLARE_WAIT_QUEUE_HEAD(module_unload_q);
 struct netfront_stats {
        u64                     packets;
        u64                     bytes;
@@ -2021,10 +2023,12 @@ static void netback_changed(struct xenbus_device *dev,
        case XenbusStateClosed:
+               wake_up_all(&module_unload_q);
                if (dev->state == XenbusStateClosed)
                /* Missed the backend's CLOSING state -- fallthrough */
        case XenbusStateClosing:
+               wake_up_all(&module_unload_q);
@@ -2130,6 +2134,20 @@ static int xennet_remove(struct xenbus_device *dev)
        dev_dbg(&dev->dev, "%s\n", dev->nodename);
+       if (xenbus_read_driver_state(dev->otherend) != XenbusStateClosed) {
+               xenbus_switch_state(dev, XenbusStateClosing);
+               wait_event(module_unload_q,
+                          xenbus_read_driver_state(dev->otherend) ==
+                          XenbusStateClosing);
+               xenbus_switch_state(dev, XenbusStateClosed);
+               wait_event(module_unload_q,
+                          xenbus_read_driver_state(dev->otherend) ==
+                          XenbusStateClosed ||
+                          xenbus_read_driver_state(dev->otherend) ==
+                          XenbusStateUnknown);
+       }

The second piece of the problem is that the interface is not usable when reloaded back. And that's a lack of initializing the state of the device so the backend notices it, and hence, connects the two drivers together (frontend and backend). This was easily solved by the following patch:

diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index c5a34671abda..9bd7ddeeb6a5 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -1326,6 +1326,7 @@ static struct net_device *xennet_create_dev(struct xenbus_device *dev)
+       xenbus_switch_state(dev, XenbusStateInitialising);
        return netdev;