Automating libvirt networking with minimal pain

A few weeks ago I started a project to experiment with kvm / libvirt / qemu virtualization on CentOS 8.2. Getting qemu and libvirt all setup was relatively easy, and went pretty painlessly. After that, setting up network bridges went relatively well also.

However, when it got to networking, things got a little less pleasant.

Non-bridged Networking

First and foremost, the project I’m working on treats the hypervisor as the firewall. It does minimal routing to ensure that the VM’s are adequately separated from the main network. Documentation on this was limited, to say the least. I won’t go into the setup details, but you basically create virtual interface bridges via the virsh net-define from an XML file.

Next, the hard part: creating firewall rules to allow certain services on VM’s to be accessed.

This proved a much more challenging task than I thought. All the information out there made this extremely daunting and non-trivial.

As it turns out, the recommended solutions are comparatively over-engineered to what I’m going to propose.

Setting up the qemu hook

First and foremost: when libvirt starts, stops, and migrates a VM it calls the qemu hook which is, by default, in /etc/libvirt/hooks. There are two parameters for this hook we’re going to care about: ${1} which is the “domain” or “virtual machine”, and ${2} which is the “action” being performed.

For our situation, we care about three actions: start, stopped, and reconnect. For reconnect, we’re simply going to stop then start.

VM / Networking Structure

My setup puts all my VM info in /home/kvm, there are three folders there:

/home/kvm/images: all my ISO’s;
/home/kvm/disks: all of the disk images for my VM’s;
/home/kvm/networks: all of the scripts for the VM’s of which we should do network operations;

I chose this structure because it’s easy to remember.

Now the biggest problem with libvirt networking is that it resets firewall rules in iptables regularly. Any time the daemon restarts, if I recall correctly. As a result, if we create iptables rules there is no guarantee they will remain after any sort of state-change.

The libvirt hook

So, my solution was to put a network script for each VM in /home/kvm/networks, named after the VM, that allows me to add/remove the appropriate iptables rules when the VM starts and stops.

To do so, I created a very simple qemu hook:

#!/bin/bash

DOMAIN=${1}
ACTION=${2}
BASE_PATH="/home/kvm/networks/$DOMAIN"

if test -f $BASE_PATH; then
    if [ $ACTION = "stopped" ] || [ $ACTION = "reconnect" ]; then
		$BASE_PATH stopped
	fi
    if [ $ACTION = "start" ] || [ $ACTION = "reconnect" ]; then
		$BASE_PATH start
	fi
fi

This hook simply calls /home/kvm/networks/VM_NAME with either start or stopped, depending on what is happening.

Next, we’ll look at a sample VM I have a script for.

Important: after creating this hook, restart libvirtd via service libvirtd restart. Otherwise, libvirt will not pick up on your new script.

The VM Script

The script for each VM is extremely simple, I define the public / external IP, the private / internal IP, and the ports I want to forward, in the format protocol:port, with commas between each combination:

#!/bin/bash
PUBLIC_IP=192.168.0.2
PRIVATE_IP=10.1.1.2
PORTS=tcp:80,tcp:443
/home/kvm/networks/modify_network $PUBLIC_IP $PRIVATE_IP $PORTS ${1}

The PUBLIC_IP is an external IP, this IP should be assigned to a physical interface on the server.

We then call another script, /home/kvm/networks/modify_network, which has the root of the iptables rules. We also pass ${1} which is either start or stopped (because that’s how qemu calls it).

The hard part: the iptables rules

Ok, so maybe this isn’t that hard. Again, we have a relatively simple script:

#!/bin/bash
PUBLIC_IP=${1}
PRIVATE_IP=${2}
readarray -d , -t PORTS <<< "${3}"

if [ ${4} = "start" ]; then
	/sbin/iptables -I FORWARD -m state -d $PRIVATE_IP/32 --state NEW,RELATED,ESTABLISHED -j ACCEPT
	for (( n=0; n < ${#PORTS[*]}; n++ )); do
		readarray -d : -t PORT_PARTS <<< "${PORTS[n]//[$'\t\r\n']}"
		/sbin/iptables -t nat -I PREROUTING -p ${PORT_PARTS[0]} --dport ${PORT_PARTS[1]} -d $PUBLIC_IP -j DNAT --to $PRIVATE_IP
	done
fi

if [ ${4} = "stopped" ]; then
	/sbin/iptables -D FORWARD -m state -d $PRIVATE_IP/32 --state NEW,RELATED,ESTABLISHED -j ACCEPT
	for (( n=0; n < ${#PORTS[*]}; n++ )); do
		readarray -d : -t PORT_PARTS <<< "${PORTS[n]//[$'\t\r\n']}"
		/sbin/iptables -t nat -D PREROUTING -p ${PORT_PARTS[0]} --dport ${PORT_PARTS[1]} -d $PUBLIC_IP -j DNAT --to $PRIVATE_IP
	done
fi

This one is, perhaps, the most complex so far, but only because I wanted comma-specified ports.

The readarray -d , -t PORTS <<< "${3}" section reads the third parameter, then splits it on commas into an array called PORTS.

Then, if the fourth parameter is start, we do an /sbin/iptables -I FORWARD insert with the private IP. This is actually extremely important, as libvirt by default only allows for RELATED,ESTABLISHED connections by default, and we need to allow NEW connections as well. I spent probably three or four days fighting this rule specifically.

Next, we use the for to loop through all the items in the PORTS array. (The ${#PORTS[*]} counts the number of elements in the array.) The readarray -d : -t PORT_PARTS <<< "${PORTS[n]//[$'\t\r\n']}" line does two things: the part of the line after the carets (<<<) removes whitespace, then the readarray section again splits the string, this time on colons.

We then do an /sbin/iptables -t nat -I that opens the protocol and port to the public and private IP combination we had.

Lastly, our stopped block does the opposite: it removes each rule we previously added to ensure that no errant ports are left open if the VM is killed.

Verify with iptables

Lastly, it’s helpful to verify the iptables -L result:

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             10.1.1.2             state NEW,RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             10.1.1.0/24          ctstate RELATED,ESTABLISHED
ACCEPT     all  --  10.1.1.0/24          anywhere
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

I left the default rules in place, but you’ll notice that the second rule, which allows connections from anywhere to 10.1.1.0/24, only allows RELATED,ESTABLISHED, which means that if you are opening an internal service users won’t be able to connect unless we add our NEW,RELATED,ESTABLISHED rule, as there’s no previous connection to associate yet.

We’ll also verify the iptables -t nat -L result:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DNAT       tcp  --  anywhere             192.168.0.2          tcp dpt:https to:10.1.1.2
DNAT       tcp  --  anywhere             192.168.0.2          tcp dpt:http to:10.1.1.2

These both show that our rules were created.

The End

This all is basically the result of my two+ weeks of researching this issue. During my research I came across dozens of articles that had much more complex solutions, but in the end my requirements (NAT’ing specific ports to specific VM’s) actually simplified my solution.

Full Send to the Compiler