So many VMs, so few IPs

I share a dedicated server with a few friends. Aside from being a fun learning experience (!), this saves a significant amount of money - for about £10/month I get 500GB disk space, 2GB RAM, 2 CPU cores and (essentially) as much bandwidth as I can eat.


Problem is, the provider only gives us a limited number of IPv4 addresses, so we have to share. Since we're using XenServer, I've set up a tiny router VM for the other VMs to connect to. This bit was simple - I set up an external network bound to the external IP address, and an internal network for the VMs. Here's what my /etc/network/interfaces looks like:

# loopback
auto lo
iface lo inet loopback

# external network
allow-hotplug eth0
iface eth0 inet static
    address EXTERNAL_IP
    broadcast EXTERNAL_IP

# internal network
auto eth1
iface eth1 inet static

I then created a set of iptables rules to forward all traffic from the VMs to the outside world:

# clear existing rules:
iptables --flush
iptables -t nat --flush
iptables --delete-chain
# forward all internal traffic (eth1) to the external network (eth0):
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT

Like on a home router, port forwarding is used to direct traffic to the relevant VM.

# tcp traffic:
iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 1234 -j DNAT --to-destination
iptables -A FORWARD -p tcp -d --dport 1234 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

# udp traffic:
iptables -t nat -A PREROUTING -p udp -i eth0 --dport 1235 -j DNAT --to-destination
iptables -A FORWARD -p tcp -d --dport 1235 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

# multiple ports:
iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 1236:1240 -j DNAT --to-destination
iptables -A FORWARD -p tcp -d --match multiport --dports 1236:1240 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

To apply the firewall rules on startup, I added the script to /etc/rc.local.

For hosting game servers, this is all well and good, as different games tend to use different ports. But for webhosting, everyone wants port 80 - addresses like look unprofessional! To achieve this, I set up apache on the router to proxy requests for specific domains to their relevant VM. Here's an example VirtualHost directive:

<VirtualHost EXTERNAL_IP:80>
ServerAlias *
RewriteEngine on
ProxyRequests off
ProxyPreserveHost on
ProxyPass /
ProxyPassReverse /

This will redirect traffic for and all subdomains to my VM. Without ProxyPreserveHost, the Host: line passed to the VM would always be set to, which would make name-based virtual hosting (e.g. for subdomains) impossible. The other issue with this setup is that the VM's access log will only contain the router's address for every request. Luckily the originating IP is sent in the X-Forwarded-For header, and an apache module is available to log this instead. On the VM:

sudo apt-get install libapache2-mod-rpaf
sudo a2enmod rpaf
sudo nano /etc/apache2/mods-available/rpaf.conf

Add the following lines (where is the router's address):

RPAFenable On
RPAFsethostname On

...and run sudo invoke-rc.d apache2 restart to apply the changes. It's worth noting that requests without a Host: line will be sent to the first site available (alphabetical if you have each VirtualHost directive in a separate file, or first in the list if you don't). I set up a dummy site with a bit of HTML to avoid this.

This all works fine for HTTP traffic, but HTTPS is slightly more involved. The name-based virtual hosts which we're using here will give certificate errors, as the SSL handshake takes place before the Host: line is sent. You either need a unique IP or a unique port for each site. Luckily, most browsers (a notable exception being IE8 or below on Windows XP) support Server Name Indication, which sends the hostname at the start of the handshaking process. We enable mod_ssl on the server with sudo a2enmod ssl, then configure each site with a VirtualHost directive as before:

<IfModule mod_ssl.c>
SSLStrictSNIVHostCheck off

<VirtualHost EXTERNAL_IP:443>
DocumentRoot /var/www/nohostname-ssl

SSLEngine on
SSLCertificateFile /etc/ssl/certs/certificate1.crt
SSLCertificateKeyFile /etc/ssl/private/certificate1.key
SSLCertificateChainFile /etc/ssl/certs/chain.pem
SSLCACertificateFile /etc/ssl/certs/ca.pem

<VirtualHost EXTERNAL_IP:443>
ServerAlias *
RewriteEngine on
ProxyRequests off
ProxyPreserveHost on
ProxyPass /
ProxyPassReverse /

SSLEngine on
SSLCertificateFile /etc/ssl/certs/certificate2.crt
SSLCertificateKeyFile /etc/ssl/private/certificate2.key
SSLCertificateChainFile /etc/ssl/certs/chain.pem
SSLCACertificateFile /etc/ssl/certs/ca.pem

Disabling SSLStrictSNIVHostCheck means the first site in the list will be used if no hostname is specified, or if the browser doesn't support SNI. There'll still be a certificate error in the latter case, but it's better than nothing! As before, I've set up a small dummy site explaining that a browser upgrade is necessary.

I noticed soon after setting this up that websites behind the proxy were occasionally giving the error  "The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /". The solution, as found here, was to add the following lines to /etc/apache2/apache2.conf:

SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1

If you want to force SSL for a given site, you'd usually replace the proxy lines for the non-SSL vhost with something like "Redirect permanent /". This works for a single site, but means all subdomains would also be redirected there - not ideal! Instead, you can use mod_rewrite to redirect based on the hostname:

RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^/?(.*) https://%{HTTP_HOST}/$1 [R,L]

You'll need a wildcard certificate (i.e. *.domain) for this to work properly, or a separate VirtualHost directive for each subdomain. Up to you really. As always, leave a comment or email me at hello at this domain if you're having any problems!

2 thoughts on “So many VMs, so few IPs

  1. Dan

    Hey dude, really nice guide, are the IP table rules & port forwarding on the VM running the router? Also if port numbers aren’t a problem, i.e only want to run game servers on different ports I don’t need to worry about the bits after the firewalls?

    1. asdfghjkl Post author

      Thanks! They are, yep. I put them in /etc/iptables.conf, then added “sh /etc/iptables.conf” to the start of /etc/rc.local so they’ll apply at startup. If you don’t need to share, say, port 80 between multiple servers, you can just ignore all the apache stuff.


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.