So many VMs, so few IPs

I share a dedicated server with a few friends. Aside from being a fun learning experience (!), this saves a significant amount of money - for about £10/month I get 500GB disk space, 2GB RAM, 2 CPU cores and (essentially) as much bandwidth as I can eat.

XenCenter

Problem is, the provider only gives us a limited number of IPv4 addresses, so we have to share. Since we're using XenServer, I've set up a tiny router VM for the other VMs to connect to. This bit was simple - I set up an external network bound to the external IP address, and an internal network for the VMs. Here's what my /etc/network/interfaces looks like:

# loopback
auto lo
iface lo inet loopback

# external network
allow-hotplug eth0
iface eth0 inet static
    address EXTERNAL_IP
    netmask 255.255.255.255
    broadcast EXTERNAL_IP
    dns-nameservers 8.8.8.8
    dns-search asdfghjkl.me.uk

# internal network
auto eth1
iface eth1 inet static
    address 192.168.1.254
    netmask 255.255.255.0

I then created a set of iptables rules to forward all traffic from the VMs to the outside world:

# clear existing rules:
iptables --flush
iptables -t nat --flush
iptables --delete-chain
# forward all internal traffic (eth1) to the external network (eth0):
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT

Like on a home router, port forwarding is used to direct traffic to the relevant VM.

# tcp traffic:
iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 1234 -j DNAT --to-destination 192.168.1.1:1234
iptables -A FORWARD -p tcp -d 192.168.1.4 --dport 1234 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

# udp traffic:
iptables -t nat -A PREROUTING -p udp -i eth0 --dport 1235 -j DNAT --to-destination 192.168.1.2:1235
iptables -A FORWARD -p tcp -d 192.168.1.2 --dport 1235 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

# multiple ports:
iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 1236:1240 -j DNAT --to-destination 192.168.1.3
iptables -A FORWARD -p tcp -d 192.168.1.3 --match multiport --dports 1236:1240 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

To apply the firewall rules on startup, I added the script to /etc/rc.local.

For hosting game servers, this is all well and good, as different games tend to use different ports. But for webhosting, everyone wants port 80 - addresses like http://asdfghjkl.me.uk:1234 look unprofessional! To achieve this, I set up apache on the router to proxy requests for specific domains to their relevant VM. Here's an example VirtualHost directive:

<VirtualHost EXTERNAL_IP:80>
ServerName asdfghjkl.me.uk
ServerAlias asdfghjkl.me.uk *.asdfghjkl.me.uk
RewriteEngine on
ProxyRequests off
ProxyPreserveHost on
ProxyPass / http://192.168.1.1:80/
ProxyPassReverse / http://192.168.1.1:80/
</VirtualHost>

This will redirect traffic for asdfghjkl.me.uk and all subdomains to my VM. Without ProxyPreserveHost, the Host: line passed to the VM would always be set to 192.168.1.1, which would make name-based virtual hosting (e.g. for subdomains) impossible. The other issue with this setup is that the VM's access log will only contain the router's address for every request. Luckily the originating IP is sent in the X-Forwarded-For header, and an apache module is available to log this instead. On the VM:

sudo apt-get install libapache2-mod-rpaf
sudo a2enmod rpaf
sudo nano /etc/apache2/mods-available/rpaf.conf

Add the following lines (where 192.168.1.254 is the router's address):

RPAFenable On
RPAFsethostname On
RPAFproxy_ips 192.168.1.254

...and run sudo invoke-rc.d apache2 restart to apply the changes. It's worth noting that requests without a Host: line will be sent to the first site available (alphabetical if you have each VirtualHost directive in a separate file, or first in the list if you don't). I set up a dummy site with a bit of HTML to avoid this.

This all works fine for HTTP traffic, but HTTPS is slightly more involved. The name-based virtual hosts which we're using here will give certificate errors, as the SSL handshake takes place before the Host: line is sent. You either need a unique IP or a unique port for each site. Luckily, most browsers (a notable exception being IE8 or below on Windows XP) support Server Name Indication, which sends the hostname at the start of the handshaking process. We enable mod_ssl on the server with sudo a2enmod ssl, then configure each site with a VirtualHost directive as before:

<IfModule mod_ssl.c>
SSLStrictSNIVHostCheck off

<VirtualHost EXTERNAL_IP:443>
DocumentRoot /var/www/nohostname-ssl
ServerName subdomain.asdfghjkl.me.uk

SSLEngine on
SSLCertificateFile /etc/ssl/certs/certificate1.crt
SSLCertificateKeyFile /etc/ssl/private/certificate1.key
SSLCertificateChainFile /etc/ssl/certs/chain.pem
SSLCACertificateFile /etc/ssl/certs/ca.pem
</VirtualHost>

<VirtualHost EXTERNAL_IP:443>
ServerName asdfghjkl.me.uk
ServerAlias asdfghjkl.me.uk *.asdfghjkl.me.uk
RewriteEngine on
ProxyRequests off
ProxyPreserveHost on
ProxyPass / http://192.168.1.1:80/
ProxyPassReverse / http://192.168.1.1:80/

SSLEngine on
SSLCertificateFile /etc/ssl/certs/certificate2.crt
SSLCertificateKeyFile /etc/ssl/private/certificate2.key
SSLCertificateChainFile /etc/ssl/certs/chain.pem
SSLCACertificateFile /etc/ssl/certs/ca.pem
</VirtualHost>
</IfModule>

Disabling SSLStrictSNIVHostCheck means the first site in the list will be used if no hostname is specified, or if the browser doesn't support SNI. There'll still be a certificate error in the latter case, but it's better than nothing! As before, I've set up a small dummy site explaining that a browser upgrade is necessary.

I noticed soon after setting this up that websites behind the proxy were occasionally giving the error  "The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /". The solution, as found here, was to add the following lines to /etc/apache2/apache2.conf:

SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1

If you want to force SSL for a given site, you'd usually replace the proxy lines for the non-SSL vhost with something like "Redirect permanent / https://asdfghjkl.me.uk". This works for a single site, but means all subdomains would also be redirected there - not ideal! Instead, you can use mod_rewrite to redirect based on the hostname:

RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^/?(.*) https://%{HTTP_HOST}/$1 [R,L]

You'll need a wildcard certificate (i.e. *.domain) for this to work properly, or a separate VirtualHost directive for each subdomain. Up to you really. As always, leave a comment or email me at hello at this domain if you're having any problems!

2 thoughts on “So many VMs, so few IPs

  1. Dan

    Hey dude, really nice guide, are the IP table rules & port forwarding on the VM running the router? Also if port numbers aren’t a problem, i.e only want to run game servers on different ports I don’t need to worry about the bits after the firewalls?

    Reply
    1. asdfghjkl Post author

      Thanks! They are, yep. I put them in /etc/iptables.conf, then added “sh /etc/iptables.conf” to the start of /etc/rc.local so they’ll apply at startup. If you don’t need to share, say, port 80 between multiple servers, you can just ignore all the apache stuff.

      Reply

Leave a Reply to asdfghjkl Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.