Minimizing downtime associated with Web/DNS/Mail IP address migrations

Migrating servers to new datacenters or IP addresses is never entirely problem free, usually due to many DNS servers on the internet not obeying TTL’s or caching old records for way too long.

These old cached DNS records (usually at different ISP’s, major mailservers like google, hotmail, yahoo, etc.) cause problems where they’ll route mail to your old server’s IP even if you update your DNS.  This also results in clients connecting to your old web IP address and not hitting your new server.

When migrating static html sites, having two servers online during a migration isn’t a real issue, but when you’ve got database driven websites (forums, blogs, etc.) this will result in updates going to two different sites, causing all sorts of hassles.  The way around this is to set up your old site as a proxy only, and proxy the results to the new IP address.

Here’s some configs to allow for proxying Web traffic via Apache, DNS traffic via bind9, and Mail traffic via sendmail.  All these configs assume 10.10.10.1 is your “new” mail/dns/web address.

The way this was done is create a brand new openvz virtual system, and editing stock CentOS configs – you can edit your current server if you wish, but you may end up with more work than it’s worth.  Then when the final migration is done, we add the old IP’s to this openvz system and allow it to forward web/dns/mail to the new servers’s IP addresses.

DNS (Bind9) Forwarding

Forwarding DNS is useful if you were running a DNS server which may not be updated yet – this allows your clients and their DNS servers to receive the newest updated DNS information from your new IP’s even if they are querying your old IP addresses.

named.conf

options {
directory “/var/named”;
version “not currently available”;
forwarders {10.10.10.1; 10.10.10.2;};
forward only;
};

Where 10.10.10.1 and 10.10.10.2 are your new nameserver IP addresses.

Web (Apache) Forwarding

You can set up a proxy for all your web addresses so that any accesses to the old IP will actually talk with your new IP address.  This can be done using Apache’s mod_proxy module.

Here’s a simple setup that will pass all traffic to any virtual host on your old IP address to a new IP address you’ve defined in the ProxyPass directives.

Apache’s httpd.conf additions:

<IfModule mod_proxy.c>

<VirtualHost *:*>
ProxyPreserveHost On
ProxyMaxForwards 256
ProxyPass / http://10.10.10.1/
ProxyPassReverse / http://10.10.10.1/
ProxyReceiveBufferSize 4096
ProxyTimeout 120
</VirtualHost>

</IfModule>

This is all you need, you’ll want to make sure there aren’t any other virtual hosts set up.  If you have multiple hosts going to different IP addresses, you’ll have to set up multiple virtualhosts for each proxy IP rather than a “catch-all” like above.

Proxy’ing SSL content is a bit more tricky and I’ve not put much time into getting this to work.  The Apache docs make it sound like it does – but I didn’t really need to figure that out!

Mail (Sendmail) Forwarding

Mail that is sent to your old IP you’ll want to have forwarded directly to your new IP.  This is easily done with Sendmail by using relay-domains and mailertable, which can forward mail directly to an IP address of your choosing.

If you only have one or two domain names, this is easy, but if you’ve got a whole list of domains it becomes a bit repetitive to set this up.  One way to do this is to use a few simple scripts to create the relay-domains and mailertable for you.  Here I parse a named.conf file that has all my domains I’d like to forward, and it creates the relay-domains file (which lists all the domain names), which is then parsed to create a mailertable file.

Sendmail configs:

# Create relay-domains for sendmail from a named.conf file:
grep zone /etc/named.conf | awk -F'”‘ ‘{print $2}’ | egrep -v ‘(^domain.com|[Ii][Nn]-[Aa][Dd][Dd][Rr]|view$|named.rfc|internal$|authoritative$)’ |grep “.[a-zA-Z]” > /etc/mail/relay-domains

example of relay-domains file:

domain1.com
domain2.com
domain3.com

# Create mailertable and forward IP
for i in `cat /etc/mail/relay-domains `; do echo “$i esmtp:[10.10.10.1]” >> /etc/mail/mailertable; done

example of mailertable file:

domain1.com esmtp:[10.10.10.1]
domain2.com esmtp:[10.10.10.1]
domain3.com esmtp:[10.10.10.1]

# Edit /etc/sendmail.mc to make sure it’s listening on port 25 – if the following has 127.0.0.1 or localhost in it, remove that:
DAEMON_OPTIONS(`Port=smtp, Name=MTA’)dnl

Make sure to type “make” inside /etc/mail to update these configs
cd /etc/mail; make

Restart sendmail
service sendmail restart

Make sure sendmail is now listening on port 25 externally!
netstat -tan |grep :25

Should result in: tcp        0      0 0.0.0.0:25                  0.0.0.0:*                   LISTEN

You should now have a server that will forward web/mail/dns to your new IP 10.10.10.1 – so any clients who have old cached DNS values will get to the right server if they send mail or try to access your webserver!

Leave a Reply

Your email address will not be published. Required fields are marked *

*