Migrating Zimbra 7.2 32bit to 64bit on centos 6.5
I did this procedure before and I had a longer outage than I wanted. This time, we are doing it without taking down the old server with a very slight outage for just a few minutes. I also use front end non zimbra MX mail servers in my configuration. The process is modified from: ‘Network Edition: Moving from 32-bit to 64-bit Server‘. I didn’t see anything that is commercially specific so it should work with the open source version also. This method allows one to test longer and hopefully mitigate most of your problems before you commit to cut over.
Steps are as follows:
- prep old 32 bit server
- install new 64 bit server
- prep 64 bit server to be like the 32 bit server
- rsync files from the 32 bit server to 64 bit server
- test
- repeat steps 4-5 as often as it takes
- final rsync
- cut over
prep old 32 bit server:
# We create convenient /backup directory that we will copy to the remote site # Perform functions as root mkdir /backup cp /opt/zimbra/conf/localconfig.xml /backup/ cp /opt/zimbra/mailboxd/etc/keystore /backup/ cp /opt/zimbra/openldap-data/DB_CONFIG /backup/ cp -r /opt/zimbra/zimlets-deployed /backup/ # Perform as zimbra # Get a list of passwords from the 32-bit server zmlocalconfig -s | grep password > /backup/zmlocalconfig-pass.txt # LDAP Config database, LDAP Data, and config file /opt/zimbra/libexec/zmslapcat -c /backup /opt/zimbra/libexec/zmslapcat /backup # Perform as root (optional) chown -R zimbra:zimbra /backup chmod -R 755 /backup/* # send over to the new machine rsync -avzH -e ssh --delete --progress /backup/ new-64-machine:/backup
install new 64 bit server
Replicate this using split-brain dns, update /etc/hosts to simulate your 32-bit host. The hostname doesn’t have to be the same and I kept it different which helped me since my tcsh prompt has the hostname. A few things to keep in mind. Install the same services as the 32 bit server. I also put a firewall in place so the new server could not connect to my old 32 bit server. Push from the 32 bit server but do not pull from the 64 bit to add an extra level of protection.
If you forgot which services are installed, find that list first. Here is the command.
[zimbra user] %zmprov gs `zmhostname` |grep zimbraServiceInstalled zimbraServiceInstalled: antivirus zimbraServiceInstalled: antispam zimbraServiceInstalled: logger zimbraServiceInstalled: mailbox zimbraServiceInstalled: memcached zimbraServiceInstalled: mta zimbraServiceInstalled: convertd zimbraServiceInstalled: stats zimbraServiceInstalled: imapproxy zimbraServiceInstalled: snmp zimbraServiceInstalled: ldap zimbraServiceInstalled: spell
It is important to install the same modules and also update the ham, spam, and admin accounts to be the same during the initial configuration as it makes things easier later. If you remember, it would be a good time to change all the passwords to the same as the 32 bit server. If you forget, no worries as this can be done later.
The official documentation assumes the same name, ip address, etc. That is not realistic to expect the same ip address. I found that modifying /etc/hosts and running a private dns server with different ip addresses as the normal dns server allowed me to work on the new server while the old one continued to serve its users. This bought me time to experiment and learn about the failures before cutting over to production. If you don’t have the same address space you will need to do this since various files like postfix will be wrong because mynetworks will have your old ip address space and your users with email clients can not relay until you change these type of problems.
prep 64 bit server to be like the 32 bit server
My goal was to get this as similar to the production server before I began updating the ldap stuff.
#as zimbra user on your 32 bit server % zmprov gacf | grep SpamAccount zimbraSpamIsNotSpamAccount: [email protected] zimbraSpamIsSpamAccount: [email protected] # as zimbra user on your 64 bit server % zmprov mcf zimbraSpamIsNotSpamAccount [email protected] % zmprov mcf zimbraSpamIsSpamAccount [email protected] # grep mysql /backup/zmlocalconfig-pass.txt to find these passwords % zmmypasswd --root 32-root-mysql-password % zmmypasswd 32-zimbra-mysql-password
At this point, we have the mysql passwords the same, the ham and spam accounts the same and hopefully the same zimbra host and all the same modules installed. Next step is to change all the passwords including ldap. You may want to take a snap shot of /opt/zimbra before you begin. Here are the steps I followed.
# /etc/init.d/zimbra stop
Using /backup/zmlocalconfig-pass.txt, I verified that my passwords on the 32 bit /opt/zimbra/conf/localconfig.xml were transferred to the 64 bit /opt/zimbra/conf/localconfig.xml
Note: if you start zimbra, it will fail because the ldap database has the old passwords. If you follow the instructions of transferring the ldap files at this time, you do not need to follow the following instructions. I wanted to be able to continuously update from the 32 bit machine so I did not update the ldap stuff until I was certain that my 64 bit zimbra instances was working how I wanted.
I followed this ldap article that was written for Zimbra version 5 but works perfectly to change the password.
# Step 1: Retrieve local ldap password % zmlocalconfig -s zimbra_ldap_password ldap_master_url
This returns the value for what is believed to be the password for the zimbra admin user and the URL to talk to the master for making the change to the LDAP database. Next change the ldap database with the password from the 32 bit machine.
<pre>% /opt/zimbra/openldap/bin/ldapmodify -x -h <ldap master URL value> -D "uid=zimbra,cn=admins,cn=zimbra" -W
When prompted with Enter LDAP password, use the value for zimbra_ldap_password returned in Step 1. Press Enter.
4. Then type:
dn: uid=zimbra,cn=admins,cn=zimbra changetype: modify replace: userPassword userPassword: value of zimbra_ldap_password
5. Press Enter twice for the changes to take place.
You can now restart zimbra and test the environment. You might also take a snapshot at this point should you need to fallback to.
Updating the ldap database on the 64 bit machine
# Backup LDAP database and then empty it mv /opt/zimbra/data/ldap/hdb /opt/zimbra/data/ldap/hdb.backup mv /opt/zimbra/data/ldap/config /opt/zimbra/data/ldap/config.backup mkdir -p /opt/zimbra/data/ldap/hdb/db/ mkdir -p /opt/zimbra/data/ldap/hdb/config/ mkdir -p /opt/zimbra/data/ldap/hdb/logs/ cp /backup/DB_CONFIG /opt/zimbra/data/ldap/hdb/db/ chown -R zimbra:zimbra /opt/zimbra/data/ldap /opt/zimbra/openldap/sbin/slapadd -q -n 0 -F /opt/zimbra/data/ldap/config -cv -l /backup/ldap-config.bak /opt/zimbra/openldap/sbin/slapadd -q -b "" -F /opt/zimbra/data/ldap/config -cv -l /backup/ldap.bak
A word about DB_CONFIG. There is a good chance if this is an older install that you will need to update DB_CONFIG first and change set_lg_dir as the directory structure has changed.
diff DB_CONFIG /opt/zimbra/data/ldap/hdb/db/DB_CONFIG < set_lg_dir /opt/zimbra/openldap-data/logs --- > <strong>set_lg_dir /opt/zimbra/data/ldap/hdb/logs</strong> (this is what we want)
At this point, you should assume old stuff like ip addresses are locked into the ldap for mynetworks and various conf files. Fix as needed.
rsync files from the 32 bit server to 64 bit server
Move the old database files out of the way and rsync as often as you need prior to cut over. As a precaution, I shut down the new 64 bit image first but left the 32 bit running.
# on 32 bit server rsync -avzH -e ssh --delete --progress /opt/zimbra/store/ new-64-server:/opt/zimbra/store rsync -avzH -e ssh --delete --progress /opt/zimbra/index/ new-64-server:/opt/zimbra/index rsync -avzH -e ssh --delete --progress /opt/zimbra/db/data/ new-64-server:/opt/zimbra/db/data
At this point, you can decide if you want to upgrade to version 8. I did that with the normal process and other than some silly things from zmstat-fd with Zimbra version 8.07, it was uneventful. This is a good time to update the SSL keys and increase there size which is the default for this version.
Problems:
- mynetworks contained the old ip addresses and values – Solution here.
- DB_CONFIG contained 5.0.x or previous LDAP setup even though I was running zimbra 7.2.1. The fix is to edit the DB_CONFIG file and update the log path with the /opt/zimbra/data/ldap/hdb/logs path. Since I started out as zimbra 4 that might have explained. If you fail to do this ldap will not get started and give a very misleading error about database corruption. The restore ldapadd commands have to be run as the zimbra user and not root or it will fail also with a misleading message.
- amavisd.conf – uses mynetworks so it will contain your previous ip address and not the current servers. Fix the mynetwork first and postfix and other programs will have the correct value.
- Backup and other admin functions failed because of ssh permissions and port 22 – not sure why but rebuilding keys and zmsshkeygen did the trick
- /opt/zimbra/conf/salocal.cf – /opt/zimbra/bin/zmtrainsa could not parse a ipv6 address that was now present in trusted_networks and the old ip address was present here.
- Any mods you have made will need to be applied. I have quite a few including some transport maps. BTW, if that isn’t installed, you may get a temporary lookup failure when you are testing your server via telnet and the recpt to: <…@your-domain.com> test.
Various links:
- How to sync Zimbra Calendar and Contacts to Microsoft Outlook using funambol
- Getting rid of the proxy or other services
Various links that were good backup material for this task but didn’t work for me:
- Zimbra migration from 7 to 8 + 32 bit to 64 bit server
- Zimbra 32-bit to 64-bit migration – really good ways to minimize downtime with rsync and getting as much data over as possible
Once up there are a bunch of things to tune if you are on a virtual environment as version 8 takes a lot of memory and I run my zimbra instances on 512MB and now 2GB environments. The zimbra version 8 ldap DB is 80GB sparse file now so be careful if you rsync or copy the directory structure without accounting for sparse file options. I need to update this article with the scripts that I created to make this quick and easy.
