Disclaimer: I have the NETWORK/PRO version but chose not to use the backups. This method should work with both the free and paid version as a result. I have used a variation of this for every update since version version 7. We perform offsite backups and use tar to handle the sparse 80GB ldap file. This can run hot without the need to stop zimbra for most cases. The plan is to run this often and just before final cut-over do your shutdown as an extra precaution. You should be able to minimize your disruption to the production server. If you are just testing out the update process, you may never have to shutdown the production. I don’t and haven’t had any ldap issues testing while learning what it going to take to upgrade zimbra on my testing machines. You can test the new server with these hot images and even perform an upgrade test with this method. As the backup server can have multiple zimbra customer instances, this method allows one to have a central backup server for catastrophic recovery for those customers. Generally, I use something like linode where I can provision a 8GB image for $0.06/hr and test my updates for less than $1.50 per day before moving to the actual production server for the real update. This also tests my catastrophic backup should I lose a data center and alerts me to problems with the backups. Testing your updates process can save a lot of downtime and stress for yourself to your production servers later.
Summary of Steps
On existing host, Backup existing zimbra instance via rsync (see note)
On new host, install zimbra software (install.sh -s)
On new host, pull zimbra instance (/opt/zimbra) via rsync from backup host
or
On production host, push zimbra instance (/opt/zimbra) via rsync
On new host, install zimbra software (install.sh)
On new host, update new ip address
Optional: Can try upgrade of new zimbra software
Note: we push vs pull zimbra from the existing host because we create a tar image of the ldap database which is a sparse file.
On ‘NEW or Temporary MACHINE’
Modify /etc/hosts on new host so that the name of your old server is represented in this new server. This is output of this command:
% zmhostname
Run an authoritative DNS server with the new ip addresses for this zmhostname and set the resolv.conf to 127.0.0.1. Install operating specific software required for a normal zimbra installation. Finally set the hostname to be exactly the same; although I have found this not to be necessary. I updated the shell prompt to something like testmachine machine to keep track of where I am should you set the hostname to the same as zmhostname.
Install initial zimbra software. This should be the same exact version at this time. Purpose is to prime the system with zimbra software. We plan on overwriting this in a moment.
#./install.sh -s --skip-activation-check
On Existing Server or Backup Server
As a precaution, I like to block access from the NEW MACHINE on my production server. This prevents any issues with zimbra reaching back if you fail to update the hosts file and run a local dns as required for this method.
Create a script like the one below. You will run this as often as you need to get the new server in sync. If your backup process includes backing up the ldap database then you can leave the ldap stuff commented out. Because the ldap is sparse and can be over 80GB in size, it is much faster to work around rsync’s slowness with huge files and use tar. I use this to backup my zimbra instances to a offsite backup. For example, I have a $20/month vultr account that has 640GB of disk space. If you use this code, you can get a $20 credit. Make sure to choose the storage instance as that is what you want for this type of storage application.
#!/bin/sh # # stage a backup to make writing tar faster between upgrades # of zimbra releases. Just some protection against a castrophic failure # # create portable ldap file to copy #cd /opt/zimbra/data/ldap/mdb/db #tar cvzSpf data.mdb.tar.gz data.mdb # Backup to X.X.X.X (newhost or temporary testing host) rsync --progress --delete --archive --verbose --hard-links --exclude=zimbra/data/ldap/mdb/db/data\.mdb --exclude=zimbra/backup /opt/zimbra X.X.X.X:/opt
On ‘NEW MACHINE’
When you are ready to bring up zimbra on this machine, follow these steps.
cd /opt/zimbra/data/ldap/mdb/db /bin/rm -f data.mdb tar xvzSpf data.mdb.tar.gz chown zimbra:zimbra data.mdb echo echo 'fixing permissions' cd /opt/zimbra /opt/zimbra/libexec/zmfixperms -extended chown -R zimbra:zimbra /opt/zimbra/.ssh
At this point, you have a broken zimbra instance but the next step will repair it. Ignore the warning below from the install step about the license.
# cd zcs-NETWORK-8.6.0_GA_1153 # ./install.sh --skip-activation-check
Almost there… fix the ip addresses. Replace X.X.X.X with new server ip address
% zmprov -l ms `zmhostname` zimbraMtaMyNetworks '127.0.0.0/8 X.X.X.X/32' % postfix reload Do the same with these files and update the ip address with the new server # vi /opt/zimbra/conf/opendkim-localnets.conf # vi /opt/zimbra/conf/amavisd.conf # vi /opt/zimbra/conf/salocal.cf
You can repeat this as often as required and test the server . At this point, you have a working clone of your zimbra production server. Test procedures and process before testing on the production machine, etc. If you plan on upgrading than read below.
The next step will prevent you from rolling back so if that happens, one must do an ./install -u first.
Install the upgraded version of Zimbra. Once you upgrade, you can’t roll back the version of zimbra to a previous release without repeating all these steps.
Preparing for upgrade to Zimbra 8.7.1
Bring everything to their patch level. That means the OS and zimbra 8.6. Verify everything is functioning by issuing the following:
zmcontrol restart zmcontrol status
Prepare 8.6 if you are not running proxy as Zimbra 8.7.1 now requires proxy and memcache service even on a single architecture. If you have installed the software previously, then you can move via this method. If you haven’t then, the upgrade will give you an option to add the proxy and memcache. Note: if you haven’t installed the software previously, you will get some weird error about nginx and invalid directive. Proceed to the install phase of Zimbra 8.7.1 to install. Otherwise, you will have a working system for 8.6+ with the commands below.
zmprov ms `zmhostname` zimbraReverseProxyMailMode https zmprov ms `zmhostname` zimbraMailReferMode reverse-proxied zmprov ms `zmhostname` zimbraReverseProxyHttpEnabled TRUE /opt/zimbra/libexec/zmproxyconfig -e -w -o -a 8080:80:8443:443 -x https -H `zmhostname` /opt/zimbra/libexec/zmproxyconfig -e -m -o -i 7143:143:7993:993 -p 7110:110:7995:995 -H `zmhostname` zmcontrol restart zmcontrol status
At this point, you have done enough that Zimbra 8.7.1 would allow an upgrade but one still needs a few more housekeeping functions performed at some point in the future to make the proxy work as one would expect. If you have a working 8.6+ systems after the proxy enable you might choose to do it now… otherwise, you can do them post install.
# because we are using a proxy, this will make sure that the client ip is shown in the /opt/zimbra/log/audit.log
zmprov mcf +zimbraMailTrustedIP 127.0.0.1 +zimbraMailTrustedIP X.X.X.X
[/code]
The next steps are requires so that the admin console “gui” can view email and the share functions work.
zmprov mcf zimbraPublicServiceHostname mail.example.com zmprov md example.com zimbraPublicServicePort 443 zmprov md example.org zimbraPublicServicePort 443 zmprov md example.net zimbraPublicServicePort 443 zmprov md example.com zimbraPublicServiceProtocol https zmprov md example.org zimbraPublicServiceProtocol https zmprov md example.net zimbraPublicServiceProtocol https zmmailboxdctl restart
# cd zcs-NETWORK-8.7.1_GA_1670 # ./install.sh
Update 10/25/2016:
When moving a 8.6 site with this method, zimbra will attempt to install their DNS caching server. That means the install will bring up the config menu because you have a port conflict on port 53. Choose option 5 and then option 1 for status and disabled that. There is no need on this test machine as we are running bind already.
Update 11/1/2016:
This method works so well that the following can be done. rsync data to newserver, then install -s –skip-activation-check; then rsync data as directed above. This is the case where you have forgot to install the zimbra software but don’t want to blow away all that real data your rsync’d. Then you rsync again without having to completely resend huge data stores.
Update Fixes (Not normal but 8.7+ was Special)
For whatever reasons, sometimes things just don’t work as expected. Here are some of the things I noticed with Zimbra 8.7+ upgrade.
ldap checksum error found via grep ldap /var/log/zimbra.log
% zmlocalconfig -s -m nokey ldap_root_password
# Run this commands and cut/paste following lines with ^D
% ldapmodify -x -H ldapi:/// -D cn=config -W
dn: olcDatabase={0}config,cn=config
changetype:modify
replace: olcMaxDerefDepth
olcMaxDerefDepth: 15
followed by a control-D.
mysql update failure – tail /opt/zimbra/mysql_error.log
Look for: “unexpected structure. Using transient stats instead”. Reference: forum
1. Obtain mysql root password: $ zmlocalconfig -s | grep mysql | grep password 2. Create missing directory and symbolic link: $ mkdir /opt/zimbra/data/tmp/mysqldata $ ln -s /opt/zimbra/data/tmp/mysql/mysql.sock /opt/zimbra/data/tmp/mysqldata/mysql.sock 3. Perform mysql_upgrade $ /opt/zimbra/common/bin/mysql_upgrade -u root -p That's it
Statistics stop working (Catagory of Problems)
1) zmstat-fd.pid not running
zmstatctl status Running: zmstat-io-x Running: zmstat-proc process 16321 in /opt/zimbra/zmstat/pid/zmstat-fd.pid not running Running: zmstat-vm Running: zmstat-cpu Running: zmstat-df Running: zmstat-ldap Running: zmstat-nginx Running: zmstat-allprocs Running: zmstat-mysql Running: zmstat-io
Fix:
rm /opt/zimbra/zmstat/pid/zmstat-fd-real.pid rm /opt/zimbra/zmstat/pid/zmstat-fd.pid zmcontrol restart
2) no longer access statistics (no data, etc)
reference link
zmloggerctl stop cd /opt/zimbra/logger rm -rf db /opt/zimbra/libexec/zmloggerinit zmloggerctl start zmlogswatchctl start</pre>
