Yce-Wiki version 7

This installation guide installs NetYCE version 7.x on a Redhat 7 or Centos 7 physical or virtual x86_64 platform.

References to EL or RHEL refer to RedHat Enterprise Linux or CentOS Linux. All OS versions and packages are required to use the x86_64 architecture, that is x86 processors running 64-bit. The installation applies to both physical and virtual platform deployments.

The choice of operating system (Redhat or CentOS), disk filesystem layout, installed packages, and security hardening are mostly defined by the customers common practice. NetYCE does have some requirements on disk-usage and directory-trees that may warrant filesystem allocations, and we do rely on a specific functional user, yce that requires some sudo permissions.

A basic set of packages should be installed which will later be amended by specific NetYCE software. The basic OS installation can easily be realized by the customer, but we recommend the NetYCE software installation and configuration to be a joint effort.

During the first install of the NetYCE software packages, the configuration preferences and details of the NetYCE system and its architecture will be defined and initialized. Subsequent software upgrades and patches can be installed by the application manager using the NetYCE front-end without requiring system privileges. Only on some major upgrades will those be required.

The NetYCE software installation consists of two self-installing packages, YCE and YCEperl, a sample database and a license file. The installation depends on MariaDB (mysql server), apache (http server), fping and some standard distribution packages (openssl, tftp, ftp, ssh, telnet, gtar, etc).

The hardware requirements of NetYCE are moderate by itself although much depends on the intended level of use and the application architecture selected.

In general we suggest to deploy two NetYCE servers in different data centers attached to Network Management (NMS) networks. These systems will provide both front-end (user and network facing) functions AND a database function. These functions can be configured to provide live failover and backup services by means of master-master replication. The front-end functions support 10-20 simultaneous users and can execute several thousand config changes per hour.

For such deployments a physical or virtual x86 server needs to have at least two CPU cores and 4 GB of memory, but 4 cores and 8 GB memory is recommended.

Disk space can be local or SAN based and should not exceed 50 GB. This disk space is allotted to a single filesystem or split across several, depending on system management preferences.

The NetYCE directory structure uses several trees for various functions. Assigning the mysql, shared and working/logs trees individual filesystems is recommended.

/ - 3 to 6 GB (OS root, bin, usr, lib, opt, etc)
/opt/yce - 100 MB
/opt/nms - 100 MB
/opt/ycelib - 500 MB
/var/opt/yce - 3 to 6 GB (logs and working data)
/var/opt/shared - 6 to 12 GB (os-files, NCCM backups)
/var/opt/mysql -  4 to 8 GB (mysql data)

You could choose to mount /var/opt/shared/public on a NFS server. This way every NetYCE server has access to the same data like OS files and NCCM backups.

Other directories are not relevant to be mounted over NFS.

During OS installation several groups of packages as a base install can be selected.

Package group selection:

• base (default)
• DNS (default)
• Development libs
• Development tools
• Editors
• FTP server
• Legacy netw server (default)
• Mail server
• Network servers (default)
• Server config tools (default)
• System tools
• Web server (default)

When installation is completed and the networking is setup, additional packages can be installed (or updated) using yum.

check programs - if not there: yum install <package>

• telnet
• cmake
• tftp
• openssl
• openssl-devel
• openssh
• mod_ssl
• mod_php
• wget

The command below, with all its arguments, will verify and install where needed, all the packages found on one of our servers. This is provided only for verification purposes. The fping package is not included in this list since it is not available using yum.

yum install -y  ConsoleKit ConsoleKit-libs SDL abrt abrt-addon-ccpp abrt-addon-kerneloops abrt-addon-python abrt-cli abrt-libs abrt-tui acpid alsa-lib alsa-utils at atk atlas autofs avahi-libs b43-fwcutter bc biosdevname blktrace bridge-utils btparser busybox bzip2 bzip2-libs cairo centos-indexhtml cpuspeed crda crypto-utils cryptsetup-luks cryptsetup-luks-libs cups-libs cyrus-sasl-plain db4-cxx db4-devel dbus dbus-python dejavu-fonts-common dejavu-sans-fonts desktop-file-utils dmidecode dmraid dmraid-events dosfstools dstat ed eggdbus eject elfutils elfutils-libelf elfutils-libs ethtool fontconfig fontpackages-filesystem fprintd fprintd-pam freetype gd gdbm gdbm-devel glibc-devel glibc-headers gnutls gtk2 hal hal-info hal-libs hdparm hesiod hicolor-icon-theme httpd-manual hunspell hunspell-en iotop irqbalance iw jasper-libs kernel-headers kexec-tools kpartx latencytop latencytop-common latencytop-tui ledmon libaio libedit libevent libfprint libgfortran libgssglue libjpeg-turbo libnl libogg libpcap libpng libproxy libproxy-bin libproxy-python libreport libreport-cli libreport-compat libreport-plugin-kerneloops libreport-plugin-logger libreport-plugin-mailx libreport-plugin-reportuploader libreport-plugin-rhtsupport libreport-python libtar libthai libtheora libtiff libtirpc libusb1 libvorbis libxcb libxml2-python lsof lzo man man-pages man-pages-overrides mdadm microcode_ctl mlocate mod_nss mod_perl mod_ssl mod_wsgi mtr nfs-utils nfs-utils-lib nfs4-acl-tools nspr ntp ntpdate ntsysv numactl numpy openldap-clients openssh-clients openswan oprofile pam_ldap pam_passwdqc pango parted pciutils pcmciautils perf perl-Archive-Extract perl-Archive-Tar perl-BSD-Resource perl-CGI perl-CPAN perl-CPANPLUS perl-Compress-Raw-Bzip2 perl-Compress-Raw-Zlib perl-Compress-Zlib perl-Crypt-SSLeay tcsh telnet tftp theora-tools time tmpwatch traceroute unzip usermode vconfig vim-common vim-enhanced vim-minimal virt-what webalizer wget wireless-tools words xdg-utils xz xz-lzma-compat yum-plugin-security yum-utils zip 

Commands are listed where needed. When the command listed starts with a # it denotes the command should be executed by the root user. The # mark can therefore also be read as (and typed as) sudo.

verify 64-bits:
$ uname -i
⇒ X86_64

verify SELinux is not active:
$ cat /etc/selinux/config
⇒ preferred SELINUX=disabled
⇒ workable SELINUX=permissive

verify ip settings:
$ hostname
⇒ hostname (pref not fqdn)
$ hostname --domain
⇒ domain name
$ hostname --ip-address
⇒ one (1) ip-address of the local interface
correct using 'setup'
correct in /etc/hosts

verify dns is configured:
- update /etc/resolv.conf is needed
- test using nslookup of a device
- check search path and domain

verify openssl is installed:
$ openssl
⇒ must start, then type 'quit'

verify rpm is functional:
- e.g. # rpm -v

verify a valid RedHat (or Centos) release is present.
$ cat /etc/redhat-release

⇒ Supported are RHEL6 releases 6.4, 6.5, 6.6, 6.7 and 6.8

To update a release to the latest RHEL6, connect the server to the internet and use the command (as root):
# yum update
When completed, reboot and verify using:
$ cat /etc/redhat-release

Should the upgrade not yield the expected version, consult the procedure in this link:
http://www.if-not-true-then-false.com/2011/upgrade-centos-6/

And retry including a cleanup:

# yum clean all
# yum update glibc* yum* rpm* python*
# yum update

Note: During the install or updates, yum will (re-)enable 'iptables'!
If your system's iptables are not configured, the default setting will only allow SSH connections and block all others, including httpd, mysql, yce_xch, yce_sched, etc.

To disable 'iptables':
# service iptables stop
# chkconfig --del iptables

Create group “nms” and user “yce”. All software will run as this functional user!
Example shows uid/gid 8000, but any unique value can be used

# groupadd -g 8000 nms
# useradd -g nms -m -u 8000 -s /bin/bash yce
# passwd yce

Adding the user yce to the cron allowed user list:
# echo “yce” >> /etc/cron.allow

A couple of 'services' will be installed in /etc/init.d for NetYCE:

- yce_psmon
- httpd
- mysql
- vsftpd

Of these, yce_psmon and httpd require 'root' permissions to start.
Since all application maintenance will (or should) be executed using the functional user 'yce', sudo should be setup to permit this.
The default setup expects /sbin/service to be available for the 'yce' user. Execution should not require a password.

Sudo is setup using the visudo command.
The example below uses four groups of command-aliases: YCE, SERVICES, SOFTWARE, PROCESSES that are used to configure one of the three permission levels for the members of the nms group.

# Yce
Cmnd_Alias YCE = /etc/init.d/yce_psmon, /opt/yce/system/init/yce_tftpd, /etc/init.d/httpd, /etc/init.d/mysql, /etc/init.d/vsftpd, /opt/yce/system/init/yce_psmon
# Services
Cmnd_Alias SERVICES = /sbin/service, /sbin/chkconfig
# Installation and management of software
Cmnd_Alias SOFTWARE = /bin/rpm, /usr/bin/up2date, /usr/bin/yum, /usr/bin/updatedb
# Processes
Cmnd_Alias PROCESSES = /bin/nice, /bin/kill, /usr/bin/kill, /usr/bin/killall, /usr/bin/pkill
# Networking
# Cmnd_Alias NETWORKING = /sbin/route, /sbin/ifconfig, /bin/ping, /sbin/dhclient, /usr/bin/net, /sbin/iptables, /usr/bin/rfcomm, /usr/bin/wvdial, /sbin/iwconfig, /sbin/mii-tool

# Storage
# Cmnd_Alias STORAGE = /sbin/fdisk, /sbin/sfdisk, /sbin/parted, /sbin/partprobe, /bin/mount, /bin/umount

## Delegating permissions
# Cmnd_Alias DELEGATING = /bin/chown, /bin/chmod, /bin/chgrp

# Cmnd_Alias SHELLS = /bin/sh,/bin/bash
# Cmnd_Alias SU = /bin/su
# Cmnd_Alias LOGIN = /bin/login
# Cmnd_Alias REBOOT = /usr/bin/reboot
# Cmnd_Alias SHUTDOWN = /usr/bin/poweroff, /usr/bin/halt, /sbin/shutdown

Defaults    !requiretty

#==== YCE user group 'nms'
# Below are a few examples. 
# For production the MINIMUM profile might be a good start.
# For testing, the MAINTENANCE is regularly used.

# MINIMUM
# No password required for YCE applications and services and processes. NO other applications are allowed to run at all!
# %nms ALL=NOPASSWD:YCE, SERVICES, PROCESSES

# RECOMMENDED: No password required for YCE applications and all applications are allowed if you know the sudo password
%nms ALL = PASSWD:ALL, NOPASSWD:YCE, SERVICES, PROCESSES

# DEVELOPMENT
# %nms ALL = PASSWD:ALL, NOPASSWD:SOFTWARE, YCE, SERVICES, PROCESSES
# %nms ALL = PASSWD:ALL, NOPASSWD:DELEGATING, NETWORKING, SOFTWARE, YCE, SERVICES, PROCESSES
# %nms ALL = NOPASSWD:ALL

During the YCE installation the sudo setup is examined so the appropriate launch and kill commands can be configured for the YCE daemons. The configuration file /opt/yce/etc/<hostname>_psmon.conf shows the results. Other processes will determine the sudo configuration dynamically (e.g. the daily database backup).

When sudo setup is altered, the appropriate modifications must be made to entries of the yce_psmon setup file. The configuration files are regenerated using /opt/yce/system/yce_setup.pl -r. Restart yce_psmon to activate the changes.

Sample section of the psmon.conf file:

<Process mysql>
      disabled    false
      ignoreflag  /opt/yce/etc/ignore_mysql
      spawncmd    /usr/bin/sudo /sbin/service mysql start
      killcmd     /usr/bin/sudo /sbin/service mysql stop
      pidfile     /var/opt/mysql/mysql.pid
      instances   1
      pctcpu      90
      noemail     False
</Process>

After making changes to the sudo configuration, verify its correct behaviour by issuing the resulting killcmd as 'yce'. When properly setup, the mysql database is momentarily stopped and then automatically restarted within 20 seconds.

A potential sudo configuration problem occurs when the sudo command still prompts for a password despite that the command is listed as a NOPASSWD (using sudo -l). This might be caused by the additional argument start or stop. Consider adding wildcards to the commands (/etc/init.d/mysql *) to allow for these arguments.

When Perl barfs at a missing locale setting correct this using:
vi /home/yce/.bash_profile
- Add: export LC_ALL=C

Some customer linux sytems have a filesystem setup where most applications subtrees have their own volume. The sizes need to be adjusted to match the required size. Use the command:
# lvextend -L <size> -r <fs-device>

On the filesystems below.

Check with the df -h command the actual device name

 mountpoint              size     device
/opt/nms
/opt/yce
/opt/ycelib            2G          /dev/mapper/vg.appl-lv.optycelib
/var/opt/yce           2G          /dev/mapper/vg.appl-lv.varoptyce
/var/opt/mysql         5G          /dev/mapper/vg.appl-lv.varoptmysql
/var/opt/shared        5G          /dev/mapper/vg.appl-lv.varoptshared

Typical systems are setup with separate filesystems for:

/opt                  10G
/var/opt/mysql         5G
/var/opt/shared        5G

NetYCE uses MariaDB for its database. MariaDB is derived from Oracle's MySQL but is free of its licensing terms and has evolved towards a more stable platform that is better suitable for distributed database applications.

Find your MariaDB repository: https://downloads.mariadb.org/mariadb/repositories/#mirror=tripleit

Select: RedHat (or Centos) - RedHat EL6 (64-bit) - 10.1

Copy the YUM repository information that resulted from this selection:

# MariaDB 10.1 CentOS repository list - created 2017-03-13 14:06 UTC
# http://downloads.mariadb.org/mariadb/repositories/
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.1/centos6-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

As root: Once you have your MariaDB.repo entry, add it to a file under /etc/yum.repos.d/. Create /etc/yum.repos.d/MariaDB.repo and insert the repository information copied above in this file

Should an earlier Mysql version be found, remove it using the --nodeps option. The RedHat EL6 uses a Mysql 5.1 library for its postfix (email) package. It should not be removed.

rpm -qa | grep -i mysql rpm --nodeps -e <package>

NOTE: do NOT use yum to remove -- it will also remove the dependent postfix!:

yum clean all
yum remove mysql-server

Then install MariaDB:

yum install MariaDB-compat MariaDB-common MariaDB-server MariaDB-client

Follow the instructions to complete the installation.

Alternatively, download the various packages from the 'baseurl' link in the repo information. The following files are required:
MariaDB-10.1.xx-centos6-x86_64-client.rpm
MariaDB-10.1.xx-centos6-x86_64-common.rpm
MariaDB-10.1.xx-centos6-x86_64-compat.rpm
MariaDB-10.1.xx-centos6-x86_64-server.rpm
MariaDB-10.1.xx-centos6-x86_64-shared.rpm
MariaDB-10.1.xx-centos6-x86_64-test.rpm
galera-2x.x.xx-x.rhel6.el6.x86_64.rpm
jemalloc-3.x.x-x.el6.x86_64.rpm

From the base repo, download the boost-program-options.
URL: http://ftp.nluug.nl/os/Linux/distr/CentOS/6.8/os/x86_64/Packages/
boost-program-options-1.xx.x-xx.el6.x86_64.rpm

Install them manually using rpm. Due to dependencies, install the required rpm's simultaneously. Place them all in the same directory and use the command below:

cd /path/to/package_dir
rpm --nodeps -Uvh *.rpm

Should an earlier Mysql version be found, remove it using the --nodeps option. The RedHat EL6 uses a Mysql 5.1 library for its postfix (email) package. Is should not be removed.

Upgrading MySQL 5.1 to MariaDB 10.0 on CentOS 6

 https://mariadb.com/blog/upgrading-mysql-51-mariadb-100-centos-6

Installing MariaDB with yum

 https://mariadb.com/kb/en/mariadb/documentation/getting-started/binary-packages/rpm/yum/

The YCE and Labs databases distributed should be validated by the new MariaDB engine. Run - as yce - the script /opt/yce/system/mysql_repair.sh or /opt/labs/system/mysql_repair.sh, depending on the product installed.

Likewise, create a new /etc/my.cnf using /opt/yce/system/yce_setup.pl -r (or use /opt/labs/system/labs_setup.pl -r). It should be installed automatically in /etc/my.cnf, otherwise copy from /opt/yce/etc/<hostname>_mysql.conf

Apache might be installed already, verify using
# rpm -qa | grep -i http
if present, the package will be listed
- Should apache needed to be installed, copy the httpd rpm
httpd-2.2.3-63.el5.centos.x86_64.rpm
- Install using:
# rpm -Uvh httpd-…
- If the dependency for /etc/mime.types is shown,
install mailcap first:
# rpm -Uvh mailcap-2.1.23-1.fc6.noarch.rpm
- Then resume installing httpd

Copy and install fping:

RHEL 6.x x86_64
Download the fping package from here: http://ftp.tu-chemnitz.de/pub/linux/dag/redhat/el6/en/x86_64/rpmforge/RPMS/fping-3.10-1.el6.rf.x86_64.rpm
Install it using: # rpm -Uvh fping-3.10-1.el6.rf.x86_64.rpm

The fping RPM doesn't support fping6 (for IPv6). In order to install fping6 the original source needs to be downloaded, compiled and installed using the following procedure:

NOTE: it requires gcc or equivalent to compile.

wget http://fping.org/dist/fping-3.10.tar.gz
gunzip fping-3.10.tar.gz && tar -xvf fping-3.10.tar
cd fping-3.10
./configure --prefix=/usr/local --enable-ipv4 --enable-ipv6
make
make check
make install
sudo setcap cap_net_raw+ep /usr/local/sbin/fping
sudo setcap cap_net_raw+ep /usr/local/sbin/fping6

Many customers will want to use SFTP or FTP for more secure and faster file transfer than TFTP. Starting at version 7.0, NetYCE supports SFTP and FTP using the 'Very Secure FTP server' named 'vsftpd'.

Install either through 'yum install vsftpd' directly from the Redhat/CentOS distribution server, or download and install the RPM package manually.

For downloading choose https://www.rpmfind.net/linux/RPM/centos/6.8/x86_64/Packages/vsftpd-2.2.2-21.el6.x86_64.html or one of the other mirrors available. Ensure the 'el6' and 'x86_64' version is selected.

Install the RPM using:

# execute as root:
su -
rpm -Uvh vsftpd-2.2.2-21.el6.x86_64.rpm

When the installation is completed, set it up as desired. Use the FTP and SFTP setup guide to configure vsftp.

A patch file is available to perform the required setup modifications:

# this patch should execute as 'yce' user,
# but requires the yce_psmon daemon to be running.

cd /opt/yce/system/patches
perl 14081902 -F -d

YCEperl is a self-installing binary that can be downloaded form the NetYCE Wiki site: https://wiki.netyce.com/doku.php/downloads:system_updates

The initial installation MUST be executed as root (to be able to create the directories), any later updates can be performed as the yce user.

YCEperl must be installed from the Linux command line. Updating is ONLY required when upgrading a major-release (6.x → 7.x) or a dot-release (7.2 → 7.3) if this is indicated.

Installation of the YCE perl distribution requires the file to be uploaded to the YCE server using the 'yce' functional user. Then, login as 'yce' and execute sh yceperl_7.0.2.bin.

Copy the yce_license file to the install location, or /opt/yce/etc if the directory exists.

During the YCE binaries install, the user will be prompted for the full path and filename of the license file. It will then be copied to its desired location: /opt/yce/etc/yce_license.

The path to the license file location may not contain any spaces. The license file itself should be readable by root or yce, depending on the user chosen to install the YCE binaries.

The YCE distribution images below can be downloaded from the NetYCE Wiki download page: https://wiki.netyce.com/doku.php/downloads:system_updates

The initial installation expects root to execute the installation, but for updates, the yce user is sufficient.

Start a NEW installation of yce using the command:

# sh YCE_<version>.bin

(e.g. sh YCE_6.2.1_20150910.bin)

For upgrades and patches the downloaded file can be installed using the Web-based front-end of NetYCE. From the Admin menu select System. The System status tool is activated by default. Please consult the Wiki page for details on performing the upgrade using this tool.

NetYCE images contain a full distribution set of NetYCE. Incremental installations are not required.

Following the binaries install, the user is prompted to configure the server setup for the YCE environment. At this stage all config files for the entire environment can be created. The relevant server information (name, domain, ip-address, role, database-id) should be available to the user at this time.

When choosing to configure the environment another time, the command /opt/yce/system/yce_setup.pl should be started as user yce.

The config files will be created in /opt/yce/etc. For each server, the config files will have the server name prepended (e.g. lsrv4439_httpd.conf) The config files created for other servers can be copied directly, or created locally using the same yce_setup.pl procedure.

It is essential that all config files are created using the same server information!

As part of the binaries install, patches are made to the system setup and/or the database. Patches are always incremental and often require the YCE database to be up an running. Since during the initial install the database will NOT be running, these patches will abort.

The patch installation should be completed at a later moment when the YCE database(s) is/are setup. Execute (as yce user) /opt/yce/system/patches/patch_install.pl to complete the installation. This should be repeated at each server in turn since some patches may apply the the local server installation and not to the shared database(s).

Copy and extract a valid YCE database.

An empty database can also be used. This empty database contains only the bare minimum, which is a user and password to access the front-end. This database is not encrypted. Customer based database archives are encrypted using the customer's license keys and can therefore not be used for distribution or initial setup.

Database: New database

The unencrypted YCE database can manually be extracted using the following steps.

as user root:

pkill mysql
rm -rf /var/opt/mysql
mkdir /var/opt/mysql
chown yce:nms /var/opt/mysql

as user yce:

su - yce
cd /var/opt/mysql
gtar xzpf /var/tmp/Ycedb_new_<date>.tgz  # assuming the file is located at /var/tmp

MySQL can be started and the new database is operational. If desired a customer YCE database archive can be restored using the front-end tools.

The Apache httpd server is only needed on servers including the YCE front-end function. This step might be skipped on servers providing the YCE database role only.

- Copy the httpd configuration file
# mv /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.org
# cp /opt/yce/etc/<hostname>_httpd.conf /etc/httpd/conf/httpd.conf
# chown yce.nms /etc/httpd/conf/httpd.conf

# mkdir /var/opt/yce/logs
# chown yce.nms /var/opt/yce/logs

# touch /var/opt/yce/logs/apache_error_log
# touch /var/opt/yce/logs/apache_access_log
# chown yce.nms /var/opt/yce/logs/apache_error_log
# chown yce.nms /var/opt/yce/logs/apache_access_log

- check and set httpd init script
ls -l /etc/init.d/httpd
# chkconfig --add httpd
Since the default httpd init-script does not specify the runlevels, these need to be set separately
# chkconfig --level 2345 httpd on

- And start!
# service httpd start

- Check for errors to fix:
less /var/opt/yce/logs/apache_error_log

YCE page will be reachable, but only the tool tree might show if perl fails, and no login is possible while mysql is unreachable. Even when mysql is running, access will fail until the YCE backend is fully functional (yce_skulker is required).
* http://<hostname>.<domain>

The MariaDB (MySQL) server is only needed on servers including the YCE database function. This step might be skipped on servers providing the YCE front-end role only.
The yce_setup will not have created a configuration file for systems not requiring one.

- Copy the mysql configuration file
# cp /opt/yce/etc/<hostname>_mysql.conf /etc/my.cnf
# chown yce.nms /etc/my.cnf

- Check and set init script
ls -l /etc/init.d/mysql
# chkconfig --add mysql

- And start!
service mysql start
less /var/opt/mysql/<hostname>.err

- In case compatibility problems listed:
mysql_upgrade --user=netYCE -p
service mysql stop
service mysql start
cat /var/opt/mysql/specter.mysql.org.err

Several daemons will be required before the YCE system becomes functional. The YCE process monitor will ensure the required processes are running.

Setup process monitor
As root:
# cd /etc/init.d
# cp /opt/yce/system/init/yce_psmon .
# chkconfig --add yce_psmon

Start back-end
# service yce_psmon start

Note: yce_psmon should be started as root. When other users start it, it will assume a different application and will look for a configuration file elsewhere (/etc/psmon.conf, ~/psmon.conf). These should not be created unless yce_psmon is used for other purposes than YCE.

The YCE web login should now be operational and allow logins. Also the YCE client should be able to connect and login.

The default user with manager permissions is netyce using the password netyce.

- Allow yce to use crontab
# vi /etc/cron.allow
Add yce user to the list
- Add the default crontab (as yce!)
cd /opt/yce/etc
crontab < sample_crontab.conf
- Check
crontab -l
- Edit
crontab -e
Comment out all references to dbarchive.pl for systems not running Mysql, and select appropriate (non-overlapping) times for the primary and secondary databases.

The MySQL database master/master setup is configured using the 'Db archives' tool when restoring a database. By restoring the the SAME archive set (near) simultaneously, the master and slave synchronisation between two YCE databases is prepared.

Then, using the 'System status' tool, Start the synchronisation slave first on one server, then on the other. Before starting the synchronisation slave on the second server, ensure the first one is running error-free.

Errors are flagged in the tool which also provides a 'Skip synchronisation error' button for SQL errors causing synchronisation conflicts. Reported SQL errors pertaining to the 'Server_setup' table can be skipped safely but should number no more than about 6 per operational server. Counters on the number of SQL updates and inserts pending on the current error is provided and updated after each 'skip'. If errors were encountered on one server that were resolved using this 'skip' procedure, then the same errors will have to be skipped when the second server has it synchronisation enabled.