This is an old revision of the document!
Backups and recovery
When it comes to backup, the CzechIdM deployments consists of three parts:
- Identity manager's data in the PostgreSQL database.
- Application's
idm.war
archive in the Tomcat. - Configuration folders under
/opt/czechidm
.
Depending on your deployment, there can also be sets of scripts, the Vault and so on. But those are highly deployment-specific.
Repository backups
Using PostgreSQL allows us to do the pg\_dump
of the repository which is a primary backup strategy. Because identity manager contains company's private data, it is desirable to encrypt the repository backup. We will store all backups in the /opt/backup/…
location. To create encrypted backups, we will use following shell script:
- enc_backup.sh
#!/bin/bash # ********************************** READ ME ********************************** # # General: # Script is intended to do encrypted backups of whatever you implement in parts # "do the dump" and "pack the dump". The result of your doing should be a tar # archive called "current_backup.tgz". This name is automtically recognized and # script will take care of everything else. Presumed shell is BASH. # # Output of the script is saved into BACKUP_LOC directory in an encrypted form. # Each backup consists of two files - symmetric key and public key. Because en- # cryption is done by openssl, which cannot process an arbitrary file directly # with RSA, files are first encrypted with random 32B key using AES-256-CBC. # This 32B key is encrypted with RSA public key which is stored on the machine. # Private RSA key SHOULD NOT be found anywhere on the same machine. If it was, # you could do plain backups and not bother with this at all and security would # be the same. # # Needed binaries and builtins: # test,echo,stat,id,tar,openssl,touch,chmod,rm,mv,find,date,basename # # Setup: # 1) Create separate system user to run this script, do not run it as root. # 2) Generate public-private key pair of at least 2048b: # openssl genrsa -out backups-rsa-key 2048 # openssl rsa -in backups-rsa-key -out backups-rsa-key.pub \ # -outform PEM -pubout # 3) The backups-rsa-key file contains private key, store it in the keepass # or somewhere safe. Do not leave it on the machine! # 4) Move backups-rsa-key.pub to BACKUP_ROOT, set correct privileges (400), # name it as you wish and set RSA_ENC_KEY_FILE accordingly. # 5) Fill in the "do the dump" and "pack the dump" parts of the script to suit # your needs. # 6) Adjust other settings in the script as needed. Ensure that service user # used for dumping the DB, LDAP, whatever is dedicated to this and has # read-only privileges! This is IMPORTANT! # 7) Run the script as a cronjob. Preferred setting is in the crontab, not in the # /etc/cron.*/whatever file. But it does not really matter. # # Recovering backups: # Backups are stored in BACKUP_LOC as a pair of files. One file is an actual # backup encrypted symmetrically. The other file is a symmetric key for the # specific backup. (New symmetric key is generated for each backup run.) # Symmetric key is encrypted with RSA. # # To recover backups, do the following: # 1) Get you backups, we will call them "data.tgz.e" and "key.bin.e". # 2) Get your private RSA key "backups-rsa-key". # 3) Decrypt the AES key, you will obtain "key.bin" file: # openssl rsautl -decrypt -inkey backups-rsa-key \ # -in key.bin.e -out key.bin # 4) Decrypt the actual backup, you will get a tarball: # openssl enc -d -aes-256-cbc -in data.tgz.e -out data.tgz \ # -pass file:key.bin # 5) Extract the tarball: # tar xf data.tgz # ***************************************************************************** # # TODO: # * better backups naming # * something like .d directory where backup scripts will lay to make whole # thing a bit more modular # * add actions like "init", "recover" and "backup" to make script more # user-friendly # # Revision history: # 2017-05-16 Petr Fiser <petr.fiser@bcvsolutions.eu> # * removed hardwired LDAP variables (original script was for LDAP backups) # * removed hardwired lockfile name # * PASS_FILE made optional # * backup timestamp with granularity to seconds instead of hours # 2016-02-25 Petr Fiser <petr.fiser@bcvsolutions.eu> # * first version of the script # basic setup export PATH="/bin:/usr/bin" #directory where everything happens #should be empty except for backup scripts, keys and BACKUP_LOC folder BACKUP_ROOT="/opt/czechidm/backup" #hic sunt backupes BACKUP_LOC="${BACKUP_ROOT}/repository" #lockfile RUN_LOCK="${BACKUP_ROOT}/`basename ${0}`.lock" BACKUP_PREFIX="backup." BACKUP_SUFFIX=".tgz.e" BACKUP_AES_KEY_PREFIX="backup." BACKUP_AES_KEY_SUFFIX=".aes.key.e" #files with public RSA key and password file #if the password file is not needed, leave the setting blank or the file nonexistent PASS_FILE="${BACKUP_ROOT}/user.passwd" RSA_ENC_KEY_FILE="${BACKUP_ROOT}/repo-backups-rsa.pub" #backups retention period BACKUP_KEEP_DAYS="30" # setup runtime variables NOW=$(date +"%Y-%m-%d-%H%M%S") BACKUP_FILE_NAME="${BACKUP_PREFIX}${NOW}${BACKUP_SUFFIX}" BACKUP_AES_KEY_FILENAME="${BACKUP_AES_KEY_PREFIX}${NOW}${BACKUP_AES_KEY_SUFFIX}" # check root, must not run as root if test "$EUID" -eq 0; then echo "Script MUST NOT be run as root." >&2 exit 1 fi # check lock if test -e "$RUN_LOCK"; then echo "${RUN_LOCK} exists. Assuming ${0} already running." >&2 exit 1 fi # check binaries we need if test ! -x `which tar`; then echo "'tar' not found or not executable" >&2 exit 1 fi if test ! -x `which openssl`; then echo "'openssl' not found or not executable" >&2 exit 1 fi # check files privileges (euid==owner, egid==owner, file privileges r-- --- ---) if [ -f "$PASS_FILE" ]; then if test ! $(stat -c %a "${PASS_FILE}") -eq 400 || ! test $(stat -c %u "${PASS_FILE}") -eq "$EUID" || ! test $(stat -c %g "${PASS_FILE}") -eq `id -g`; then echo "File ${PASS_FILE} has incorrect permissions (should be 400) or owner/group (should be `stat -c %U ${0}`)." >&2 exit 1 fi fi if test ! $(stat -c %a "${RSA_ENC_KEY_FILE}") -eq 400 || ! test $(stat -c %u "${RSA_ENC_KEY_FILE}") -eq "$EUID" || ! test $(stat -c %g "${RSA_ENC_KEY_FILE}") -eq `id -g`; then echo "File ${RSA_ENC_KEY_FILE} has incorrect permissions (should be 400) or owner/group (should be `stat -c %U ${0}`)." >&2 exit 1 fi #create lock so we cannot run it more than once touch "${RUN_LOCK}" #cd to our working dir cd "$BACKUP_ROOT" #generate symmetric key here and push it (asymmetrically encrypted) into a file. this file will accompany symmetrically encrypted tar #we use aes-256 to encrypt our dumps so we need 32*8=256b symmetric key SYM_KEY=`openssl rand -base64 32` #encrypt the symmetric key openssl rsautl -encrypt -pubin -inkey "$RSA_ENC_KEY_FILE" -out current_key.bin.e <<< "$SYM_KEY" chmod 600 current_key.bin.e #do the dump # say we run the actual backup and create dump1.dmp, dump2.dmp and dump3.dmp here #pack the dump #tar usage "tar [parameters] archive_name file1 [file2 file3 ...]" tar --remove-files -czf current_backup.tgz dump1.dmp dump2.dmp dump3.dmp chmod 600 current_backup.tgz #encrypt the dump with current symmetric key, also add a pinch of salt openssl enc -aes-256-cbc -salt -in "current_backup.tgz" -out "current_backup.tgz.e" -pass stdin <<< "$SYM_KEY" #remove unencrypted dump and key rm -f current_backup.tgz #move encrypted things to backup_loc mv current_backup.tgz.e "${BACKUP_LOC}/${BACKUP_FILE_NAME}" mv current_key.bin.e "${BACKUP_LOC}/${BACKUP_AES_KEY_FILENAME}" #clean up backups older than $BACKUP_KEEP_DAYS days find "$BACKUP_LOC" -name "${BACKUP_PREFIX}*${BACKUP_SUFFIX}" -type f -mtime "+${BACKUP_KEEP_DAYS}" -delete find "$BACKUP_LOC" -name "${BACKUP_AES_KEY_PREFIX}*${BACKUP_AES_KEY_SUFFIX}" -type f -mtime "+${BACKUP_KEEP_DAYS}" -delete #we have finished, remove lock rm -f "${RUN_LOCK}" exit 0
The script needs a public-private keypair to be set up. When making a backup, a symmetric key will be generated. This symmetric key is used to encrypt the actual backup. Symmetric key is then asymmetrically encrypted by the public key and stored alongside the backup. For recovery, you need the backup and its corresponding symmetric key. When recovering, first, you have to decrypt the symmetric key with the private key generated earlier. Then you will use symmetric key to restore the data backup. For instructions about keypair initialization, backup creation and recovery and also for the actual command to carry out these actions, please refer to the script itself.
When you obtain the repository backup, you can restore the repository:
- Stop the identity manager container.
- Backup current repository somewhere else - in case you need to check some data later.
- Delete all data from the repository / drop the repository itself. This depends on the backup created - if you have database creation statements in there and such.
- Restore the data from the pgdump with
psql [parameters] < your_backup_name.sql
- Start the identity manager.
Script deployment
This is a standard deployment scenario. First, create the directory structure and setup the script and keys:
mkdir -p /opt/backup/database chown -Rf postgres:postgres /opt/backup chmod 700 /opt/backup/database # deploy the script above: vim /opt/backup/enc_backup_database.sh chown root:postgres /opt/backup/enc_backup_database.sh chmod 750 /opt/backup/enc_backup_database.sh # create public-private keypair on YOUR machine, NOT on the CzechIdM server # copy ONLY the public key to the CzechIdM server, STORE the private key SAFELY scp publickey.key czechidm-server:/opt/backup/backup_database-rsa.pub chown root:postgres /opt/backup/backup_database-rsa.pub chmod 440 /opt/backup/backup_database-rsa.pub
Then, configure the backup script:
BACKUP_ROOT="/opt/backup" BACKUP_LOC="${BACKUP_ROOT}/database" BACKUP_PREFIX="backup_database." BACKUP_AES_KEY_PREFIX="backup_database." RSA_ENC_KEY_FILE="${BACKUP_ROOT}/backup_database-rsa.pub"
Plain version of the script does not come with commands for making the actual backup, script serves as a wrapper which will encrypt whatever you need. Therefore, you have to check the script if there is actually a backup command specified and, if not, fill it in. If you want, you can add other things to the encrypted backup. See the script for lines:
#do the dump # say we run the actual backup and create dump1.dmp, dump2.dmp and dump3.dmp here #pack the dump #tar usage "tar [parameters] archive_name file1 [file2 file3 ...]" tar --remove-files -czf current_backup.tgz dump1.dmp dump2.dmp dump3.dmp
And change them to (expected name of the czechidm database is "idm"):
#do the dump pg_dump --create --dbname=idm > idm.sql #pack the dump #tar usage "tar [parameters] archive_name file1 [file2 file3 ...]" tar --remove-files -czf current_backup.tgz idm.sql
Last thing to do is to set up a cronjob. Here we set up a daily backup to 04:00. It may be useful to change the time the backup is done because there may be batch jobs running inside the identity manager. Set the job to run under the postgres user.
crontab -l -u postgres 00 04 * * * /opt/backup/enc_backup_database.sh
Configuration backups
Everything about the identity manager lives in /opt/czechidm
, the configuration itself in the /opt/czechidm/etc/
directory. Some files there (logging configuration, Quartz configuration and most parts of application property file) can be reconstructed from documentation. Crucial parts (repository password and especially secret.key
) have to be backed up.
- Losing repository password is not a problem (you can always set the new password in postgres) but it is a complication.
- Losing
secret.key
effectively means losing all contents of the Confidential Storage.
For simplicity reasons (and also because all the files count up to few kilobytes in size) we recommend backing up whole configuration folder. It is vital that this backup is encrypted, as it contains the confidential storage key.
Implementation
- For implementation, use the encrypted backups script as above.
- Create new instance of the script.
- Ideally generate separate RSA keypair for it.
- Do not pack the
secret.key
and repository backup together as this potentially lowers the security. (However, this somewhat depends where the backups are stored when off machine.)
Application backups
CzechIdM is a Java application distributed as a WAR archive. This archive is deployed inside a Apache Tomcat container. For recovery of the application, only the WAR is needed.
- If you use standard application release without any added modules and customizations, the
idm.war
does not need to be backed up. You can always download the application release again. - If you customized identity manager's binary in any way, you should rely on your version control system and development lifecycle and you should be always able to produce the
idm.war
again. - If you still want to backup the
idm.war
, simply copying it somewhere safe is completely sufficient. This could look something like:- backup_app.sh
#!/bin/bash BACKUP_ROOT="/opt/backup" BACKUP_DIR="${BACKUP_ROOT}/app" LOCKFILE="${BACKUP_ROOT}/`basename ${0}`.lock" BACKUP_KEEP_DAYS="30" NOW=$(date +"%Y-%m-%d-%H%M%S") if [ -f "${LOCKFILE}" ]; then echo "Backup is already running, exitting..." >&2 exit 1 fi touch "${LOCKFILE}" tar cpfz "${BACKUP_DIR}/backup_app.${NOW}.tgz" /opt/tomcat/current/webapps/* 2>/dev/null || echo "Error when performing backup." >&2 find "$BACKUP_DIR" -name "*tgz" -type f -mtime "+${BACKUP_KEEP_DAYS}" -delete rm "${LOCKFILE}" exit 0
When doing the recovery, simply put the CzechIdM WAR archive into the same folder as you would normally do and restart the Tomcat container.
Script deployment
If you really want to use the backup\_app.sh
, here is an example deployment scenario:
mkdir -p /opt/backup/app # this is because we usually do repository backup to sibling directories in the tree chown postgres:postgres /opt/backup chown root:root /opt/backup/app chmod 700 /opt/backup/app # deploy the script above: vim /opt/backup/backup_app.sh chown root:root /opt/backup/backup_app.sh chmod 750 /opt/backup/backup_app.sh
At last, set up a cronjob:
crontab -l -u root 50 03 * * * /opt/backup/backup_app.sh
In some cases, CzechIdM is not deployed with frontend and backend bundled together in the idm.war
. When backing up such environment, the backend should be backed up the way as was just described. The frontend, which may be deployed somewhere else, should be backed up in a similar way using the same script. For example, when running frontend application from separate Apache HTTPD, you should deploy another backup script which backs up /var/www/html/\*
directory instead of /opt/tomcat/current/webapps/\*
.