©2015 -
Information Technology
COUCHBASE BACKUP AND RECOVERY
Couchbase provides a backup tool -
In this article, we'll focus mostly on a backup/recovery of an entire cluster which is the most common production implementation.
For a full documentation of cbbackup options and usage, please refer to the Couchbase Developer doc -
For understanding the concepts of incremental backups, see Differential and Cumulative Backups doc.
From its documentation, I am re-
Tip: We recommended that cbbackup output be generated on a file system local to the server. Specifically, back up to a dedicated partition. A dedicated partition prevents the backup from filling the Couchbase Server data stores and operating system partitions.
Caution: Avoid routing the cbbackup output to remote share file systems (NFS). This is because cbbackup files are based on sqlite format which have issues being written to remote file systems.
The following is a brief description of the parameter options of the cbbackup command:
[couchbas@lxnode5 ~]$ cbbackup -
Usage: cbbackup [options] source backup_dir
Online backup of a couchbase cluster or server node.
Examples:
These are a full backup plus two incremental backups for a cluster.
cbbackup http://HOST:8091 /backup-
cbbackup http://HOST:8091 /backup-
cbbackup http://HOST:8091 /backup-
These are a full backup plus two differentials and one accumulative for a single node.
cbbackup couchbase://HOST:8091 /backup-
cbbackup couchbase://HOST:8091 /backup-
cbbackup couchbase://HOST:8091 /backup-
cbbackup couchbase://HOST:8091 /backup-
Note: A full backup task is always triggered for a new sink location
no matter what backup mode is specified.
Options:
-
-
single bucket from source to backup
-
not all server nodes from the entire cluster;
this single server node is defined by the source URL
-
-
-
-
connectivity and configurations
-
REST username for source cluster or server node
-
REST password for source cluster or server node
-
-
Number of concurrent workers threads performing the
transfer
-
is -
-
-
Provide extra, uncommon config parameters;
comma-
Available extra config parameters (-
backoff_cap=10 (Max backoff time during rebalance period);
batch_max_bytes=400000 (Transfer this # of bytes per batch);
batch_max_size=1000 (Transfer this # of documents per batch);
cbb_max_mb=100000 (Split backup file on destination cluster if it
exceeds MB); data_only=0 (For value 1, only transfer data from a
backup file or cluster); dcp_consumer_queue_length=1000 (A DCP client
needs a queue for incoming documents/messages. A large length is more
efficient, but memory proportional to length*avg. doc size. Below
length 150, performance degrades significantly.); design_doc_only=0
(For value 1, transfer design documents only from a backup file or
cluster); flow_control=1 (For value 0, disable flow control to improve
throughput); max_retry=10 (Max number of sequential retries if
transfer fails); mcd_compatible=1 (For value 0, display extended
fields for stdout output.); nmv_retry=1 (0 or 1, where 1 retries
transfer after a NOT_MY_VBUCKET message); recv_min_bytes=4096 (Amount
of bytes for every TCP/IP call transferred); rehash=0 (For value 1,
rehash the partition id's of each item; this is needed when
transferring data between clusters with different number of
partitions, such as when transferring data from an OSX server to a
non-
progress bar in console); report_full=2000 (Number batches transferred
before emitting progress information in console); seqno=0 (By default,
start seqno from beginning.); try_xwm=1 (Transfer documents with
metadata. 0 should only be used if you transfer from 1.8.x to 1.8.x);
uncompress=0 (For value 1, restore data in uncompressed mode)
[couchbas@lxnode5 ~]$
The following is an example of simulating a backup without dumping data.
This is useful for checking the parameters supplied to the cbbackup command for validity prior to running the actual procedure or embedding to a script.
Response example of a wrong password supplied:
[couchbas@lxnode5 backup]$ cbbackup http://lxnode5.vlabs.net:8091 /app/couchbase/inst1/backup -
error: unable to access REST API: lxnode5.vlabs.net:8091/pools/default/buckets; please check source URL, server status, username (-
[couchbas@lxnode5 backup]$
Response example of invalid destination directory level:
[couchbas@lxnode5 backup]$ cbbackup http://lxnode5.vlabs.net:8091 /app/couchbase/backup/week1 -
error: unknown type of sink: /app/couchbase/backup/week1
[couchbas@lxnode5 backup]$
Response example of correct parameters:
[couchbas@lxnode5 backup]$ cbbackup http://lxnode5.vlabs.net:8091 /app/couchbase/inst1/backup -
done, but no data written due to dry-
[couchbas@lxnode5 backup]$
NOTE: If there's no -
example:
[couchbas@lxnode5 backup]$ cbbackup http://lxnode5.vlabs.net:8091 /app/couchbase/inst1/backup/test1 -
[####################] 100.0% (4/estimated 4 msgs)
bucket: CRUD_Lab, msgs transferred...
: total | last | per sec
byte : 668 | 668 | 336.8
.
bucket: default, msgs transferred...
: total | last | per sec
byte : 668 | 668 | 443.1
[####################] 100.0% (6/estimated 6 msgs)
bucket: syncGW_CRUD_Lab, msgs transferred...
: total | last | per sec
byte : 2216 | 2216 | 818.7
[####################] 100.0% (3/estimated 3 msgs)
bucket: sync_gateway, msgs transferred...
: total | last | per sec
byte : 2312 | 2312 | 1309.6
[####################] 100.0% (31569/estimated 31569 msgs)
bucket: travel-
: total | last | per sec
byte : 31052988 | 31052988 | 737589.0
done
[couchbas@lxnode5 backup]$
Couchbase creates a single backup file (data-
Below is an example of generated backup files. Notice that there are directories created for each bucket that had been backup.
[couchbas@lxnode5 backup]$ cd test1
[couchbas@lxnode5 test1]$ ls -
total 4
drwxr-
[couchbas@lxnode5 test1]$ cd *
[couchbas@lxnode5 2016-
total 4
drwxr-
[couchbas@lxnode5 2016-
[couchbas@lxnode5 2016-
total 20
drwxr-
drwxr-
drwxr-
drwxr-
drwxr-
[couchbas@lxnode5 2016-
[couchbas@lxnode5 bucket-
total 12
-
drwxr-
drwxr-
[couchbas@lxnode5 bucket-
[couchbas@lxnode5 node-
total 18316
-
-
-
-
-
[couchbas@lxnode5 node-
If your data is of hundreds of gigabytes in size or even perhaps in terabytes, it is better to split-
You would use the parameter option -
Example of split backup files in 5MB size:
[couchbas@lxnode5 backup]$ cbbackup http://lxnode5.vlabs.net:8091 /app/couchbase/inst1/backup/test2 -
[####################] 100.0% (4/estimated 4 msgs)
bucket: CRUD_Lab, msgs transferred...
: total | last | per sec
byte : 668 | 668 | 329.9
.
bucket: default, msgs transferred...
: total | last | per sec
byte : 668 | 668 | 455.1
[####################] 100.0% (6/estimated 6 msgs)
bucket: syncGW_CRUD_Lab, msgs transferred...
: total | last | per sec
byte : 2216 | 2216 | 976.9
[####################] 100.0% (3/estimated 3 msgs)
bucket: sync_gateway, msgs transferred...
: total | last | per sec
byte : 2312 | 2312 | 1479.4
[####################] 100.0% (31569/estimated 31569 msgs)
bucket: travel-
: total | last | per sec
byte : 31052988 | 31052988 | 624102.4
done
[couchbas@lxnode5 backup]$
[couchbas@lxnode5 backup]$ cd /app/couchbase/inst1/backup/test2
[couchbas@lxnode5 test2]$ ls -
total 4
drwxr-
[couchbas@lxnode5 test2]$ cd *
[couchbas@lxnode5 2016-
total 4
drwxr-
[couchbas@lxnode5 2016-
[couchbas@lxnode5 2016-
total 20
drwxr-
drwxr-
drwxr-
drwxr-
drwxr-
[couchbas@lxnode5 2016-
[couchbas@lxnode5 bucket-
total 12
-
drwxr-
drwxr-
[couchbas@lxnode5 bucket-
[couchbas@lxnode5 node-
total 18604
-
-
-
-
-
-
-
[couchbas@lxnode5 node-
Note the file size of the generated split files (data-
SAMPLE COUCHBASE BACKUP SCRIPT
#!/bin/bash
# File: couchbase_backup.sh
# Author: Chad Dimatulac
# Description: This is a sample backup for Couchbase.
# A full backup kicks in at a designated day (i.e. Saturday) and will have a cumulative backup on succeeding days.
# The script uses the BACKUP_PATH/current as its active directory path to store the backup for the week.
# As the week cycles to the next full backup, the current dir data will be moved to archive and an empty
# current dir will be available to the backup process.
# To run:
# nohup ./couchbase_backup.sh &
#
LOG_DATE=`date +%Y%m%d%H%M`
DAY_OF_WEEK=`date +%a`
CB_HOME=/app/couchbase/inst1/opt/couchbase
PATH=$PATH:$CB_HOME/bin ; export PATH
CBBACKUP=$CB_HOME/bin/cbbackup
BACKUP_PATH=/app/couchbase/inst1/backup
CLUSTER_NODE_URL=http://lxnode5.vlabs.net:8091
RSTATUS=1
HNAME=`hostname`
RTUSER=`id |cut -
CBUSER=couchbas
DBU=Administrator
DBP=mypass
FBU_DAY=Sat
#-
if [ ! -
then
mkdir -
fi
#-
if [ ! -
then
mkdir -
fi
#-
if [ ! -
then
mkdir $BACKUP_PATH/current
fi
if [ ! -
then
mkdir $BACKUP_PATH/archive
fi
#-
if [ "$DAY_OF_WEEK" = "$FBU_DAY" ]
then
#-
CBB_MODE=full
else
#-
FCOUNT=`find $BACKUP_PATH/current -
if [ $FCOUNT -
then
CBB_MODE=full
else
CBB_MODE=accu
fi
fi
#-
CBB_LOG_FILE=${BACKUP_PATH}/log/couchbase_backup_${CBB_MODE}_${LOG_DATE}.log
echo >> $CBB_LOG_FILE
chmod 666 $CBB_LOG_FILE
echo Runtime Script: $0 >> $CBB_LOG_FILE
echo Runtime User: $RTUSER >> $CBB_LOG_FILE
echo PID: $$ >> $CBB_LOG_FILE
echo Hostname: $HNAME >> $CBB_LOG_FILE
echo Couchbase OS User: $CBUSER >> $CBB_LOG_FILE
echo ==== started on `date` ==== >> $CBB_LOG_FILE
echo >> $CBB_LOG_FILE
#-
if [ ! -
then
#-
echo $$ >> $BACKUP_PATH/.cbb_lock
#-
if [ "$CBB_MODE" = "full" ]
then
mv $BACKUP_PATH/current $BACKUP_PATH/archive/cb_backup_$LOG_DATE
mkdir $BACKUP_PATH/current
fi
echo Backup Mode: $CBB_MODE >> $CBB_LOG_FILE
CMD_STR="$CBBACKUP $CLUSTER_NODE_URL $BACKUP_PATH/current -
if [ "$RTUSER" = "root" ]
then
su -
RSTATUS=$?
else
/bin/bash -
RSTATUS=$?
fi
#-
rm $BACKUP_PATH/.cbb_lock
else
echo >> $CBB_LOG_FILE
echo WARNING: Backup aborted. >> $CBB_LOG_FILE
echo An existing backup process has held a lockout and may still be running, please check. >> $CBB_LOG_FILE
echo PID: >> $CBB_LOG_FILE
cat $BACKUP_PATH/.cbb_lock >> $CBB_LOG_FILE
echo >> $CBB_LOG_FILE
fi
if [ "$RSTATUS" = "0" ]
then
LOGMSG="Ended successfully"
else
LOGMSG="Ended in error"
fi
echo >> $CBB_LOG_FILE
echo Runtime Script: $0 >> $CBB_LOG_FILE
echo ==== $LOGMSG on `date` ==== >> $CBB_LOG_FILE
echo >> $CBB_LOG_FILE
#if [ $RSTATUS -
#then
#cat $CBB_LOG_FILE | /bin/mailx -
#fi;
exit $RSTATUS