Problem on CentOS 6.6 / Maria 10.0.16

[mrtdc]

Hi, I wanted to ask if this is known issue, or if there is a way to get more verbose error messages to give me a hint on what is the issue.

I am able to run innobackupex and successfully generate a backup via command line, approx thus:

innobackupex --user=root --password=SecretPassGoesHere /nfs-mount/dump-path/

however when I use xtrabackup script it throws an error, thus:

./xb-backup-incremental.sh -r /nfs-mount/dump-path/-u root -p SecretPassGoesHere
Command: mkdir -pv /nfs-mount/backuppc...20151206_1450 [ OK ]
Starting base backup. [ INFO ]
An exception occured while trying to use innobackupex.

and similarly when I look in the logfile,

[root@gpli-vsu2 xb-backup]# cd /var/log/mysql
[root@gpli-vsu2 mysql]# ls -la
total 12
drwxr-xr-x 2 root root 4096 Nov 25 14:08 .
drwxr-xr-x. 7 root root 4096 Dec 6 03:40 ..
-rw-r--r-- 1 root root 1328 Dec 6 14:50 xb-backup-incremental.log
[root@gpli-vsu2 mysql]# more xb-backup-incremental.log

...

2015-12-06T18:50:17+0000 - Executing: mkdir -pv /nfs-mount/backuppc/20151206_1450
2015-12-06T18:50:17+0000 - Command: mkdir -pv /nfs-mount/backuppc... - Status: OK
2015-12-06T18:50:17+0000 - Starting base backup. - Status: INFO
2015-12-06T18:50:17+0000 - An exception occured while trying to use innobackupex. - Status: FAILED
[root@gpli-vsu2 mysql]#

For ref:

host:

[root@srv mysql]# cat /etc/redhat-release
CentOS release 6.6 (Final)

MariaDB Version:

Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 45064
Server version: 10.0.16-MariaDB-log MariaDB Server

Just curious what might be the next step to debug this.

Thanks!

Tim

[deviantony]

Have a look at the logs files of the tool, especially the --out-file option: Log file for innobackupex output (default: /var/log/mysql/xtrabackup.out)

[mrtdc]

Hi,

thanks for the reply!

I had posted in my query, the content of the
/var/log/mysql/xtrabackup.out file already. It was equally concise. I
wasn't sure if there were 'secret' options to make the script more
verbose ?

Tim

---

Tim Chipman
Fortech I.T. Solutions
(902) 442-6633
http://FortechITSolutions.ca

------ Original Message ------
From: "Anthony Lapenna" notifications@github.com
To: "deviantony/xtrabackup-scripts"
xtrabackup-scripts@noreply.github.com
Cc: "mrtdc" tim.chipman@fortechitsolutions.ca
Sent: 12/8/2015 3:45:08 AM
Subject: Re: [xtrabackup-scripts] Problem on CentOS 6.6 / Maria 10.0.16
(#33)

Have a look at the logs files of the tool, especially the --out-file
option: Log file for innobackupex output (default:
/var/log/mysql/xtrabackup.out)

—
Reply to this email directly or view it on GitHub.

[deviantony]

Hey Tim, I just found out that you're using the bash scripts, which are deprecated ! Is this really necessary? I recommend you to use the Python version !

[mrtdc]

Hi,

No, not at all necessary. Rather, it was the only version I knew
existed. It seems I must have been led down the garden path, by finding
?now obsolete? web page where you discuss using the tool (bash scripts).

I believe the page was,
https://deviantony.wordpress.com/2014/05/18/percona-xtrabackup-scripts-part-2/

(maybe it is easily possible to put a 'warn' message at top of that
page, "Hey, the bash scripts are out-of-date, don't use them ! " :-)

Sigh! My mistake.

So - I will get the Python version, repeat my tests and let you know how
it goes.

Thanks for your help with this!

Tim

---

Tim Chipman
Fortech I.T. Solutions
(902) 442-6633
http://FortechITSolutions.ca

------ Original Message ------
From: "Anthony Lapenna" notifications@github.com
To: "deviantony/xtrabackup-scripts"
xtrabackup-scripts@noreply.github.com
Cc: "mrtdc" tim.chipman@fortechitsolutions.ca
Sent: 12/8/2015 9:14:14 AM
Subject: Re: [xtrabackup-scripts] Problem on CentOS 6.6 / Maria 10.0.16
(#33)

Hey Tim, I just find out that you're using the bash scripts, which are
deprecated ! Is this really necessary? I recommend you to use the
Python version !

—
Reply to this email directly or view it on GitHub.

[deviantony]

Yeaye I'll update my post !

[mrtdc]

Hi Anthony,

I wanted to briefly followup. (I can do this via git hub if you prefer
?)
(I am now using latest-current python based scripts ..)

I've had backups / running for a while now - and thought it would be a
good idea to test the restore process. (concept!)

So:

I was testing the restore-from-incrementals today. and I appear to hit a
snag. Curious if this is a known issue, or if I am simply not using
this restore process correctly.

Context:

• one full backup runs per month
• then other nights of month, incrementals run
• so the base data structure right now, looks as follows:

---paste---

[root@gpli-vsd2 bin]# cd /nfs-mount/backuppc/
[root@gpli-vsd2 backuppc]# ls -la
total 88
drwxr-xr-x 22 root root 4096 Jan 29 03:03 .
drwxr-xr-x 3 root root 4096 Oct 16 15:00 ..
drwxr-xr-x 3 root root 4096 Jan 10 01:00 20160110 << Base backup is
inside here, see below.
drwxr-xr-x 3 root root 4096 Jan 11 03:03 20160111
drwxr-xr-x 3 root root 4096 Jan 12 03:03 20160112
drwxr-xr-x 3 root root 4096 Jan 13 03:03 20160113
drwxr-xr-x 3 root root 4096 Jan 14 03:03 20160114
drwxr-xr-x 3 root root 4096 Jan 15 03:03 20160115
drwxr-xr-x 3 root root 4096 Jan 16 03:03 20160116
drwxr-xr-x 3 root root 4096 Jan 17 03:03 20160117
drwxr-xr-x 3 root root 4096 Jan 18 03:03 20160118
drwxr-xr-x 3 root root 4096 Jan 19 03:03 20160119
drwxr-xr-x 3 root root 4096 Jan 20 03:03 20160120
drwxr-xr-x 3 root root 4096 Jan 21 03:03 20160121
drwxr-xr-x 3 root root 4096 Jan 22 03:03 20160122
drwxr-xr-x 3 root root 4096 Jan 23 03:03 20160123
drwxr-xr-x 3 root root 4096 Jan 24 03:03 20160124
drwxr-xr-x 3 root root 4096 Jan 25 03:03 20160125
drwxr-xr-x 3 root root 4096 Jan 26 03:03 20160126
drwxr-xr-x 3 root root 4096 Jan 27 03:03 20160127
drwxr-xr-x 3 root root 4096 Jan 28 03:03 20160128
drwxr-xr-x 3 root root 4096 Jan 29 03:03 20160129
[root@gpli-vsd2 backuppc]# cd 20160110
[root@gpli-vsd2 20160110]# ls -la
total 12
drwxr-xr-x 3 root root 4096 Jan 10 01:00 .
drwxr-xr-x 22 root root 4096 Jan 29 03:03 ..
drwxr-xr-x 2 root root 4096 Jan 10 03:09 INC
[root@gpli-vsd2 20160110]# cd INC/
[root@gpli-vsd2 INC]# ls -la
total 3009624
drwxr-xr-x 2 root root 4096 Jan 10 03:09 .
drwxr-xr-x 3 root root 4096 Jan 10 01:00 ..
-rw-r--r-- 1 root root 2818468399 Jan 10 01:19
base_backup_20160110_0100.tar.gz << Base backup
-rw-r--r-- 1 root root 263371030 Jan 10 03:09
inc_0_backup_20160110_0303.tar.gz << First incremental, ran same day
[root@gpli-vsd2 INC]#

(note other intermentals are in the other directories, as per their date
stamps, sub directory INC, and the inc_#_ name is suitably incremented,
up to the last one:

[root@gpli-vsd2 INC]# pwd
/nfs-mount/backuppc/20160129/INC

[root@gpli-vsd2 INC]# ls -la
total 581132
drwxr-xr-x 2 root root 4096 Jan 29 03:08 .
drwxr-xr-x 3 root root 4096 Jan 29 03:03 ..
-rw-r--r-- 1 root root 595065971 Jan 29 03:08
inc_19_backup_20160129_0303.tar.gz

---endpaste---

So, then I try to kick off a restore ..

-- have a clean /var/lib/mysql directory
-- kick off the restore command, thus,

pyxtrabackup-restore
--base-archive=/nfs-mount/backuppc/20160110/INC/base_backup_20160110_0100.tar.gz
\

--incremental-archive=/nfs-mount/backuppc/20160129/INC/inc_19_backup_20160129_0303.tar.gz

--user=root --password=PASSWORD_GOES_HERE

and for reference, here is how I generated the (a) full backup once per
month (b) incremental backups each night:

(A) FULL once per month
/usr/bin/pyxtrabackup-inc /nfs-mount/backuppc --user=root
--password=PASSWORD_GOES_HERE --tmp-dir=/var/lib/mysql/temporary

(B) Incremental all nights per month (slightly later start time than the
full)
/usr/bin/pyxtrabackup-inc /nfs-mount/backuppc --user=root
--password=PASSWORD_GOES_HERE --tmp-dir=/var/lib/mysql/temporary
--incremental

And.

When I ran the pyxtrabackup-restore ... It gave me output thus: (after a
few minutes of grinding)

[root@gpli-vsd2 ~]# pyxtrabackup-restore
--base-archive=/nfs-mount/backuppc/20160110/INC/base_backup_20160110_0100.tar.gz
\

--incremental-archive=/nfs-mount/backuppc/20160129/INC/inc_19_backup_20160129_0303.tar.gz

--user=root --password=PASSWORD_GOES_HERE

INFO:xtrabackup.restoration_tools:Base backup restoration time:
0:02:55.552947 - Duration: 175
ERROR:xtrabackup.restoration_tools:An error occured during an
incremental backup restoration.
Traceback (most recent call last):
File
"/usr/lib/python2.6/site-packages/xtrabackup/restoration_tools.py", line
85, in apply_incremental_backup
archive_repository, prefix)
File
"/usr/lib/python2.6/site-packages/xtrabackup/filesystem_utils.py", line
95, in get_prefixed_file_in_dir
return files[0]
IndexError: list index out of range
ERROR:xtrabackup.restoration_tools:An error occured during the
incremental backups restoration process.
Traceback (most recent call last):
File
"/usr/lib/python2.6/site-packages/xtrabackup/restoration_tools.py", line
71, in restore_incremental_backups
self.apply_incremental_backup(repository, step)
File
"/usr/lib/python2.6/site-packages/xtrabackup/restoration_tools.py", line
85, in apply_incremental_backup
archive_repository, prefix)
File
"/usr/lib/python2.6/site-packages/xtrabackup/filesystem_utils.py", line
95, in get_prefixed_file_in_dir
return files[0]
IndexError: list index out of range
ERROR:xtrabackup.restoration:pyxtrabackup failed.
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/xtrabackup/restoration.py",
line 65, in main
arguments['--restart'])
File
"/usr/lib/python2.6/site-packages/xtrabackup/restoration_tools.py", line
148, in start_restoration
self.restore_incremental_backups(incremental_archive)
File
"/usr/lib/python2.6/site-packages/xtrabackup/restoration_tools.py", line
71, in restore_incremental_backups
self.apply_incremental_backup(repository, step)
File
"/usr/lib/python2.6/site-packages/xtrabackup/restoration_tools.py", line
85, in apply_incremental_backup
archive_repository, prefix)
File
"/usr/lib/python2.6/site-packages/xtrabackup/filesystem_utils.py", line
95, in get_prefixed_file_in_dir
return files[0]
IndexError: list index out of range
[root@gpli-vsd2 ~]#

and we have output dumped to disk, not a trivial amount, but not the
full amount for sure,

[root@gpli-vsd2 FAILED_restore-test]# ls -la
total 10006592
drwxr-xr-x 7 root root 4096 Jan 29 14:07 .
drwxr-xr-x. 21 root root 4096 Jan 29 14:09 ..
-rw-r--r-- 1 root root 360 Jan 10 01:00 backup-my.cnf
drwx------ 2 root root 12288 Jan 10 01:06 gpli_intranet
-rw-r----- 1 mysql root 475 Jan 29 14:07
gpli-vsd2.gpli.local.err
-rw-r----- 1 root root 10244587520 Jan 29 14:02 ibdata1
drwx------ 2 root root 4096 Jan 10 01:06 mysql
drwxr-xr-x 2 root root 4096 Jan 10 01:06 performance_schema
drwx------ 2 root root 4096 Jan 10 01:06 ssd
drwxr-xr-x 2 root root 4096 Jan 10 01:06 visualization
-rw-r--r-- 1 root root 40 Jan 10 01:06
xtrabackup_binlog_info
-rw-r--r-- 1 root root 28 Jan 29 14:02
xtrabackup_binlog_pos_innodb
-rw-r----- 1 root root 121 Jan 29 14:02
xtrabackup_checkpoints
-rw-r--r-- 1 root root 688 Jan 10 01:06 xtrabackup_info
-rw-r----- 1 root root 2097152 Jan 29 14:02 xtrabackup_logfile
[root@gpli-vsd2 FAILED_restore-test]# du -sh .
13G .

The date-stamps on some of the files, seems to be giving me a hint -
that they are as of Jan.10 versions; and a few are more current (today,
jan.29)

Anyhow.

If you see anything here that immediately says,

oh gosh, that is not correct - do this, not that!

or other thoughts - any comments are certainly appreciated.

(For example? I'm not 100% clear that the incremental restore script ..
is happily able to navigate the full historical data structure path for
all the incrementals? or do I need to do that manually somehow? Or
something else ?)

Many thanks for your assistance with this,

Tim Chipman

---

Tim Chipman
Fortech I.T. Solutions
(902) 442-6633
http://FortechITSolutions.ca

------ Original Message ------
From: "Anthony Lapenna" notifications@github.com
To: "deviantony/xtrabackup-scripts"
xtrabackup-scripts@noreply.github.com
Cc: "mrtdc" tim.chipman@fortechitsolutions.ca
Sent: 12/8/2015 9:14:14 AM
Subject: Re: [xtrabackup-scripts] Problem on CentOS 6.6 / Maria 10.0.16
(#33)

Hey Tim, I just find out that you're using the bash scripts, which are
deprecated ! Is this really necessary? I recommend you to use the
Python version !

—
Reply to this email directly or view it on GitHub.

i A

[deviantony]

Hey @mrtdc, I'm not really available right now, I'll try to have a look at this issue next week !

[mrtdc]

Hi, no worries! Thanks for the reply. I really appreciate it. Any
timeline is good, I appreciate your assistance.

Tim

---

Tim Chipman
Fortech I.T. Solutions
(902) 442-6633
http://FortechITSolutions.ca

------ Original Message ------
From: "Anthony Lapenna" notifications@github.com
To: "deviantony/xtrabackup-scripts"
xtrabackup-scripts@noreply.github.com
Cc: "mrtdc" tim.chipman@fortechitsolutions.ca
Sent: 1/30/2016 7:19:27 AM
Subject: Re: [xtrabackup-scripts] Problem on CentOS 6.6 / Maria 10.0.16
(#33)

Hey @mrtdc, I'm not really available right now, I'll try to have a look
at this issue next week !

—
Reply to this email directly or view it on GitHub.