Remark
Ingres and Vector properly handle files that are changing during backup. In Ingres all changes to files are logged in the dump location, Vector uses a snapshot for consistency.
It is important not to ignore the return code of the tar command because serious errors (such as disk full and other problems) must be detected. Unfortunately, the return codes are not always documented and there is not always a special return code that can be handled.
Solution
Depending on the platform there are different solutions for this problem.
Linux (using GNU tar)
For a longer time now GNU tar returns 2 for serious errors, but 1 for warnings. This can be handled in the cktmpl.def file. It is already done in Ingres 11, but the necessary changes could be done in Ingres 10.2 or Vector 4.2 and 5.0 too.
These are the lines in the cktmpl.def file which have been changed:
WSTD: cd %D; /bin/tar cbf 20 %C --warning=no-file-changed * || [[ $? -eq 1 ]]
WSDD: cd %D; /bin/tar cfS %A --warning=no-file-changed * || [[ $? -eq 1 ]]
WSTT: cd %D; /bin/tar cbf 20 %C --warning=no-file-changed %B || [[ $? -eq 1 ]]
WSDT: cd %D; /bin/tar cfS %A --warning=no-file-changed %B || [[ $? -eq 1 ]]
What does this actually mean?
Lets start with the end, i.e. the
|| [[ $? -eq 1 ]] part. That means in case where tar doesn't return with 0 we check the return value and if that is equally to 1 we return 0 (so the checkpoint becomes valid), if it is different, 1 is returned which is seen as error by the ckpdb command.
Please note that the
[[...]] syntax is bash specific, so this construct wouldn't work on other Unix platforms (we use the bash on Linux only) even if you would use GNU tar there.
The
--warning=no-file-changed flag suppresses the file changed warning, you could omit that if you still want to see the warning. Please test whether your tar version supports this warning flag. (It was probably added in tar version 1.23)
The uppercase S is used to handle sparse files efficiently, something which isn't used in Ingres 10.x, but in Ingres 11 when having x100 tables and in Vector.
If you have to change the cktmpl.def file it is a good idea to give the edited file a new name and to use the Ingres variable II_CKTMPL_FILE to point to the new version, otherwise a patch may overwrite the template file.
In Ingres 11 (and in Vector releases after VW 5.0) we distinguish between offline and online checkpoints. For an offline checkpoint we use the
WFDD line (if exists) and there we don't analyze the return code as we do with the changes described above.
In Ingres 10, VW 5.0 etc we don't look for this line, so we use the
WSDD line too and if you change the line as described above this will affect offline checkpoint as well, but this is not really a problem.
Finally, while not related to the issue discussed here, GNU tar allows to compress the checkpoint on the fly, you simply can add the z flag to the tar commands above, but don't forget to do this for the corresponding restore lines as well. (
WRDD etc.)
GNU tar is usually the standard tar command on Linux, tar --version displays the exact version.
Solaris (Sparc and x86_64)
The Solaris tar command (Sparc and x86_64) knows the -D flag which forces tar to handle file size changes while the file is being archived as warning and not as an error.
Using this flag avoids the need of the checkpoint wrapper script(explained below):
WSTD: cd %D; /bin/tar cbfED 20 %C *
WSDD: cd %D; /bin/tar cfED %A *
WSTT: cd %D; /bin/tar cbfED 20 %C %B
WSDT: cd %D; /bin/tar cfED %A %B
It is a good idea to give the edited file a new name and to use the Ingres variable II_CKTMPL_FILE to point to the new version, otherwise a patch may overwrite the template file.
Other Unix platforms
Using GNU tar
Gnu tar is available on other Unix platforms available too, so you could use GNU tar as well. As noted above you can't use the the
[[...]] syntax in the template file as this syntax is bash specific, but ckpdb doesn't use the bash on non-Linux.
The following is an example on how to use GNU tar on Unix - assuming GNU tar is saved as
/usr/local/bin/gtar - change this as appropriate.
WSTD: cd %D; /usr/local/bin/gtar cbf 20 %C --warning=no-file-changed *; s=$?;[ $s -eq 1 ] && return 0 || return $s
WSDD: cd %D; /usr/local/bin/gtar cf %A --warning=no-file-changed ; s=$?;[ $s -eq 1 ] && return 0 || return $s
WSTT: cd %D; /usr/local/bin/gtar cbf 20 %C --warning=no-file-changed %; s=$?;[ $s -eq 1 ] && return 0 || return $s
WSDT: cd %D; /usr/local/bin/gtar cf %A --warning=no-file-changed %; s=$?;[ $s -eq 1 ] && return 0 || return $s
Using the Wrapper Script
When not using GNU tar you can use the tar wrapper script. This script is called by the ckpdb command instead of the tar command. The script uses the standard tar command for backup. If the tar commands exits with 0, the script returns with 0. If the tar command returns with an error, it scans the list of printed errors and warnings. If there are any errors or warnings that are different than the file changing error, the wrapper script considers this as a serious error and returns with the return code of the tar command (that is, the checkpoint will become invalid). If there is no other errors listed, the script returns with 0 and the checkpoint continues.
Quick Guide how to activate the wrapper script:
cd $II_SYSTEM/ingres/files
sed -e "/^W/s^--II_SYSTEM^$II_SYSTEM^g" cktmpl_wrap.tpl >cktmpl_wrap.def
ingsetenv II_CKTMPL_FILE $II_SYSTEM/ingres/files/cktmpl_wrap.def
Please do the above steps as the installation owner (usually ingres or actian).
Use a small test database to verify that it works.
Details:
The wrapper script (cktmpl_wrap) and a template (cktmpl_wrap.tpl) for the checkpoint template file are delivered with the Ingres or Vector distribution
The cktmpl.def must be tailored to your platform and environment. Usually, there is no need to change the cktmpl_wrap file unless you are running an older Ingres version (2.6, 9.x without the fix for bug 122689). If using a localized tar command with non-English messages you may need to adjust cktmpl_wrap as well.
Within the template file, you must provide the full path to the wrapper script. The wrapper script must be added to the
$II_SYSTEM/ingres/bin directory. For security reasons, you must add the value of your $II_SYSTEM to the template file. We used
--II_SYSTEM inside the file to mark where to edit the template file and replace it with the actual value of $II_SYSTEM. Save the file as cktmpl_wrap.def. As described in the template file, you can use
sed for this.
(
sed -e /^W/s^--II_SYSTEM^$II_SYSTEM^g" cktmpl_wrap.tpl >cktmpl_wrap.def)
To activate the script you could replace
$II_SYSTEM/ingres/files/cktmpl.def with the new one, but is is better to set
II_CKTMPL_FILE to point to the new cktmpl_wrap.def using
ingsetenv (otherwise a patch may overwrite your cktmpl.def). Be sure that the file is owned by the installation owner (usually ingres or actian), otherwise the checkpoint will fail.
If you need to change the wrapper script itself, there are some variables that allow to configure the script. The list with the default values are provided below.
cmd="$1"
warning='file changed size$|file changed as we read it$|Error exit delayed from previous errors$'
expected="^a "
tmp_dir="/tmp"
The following is only for older releases as mentioned above:
cmd_cat='/bin/cat'
cmd_cp='/bin/cp'
cmd_egrep='/bin/egrep'
cmd_rm='/bin/rm'
cmd_wc='/bin/wc'
where
cmd:Contains the actual OS command to backup the files. Normally, this is defined in the checkpoint template file and given as the first argument to the wrapper script, so there should be no need to change it.
warning:These are the message to be ignored. Unless using a localized tar command with non-English messages, the default value is correct for Solaris, HP, AIX and Linux(or Gnu tar in general). This problem has not been observed on OSF (Tru64), but this may change in the future. To determine which messages your tar produces, backup a directory while copying a large file over the network into the directory. You may need to repeat the tar command several times to see the error.
expected:This contains a message we expect to see, but should be ignored. If you use the verbose flag (-v), the tar command logs the files added to the archive in the form ^a , where ^ is the Start of the line. Normally, there should be no need to change this.
tmp_dir:This is the directory used by the wrapper script for temporary files. You can choose any directory that is writable by the installation owner.
cmd_cat, cmd_cp, cmd_egrep, cmd_rm, cmd_wc:These variables contain the full path to the corresponding OS commands. This is needed for security reasons. Do not remove the complete paths. On most platforms, the default setting is acceptable, but on Linux you must change the setting of cmd_wc to cmd_wc=/usr/bin/wc (the script will generate an error if a command cannot be found).