$Id: CHANGES,v 1.159 2003/10/10 15:42:41 edwinh Exp $ $Name: v1_2_1 $ ---------------------------------------------------------------------- Version 1.2.1 (20031010) Changes: - Added "-ignore-errors" flag to allow backups to continue even if commands return non-zero exit status. Use at your own risk. - Added contributed manpages - Add 'rsync' type that acts like 'copy' but using rsync instead of piped cpio's. You need rsync >= 2.5.6 for this to work! - Added lzop compression support - Spinning bar indicator for long-running pkg delta operations if output is a terminal - Recognize .deb files as ar archives - Skip proc/devpts/devfs/tmpfs for traverse_fs=local or all - Add hooks for mbuffer's multivolume support (experimental) Bug fixes/cleanup: - FreeBSD pkgdelta support fixed up - Cleanup better if error detected - Fix quoting in check_remote_progs if your remote shell is csh - Make sure we don't generate conflicting index keys (as long as -newtape is not run in parallel) - Cleanup a bit regarding block devices ---------------------------------------------------------------------- Version 1.2.0 (20030707) Changes: - Merge devel -> stable - Added -pipe option to archive single directory to stdout, or list/extract/compare archive from stdin - No null padding necessary for reads, even with $pad_blocks true - only needed for writes. Should take harmless warnings way from gzip,tar, and others when reading - Added -nodefaults option for debug - Allow compression for filelist type Bugfixes/cleanup: - Fix FreeBSD find w/ prune regex - Truncate tar label to 99 chars - Error if -num flag with backup write mode or using file - Error if -erase with archive read mode or using file - Fix afio volume header print filehandle - Fix if on-disk backups, and "-rmfile all", the index for the keyfile was removed but not the keyfile itself - use $:: rather than $main:: for globals, its shorter... ---------------------------------------------------------------------- Version 1.1.8 (20030621) Changes: - Detect old config file - Config file parsing not so rigid - if values not found, it now uses defaults (w/ message printed so you know it) - Can configure buffer percent full and write sleep time options - Compare works for copy type (just diff -r -q) Bugfixes/cleanup: - Doh! index key creation broken for to-disk backups as a result of 1.1.7 changes. Fixed. - Relative links for $device get chased correctly - $pkgdelta_archive_changed = false didn't work. - README updated; example for filesystem setup & cron. - find -depth w/ -prune isn't supposed to work, no bug. (see single unix spec and various manpages). ---------------------------------------------------------------------- Version 1.1.7 (20030618) Changes: - Added "copy" type that just mirrors the files into a directory tree if archiving to disk - Initial support of "-pkgdelta" with FreeBSD packages - On FreeBSD, use mt to set 1-filemark behavior globally - Globally ignore sockets - Use "find -prune" for subtree pruning. (When this happens, we have to turn off "-depth" switch, as with GNU find 4.1.7 they don't work together. Need to check into findutils more.) Bugfixes/cleanup: - Don't use --same-order for tar. -extract -flist would fail if you had more than one file and they weren't in the expected order - Use --ignore-failed-read for tar - Remote tape drive device didn't create index key correctly - Typo/update star flags setup - Tweak quiet flags ---------------------------------------------------------------------- Version 1.1.6 (20030313) Changes beyond stable 1.0.3 release: - Different way to spec the filesystems. This lets us have arbitrarily named backup "sets". Config file: $set{'tag1'} = "/dir1 /dir2 ..."; $set{'tag2'} = "/dir3 host:/dir4 ..."; Flags: "-dir " backs up one directory tree (old "-fs ") "-set " backs up a named set. Useful new feature. "-set all" acts like old "-fs all". If using tapes & level 0, each set is a different tape (acts like the old numeric index to @filesystems) - Per-filesystem subtree pruning $prune{'/'} = "tmp proc mnt"; $prune{'host:/'} = "disk.?"; - Added packaging system delta capability. If you give "-pkgdelta rpm", after the level/timestamp checks, we further cut down the backup file list by only archiving files that are either (a) not owned by a package, or (b) owned by a package but modified from the distributed version. New config variables to tweak behavior: $pkgdelta_archive_list $pkgdelta_archive_unowned $pkgdelta_archive_changed - Added 'mbuffer' program as alternative to 'buffer' - Added 'filelist' type that doesn't back up anything, but prints/saves the list of files that would have been archived. Good for debugging timestamps, levels, and file exclusion. - Added "-extract -onefile " option; if you are extracting a single file just spec it on the command line, no list file needed - Changed boolean mt_var_blksize into mt_blocksize integer (in bytes). Set to 0 for variable blocksize (old "true" value), comment out or set to "$blksize * 1024" for old "false" value. - If staticfiles or staticlogs true, don't timestamp list/extract/compare log - Added -wday option (helps with once-a-month, on a certain day of the week only cron setups) - test-tape-drive now diffs 3 files, one in the middle is different (will catch people using rewind-on-close device) - "-extract -files" -> "-extract -flist" to hopefully be more clear - db/file rm flags tweaked: Redid rmindex semantics again, see help -rmfile can be specified multiple times -rmindex can be specified multiple times - Added support for lha, shar, and ar archives just to be stupid Cleanup: - Allow dirs/prune paths with spaces in them - to use, enclose all items in the lists in quotes. - When level > 9 and using files, -toc sorts better now - Config file with spaces in path works - Don't check for writeable stampdir for non-backup - Don't check for writeable logdir for non-backup - Don't tie to db for non-backup ---------------------------------------------------------------------- Version 1.0.3 (20030313) Bug fixes: - Only look for mt if we are using tape device - If using files, "-fs all", and and multiple array entries for $filesystems, index was getting hosed up. - newtape/erase when using dirs was deleting keyfile entry from index db - remove bogus OS_ERROR string from failed tie ---------------------------------------------------------------------- Version 1.0.2 (20030219) Changes: - Added '-test-tape-drive' switch that just makes sure blocks, filemarks, and padding work as expected and we can read,write,compare. If people can't get that running, then the problem is somewhere else... - Allow mt command override in the config file: mt{'old'} = 'new'. Useful if there is a new platform-specific combination, you want 'status' vs. 'tell' output in the log, etc... Bug fixes: - If remote tape device, don't check for writable local device - Fix pattern-match so "sudo -u username command" works (Tweak sudo comment in config file and faq as well) Cleanup: - Update mt op used for hardware compression (if anyone uses it) - Cleanup more uses of $_ ---------------------------------------------------------------------- Version 1.0.1 (20030202) Bug fixes: - Quote the directory when doing initial "cd" (logfiles, stamps, filenames too). Also needed to fix a few regexps that matched against the backup dir. - afio + null fix in pre6 didn't work in FreeBSD due to a problem with its printf(1). Switch back to newline-separated for afio types for now. Linux, HPUX, Solaris, AIX printf(1) utilities did do nulls. ---------------------------------------------------------------------- Version 1.0pre6 (20030126) Bug fixes: - Fix problem with afio volume headers and null-terminated file lists. The first file in afio archives was being missed (actually showed up in the label instead of the archive) Cleanup: - "-rmfile" yells if no argument - Move shell/remote/buffer check to its own function after option checking. (So commandline/config errors and these pre-tests are displayed separately) ---------------------------------------------------------------------- Version 1.0pre5 (20030124) Bug fixes: - Fix sh status collector for the front of remote commands - Fix it for commands in ( .. ) subshells also (echo + find for afio volume header input for example) Cleanup: - Don't do status collection work for non-pipelined remote commands ---------------------------------------------------------------------- Version 1.0pre4 (20030121) Bug Fixes: - Try harder to catch pipeline errors in different environments: o zsh has a pipestatus array variable similar to bash, use it o Tweaks to shell detection to detect remote vanilla csh o Add exit-status recording shell trick for local sh - If we get an error, rm index & botched archive file if using files - If we get an error, mark problem in the index if using tapes Cleanup: - Replaced manual line-wrapper with Text::Wrap (other was buggy, missed single-char "words") ---------------------------------------------------------------------- Version 1.0pre3 (20030119) Bug Fixes: - Fix PIPESTATUS trick to not require extra 'seq' utility, just a while loop - Using "-toc" gave uninitialized var warning, obvious problem, fixed. Eradicated some other spots of code that used $_ too. - "-rmindex" yells if you don't give an argument Cleanup: - Cosmetic - fix missing line in output if no remote and buffer=false - Remove duplicated index delete code ---------------------------------------------------------------------- Version 1.0pre2 (20030118) Bug fixes: - atime preserve option to tar/star/afio modifies the ctime of the files. Using ctime checks then says everything is newer, no matter what the level. Make atime preserve a global option defaulting to false; if true turn off ctime checks - When reading from given filename, set device to dir so that we don't pad blocks by accident if default device is tape - Detect shell type for local and any remote hosts Only use bash PIPESTATUS trick if we find bash 2.x ---------------------------------------------------------------------- Version 1.0pre1 (20030118) Changes: - Allow non-root execution along with "sudo". Set path{'command'} = "sudo command" or "sudo -u username command" in the config file. Idea from Alex Aiminoff's patch to 0.9.8 - Added $remoteuser to config for rsh/ssh - Check that 'buffer' can run with set $megs. From John Reynolds, brought forward from 0.9.8. Remote as well. - Check for existence of remote executables before doing remote backups Based on John Reynolds' modifications - A "no blocking at all" option. Set $blksize=0 in the config file. - Added support for "zip" as valid archive compression type - Added support for "zip" as valid log compression type - "-device to override config file (tired of "-d device=/path/to/dir" during testing...) - "-rmfile all"' when using files will rm all files/db entries for that dir Bug Fixes: - $erase_rewind_only wasn't working right if we didn't call &mt('rewind') right before or after &mt('erase') (I think all spots are covered now, but anyway...). Now we might call rewind twice if this is set, but it won't hurt. (from e-smith patches) Cleanup: - Redid the way buffer/read/write strings were constructed to make more sense - Don't set tape blocksize with -toc unless we want current tape (from John) - Interactive restore creates log file (was in e-smith stuff) - Extract/list/compare/restore use date-stamped logfiles (e-smith) - Simplified rmfile() to call existing rmindex() routine - Symlink latest log even if staticlogs = true ---------------------------------------------------------------------- Version (20030115) Changes: (just so we can say it supports *everything* for archive formats) - Initial support for "star" archiver. Like tar but does acl's, has its own fifo buffering mechanism, and more. see: http://www.fokus.gmd.de/research/cc/glone/employees/joerg.schilling/private/star.html - Initial support for "pax" archiver. Really just tar or cpio, but might be good for portability across unices... - Put .zip back in, same way as in old releases (kinda hokey with the tmpfiles, but there is no cleaner way to do it with info-zip utils like I wanted) Bug fixes: - Fixed problem with other releases trying to use find w/ -cnewer. Desired so we get files that change perms, not content. ctime can't be set backward with touch -t (or any mechanism), so we can't back-date a reference file. Now fixed so at least for local filesystems this works. Remote backups still won't catch that type of change. Code changes: -don't use generated timestamp files for local fs's; use the original stampfile -refactor remote fs variable -get_last_date function changes -no stamps at all for level 0 Cleanup: - '-rmindex all' says it will wait 5 seconds - '-incremental|full|differential' same as '-level ' - Fix warning with no args - touch -t the same for everyone. buggy touch in pre-4.0 fileutils is old by now.. - Clean up date routines ---------------------------------------------------------------------- Version (20030112) Changes: - Can now backup to / read from remote tape drives. Set $device = "host:/dev/tape" Uses ssh pipelines like our remote fs backups - not rmt. Old mail idea from Caleb Crome , reworked. Bug fixes: - Some commands still weren't using path overrides (dd,mt) Cleanup: - Refactored some code due to the above. - afio volume header had extra newline - File list routine to nuke duplicated code - Echo list/extract log filename. - Ffind -depth is recommended. - Add .tbz auto file extension. - Fix incorrect tapedevice test in read setup function. (cosmetic result) ---------------------------------------------------------------------- Version (20030111) Changes: - Static filenames option when using disk (same fs, same level overwrite) based on 0.9.8 patch from Michael Matsumura split into $staticlogs / $staticfiles config options Bug fixes: - Old bug: stamp files from the various parts of 'all' were not getting marked as 'old' correctly for removal (we were looking for "all.") - Fixed '-level incremental' with fs=all. Had to refactor some code and not use a global level variable. logs filenames a little different for 'all' in this case. Cleanup: - Make path overrides also search $PATH (say if we just changed the name tar == gtar but still want to do normal path-search) - $nocompress_types -> $afio_nocompress_types since that's the only one that uses it - A few cosmetic fixes: fix double dash-line on last line of logfile make logging routines suable for screen/log/both fix so -n won't nuke old files either ---------------------------------------------------------------------- Version (20030110) Bug fixes: - DOH! we *were* doing levels slightly wrong. use < rather than <= for comparing levels. popped out of investigating incremental and differential requests Now we act like dumplevels should - This also changes what we regard as "old" logs/stamps as a result Changes: - Several requests were in old email for differential/incremental... Just a misunderstanding I think. Check these links: http://docs.sun.com/db/doc/805-7228/6j6q7uf18?a=view http://www.arkeia.com/archives_indexed/2000/10/msg00006.html http://www.sans.org/rr/backup/central.php To sum up, no changes are *needed* if dump behavior mimiced (we had a bug here BTW). We can do what people call "incremental" as long as we increase the backup level each day. We can do what people call "differential" just by keeping the number the same for consecutive backups - Implemented support for -level [full|differential|incremental] as aliased dump levels (for people not as familiar with dump-style levels) full = 0 differential = 1 incremental = + 1 [broken for fs=all at this stage] - Support for > 9 levels (for all types but 'dump') ---------------------------------------------------------------------- Version (20030109) Bug fixes: - BSD find wants "!" not "-not" ---------------------------------------------------------------------- Version (20030107) Bug fixes: - 'Argument "200301072208|0" isn't numeric in numeric comparison' when using tapes... redid toc routine to use two hashes for disk/tape files, sort them differently - Existing tape indexes weren't found to a missed edit during the past few days' cleanup ---------------------------------------------------------------------- Version (20030107) Changes: - use -print0 with find and null-terminated list flags for the archive programs. Now any weird filenames should be valid. This necessitated using find -regex rather than an egrep pipeline. (FreeBSD find now has "-regex" so the old argument is gone). - also took out the sed of the leading "./" since the null-terminated file list messed it up too. Was just for looks anyway. - cpio flags use the short options so more compatible with other cpio's - cleaned out tape-only messages when using files Bug fixes: - Pipeline error status - if something went wrong with a shell pipeline, and it was not the _last_ command, you can't catch it! I never knew this was impossible... bash has a PIPESTATUS array variable for this, so I took care of it for systems where SHELL is bash... tcsh seems to set $? if anything fails, so it's ok too. I don't know if there is a way to fix it generically. - Actually log output. doh! (related to above, actually) - afio compress programs honor the path overrides ---------------------------------------------------------------------- Version 0.9.9 (20030106) Wonder of wonders... I'm actually using this again. The guy who registered flexbackup.sourceforge.net never picked up the files or updates his page so I'm flipping pointers back to sourceforge.net I guess. I'm mainly just using on-disk files now - will try to keep tape operation tested. Changes: - changed the index delete flags worked, it was too confusing overloading the -toc flag (WTF was I smoking 2 years ago?). Now toc only lists stuff, and db-fixing can be done with: flexbackup -rmindex all flexbackup -rmindex flexbackup -rmindex -f - added "-rmfile " option if we are backing up to disk. nukes archive and its related index entry - take the seconds field out of the archive + log names... it was just too long to interact with, and who really does more than 1 backup in a minute? - removed zip archive support. They just don't like to work with stdout/stdin like other unix tools and are a pain to special case. - "-newtape" when using files will rm the files in the dir - support non $PATH or hardcoded locations to commands in config file: $path{'program'} = /path/to/program; Bug Fixes: - doh - can't use find -cnewer & a reference file with touch -d together since the change time is still "now". I remembered the suggestion but the long lapse in devel time made me forget to see if it worked. REMOVED cnewer mistake. - default config uses /var/lib rather than /var/state (FHS spec on this went away long ago) - don't pad blocks when archiving to files on disk - was using numeric sort when listing alphanumeric files - "-num" invalid when using files - newer cpio didn't like sparse in --create mode - errors in command chain weren't detected. now separate some of the steps; remote still hosed up ---------------------------------------------------------------------- Version (20020824) Revive the tree Move support info/web site so this isn't stagnant anymore Check in what I had pending New Features: - works for block devices (like floppies). Turns the indexing off since you can't do multiple files, but otherwise it works fine Bug fixes: - use "&&" rather than ";" in between commands - use "-cnewer -or -newer" rather than "-newer" - set locale so touch works - tar - take off "! -type d" since we have --no recursion - log/backup files now use yyyymmddhhmm.ss times in the filenames. Previously you could get messed up if you ran two backups on the same minute; files would clobber themselves - OpenBSD didn't have "mt rdhpos". Split the BSD flavors up. - add a rewind before erase when deleting old index - "-n" was touching the stamp file, now it won't ---------------------------------------------------------------------- Version 0.9.8 (19991102) New Features: - add -newtape option to initialize index, but no backup - add -num option for reads that will position the tape at the specified file number before reading - added config file option for whether "erase" actually calls "mt erase" or just rewinds - for some drives, mt erase takes hours... (variable is $erase_rewind_only) - tried to add intelligence for various mt flavors on different Unix variants - made block padding a config file option ($pad_blocks). conv=sync was causing problems for Clemmit on OSF1. Bug fixes: - sort the toc output correctly - rewind after printing toc - fix bug where tape indexes always started at 10 after awhile (similar sorting problem with setting $main::nextfile) Cleanup: - web site moved - other minor cleanups to tape key handling - "-n" bypasses index operations as well - fix some messages/formatting ---------------------------------------------------------------------- Version 0.9.7 (19991019) New Features: - table-of-contents support - tells you which backups are on which tapes. uses a simple "key" on the tape plus on-disk database files - added "traverse mountpoints" config option for non-dump types can skip nfs mounts or really do everything - added config option to turn off assumption that level zero "all" should erase the tape Bug fixes: - add "conv=sync" to dd flags so it will pad blocks some tape drivers complain if writes are not padded "buffer" already did this since I used the B flag by default - don't use `date ..`, but POSIX::strftime() instead incremental backups on FreeBSD were busted (no date -r) - using $ in exclude expressions broke remote backups if the user's login shell was csh/tcsh due to it's insane syntax quirks. Worked around it putting a trailing $ outside the second level quotes - works on OSF/1 boxes Cleanup: - remove ftape table of contents now that I have a generic one - autodetect Linux ftape device, eliminates configuration - change time format slightly for consistency - cleaned up handling of reten/erase/rewind options - tape position display under FreeBSD is just "mt rdhpos", under others is "mt tell" ("mt status" cluttered things w/ lots of useless info) ---------------------------------------------------------------------- Version 0.9.6 (19991003) New Features: - buffing is optional - log files for list/extract/compare (flexbackup..log in cwd) - can now have more than one exclude expression defined - add -norewind flag - make symlink to latest log file - use our own timestamps for "dump", with -T config file variable to use /etc/dumpdates - flag in configuration file for mt variable block size - do buffering on archive read if enabled - add "-extract -files " to extract a list of files from an archive (will need this later for interactive restore shell) - minimal Makefile that lets you set config path, bin path, etc. for install - can use "compress" now in addition to gzip/bzip2 - added zip type - var in configuration file for afio compression cache size - flag in configuration file for tar atime preserve - flag in configuration file for tar record number printing - flag in configuration file for sparse file handling - add -compress flag to override config file setting Bug fixes/cleanup: - afio nocompress types bug - have to list them all if we give the -E flag.. (I added mp3 and it ignored the builtin defaults) made generic \$nocompress_types since zip uses it too - find -regex doesn't exist on BSD, redo with egrep to be more generic - GPL clause in program itself not readme - redid remote commands to use only 1-way rsh/ssh archive command building done differently now as a result - redid command echo wrapping - dates back to string format (easier to read), use numeric only on BSD will work with pre-4.0 fileutils that have touch -t bug - tar exclude mess is fixed, using --files-from again as well. problem before was I was not stripping directory names - make better afio control file handling for volume labels - use -B (--read-full-blocks/--read-full-records) for tar - use $verbose to set restore -v for list/extract - log files and archives to disk use same filename pattern now - do obsolete timestamp removal at end of job not beginning - forgot to remove tar_verbose and cpio_verbose from conf file - debug shows real mt command - clean up flag handling some more, use long flags - sleep at end of backup not done if file or debug - if -noerase, move to eod before starting ---------------------------------------------------------------------- Version 0.9.5 (19990927) New features: - buffering for all backup types now requires the "buffer" utility, everything piped through it if remote backup, buffers on both sides - every backup type now has an embedded "title" (level,date,host,etc) dump has had this info. by default for afio we make a tiny control script for tar we use -V for a volume label for cpio we touch a filename wih the info before backing up - config file regular expression for excluding files from backup doesn't work for dump, of course - config option for afio file size compression threshold - config option for afio nocompress file extension list - compression for all backup types - if reading from a file, auto-set compression & archive type whee we can also list & extract rpm files :) Bug fixes/cleanup: - nuke trailing slashes on given dirs - typo! mt defblksize, not mt defsetblk. also, that whole routine was just broken. fixed. - tar was archiving things twice. "find . | tar -c -T -" gives double result. revert back to non-find mode for tar, works fine - remote tar didn't quite work (needed full path for ssh/rsh) - remote broken if using filenames (and they were long) - error if remote host and this host are the same - error if device is dir and its relative - stderr output from new pipeline commands wasn't in log file - fix dirs going on top of each other when mult. dumps. + files - log file removal at end of job; undef logfile after compressing - compress, compr_level, verbose, buffer are generic options - commands are an array; command build cleanup; mt takes multiple commands --------------------------------------------------------------- Version 0.9.4 (19990924) New features: - bzip2 option for afio and tar archives, as well as log files - configurable compression level for afio - multiple tapes for an "all" level 0. This just lets you split up backups onto different tapes, not end-of-media type of multivolume problems (that's on the todo list) - support for remote dump over ssh - use dd over an ssh pipe - support ssh for cpio remote backups - use dd over an ssh pipe - add -reten switch to retension before backup or any other operation - print "date of this level" sort of like dump does Bug fixes/cleanup: - test if mt setblksize or mt defblksize is the right command to use - dump -a (autosize) works now. Found my problem was a known bug. >>> NOTE for Linux w/dump, you MUST install dump-0.4b4-10 (redhat lorax) or newer, or equivalent patches if you want dump -a to work - afio doublebuffer only if local backup - dump length bug for "-fs all" - get last timestamp was off-by-one - output of archive commands didn't go to log file - with label "host:path-dir"; didn't work w/ remote + files labels are now "host-path-dir" - sanity checks in the setup, flag variables - check for programs in your path --------------------------------------------------------------- Version 0.9.3 (for testing only) New features: - can now archive to files instead of devices just set $device to a directory for backup compare/extract/list/restore can now take a filename argument - add -n option that doesn't run mt or the actual backup - timestamp changes: use "touch -t" not "touch -d" so FreeBSD works all dates now in YYYYMMDDhhmm.SS format use timestamps of the epoch for level 0 (simplifies code) - add "-d var=val" for overriding config values - tar mode uses "find" just like afio & cpio - afio doublebuffering option Bug fixes/cleanup: - FreeBSD: if $dump_length set to 0 use dump -a - FreeBSD: fix for mt setblk - error if level > 0 and no level 0 stamp exists - fix label string for remote paths --------------------------------------------------------------- Version 0.9.2 (19990920) - initial release to the public - gzip log files option - fix afio buffer to be in units of k in cfg file Version 0.9.1 (1999019) - incrementals w/ afio cpio tar done (remote fs too!) - timestamp only after sucessful backup, but use the pre-backup time - noerase option Version 0.8 - redone with "use strict" - incremental for non-dump types almost done - automated pacakging - clean some duplicated code - rpm build, lsm file - previously just a local hack I had done for myself Local Variables: mode: flyspell end: