Odd Date Comparison Error
At work I wrote a shell script to sync certain files from a company repository. The repository is read-only.
The script has been functioning as expected for a couple of years. Until I started testing Debian 10 in the infrastructure.
The script uses bash and uses the
-ot file comparison operators. For some reason, on Debian 10 systems the comparisons seemed to be failing. On Debian 10 systems the script insisted the local file was newer than the repository file.
I knew otherwise. The files were exact copies.
Or so I thought.
Adding some temporary code showed the date stamps were the same. There is some nuance in the Debian 10 systems that cause the file comparison operator anomaly.
I found no answers after poking around the web. I found references to POSIX requirements but nothing explicit why the problem might appear only in Debian.
I compared file date stamps using
ls -la --time-style=full-iso. The files were indeed different based on the nanosecond portion of the time stamp.
I added that comparison to the shell script. I again experienced messages about time differences.
Eventually I discovered the
scp -p command, which preserves time stamps, does not support nanoseconds. The target file nanoseconds always get zeroed. For example:
Source files: file_a: 2020-01-29 16:16:15.404379767 file_b: 2020-01-28 16:57:03.838916716 Target files (after sending through scp -p): file_a: 2020-01-29 16:16:15.000000000 file_b: 2020-01-28 16:57:03.000000000
scp -p wipes the nanoseconds even if the destination time stamp already includes nanoseconds. If the destination file has nanoseconds in the time stamp, using
scp -p will overwrite the file and lose the nanoseconds.
If the file does not exist on the destination/target system,
rsync -t|a will correctly transfer the time stamp including nanoseconds.
For comparing differences, seems
rsync -t|a only uses the base modification time and size. After copying a file with
scp -p, subsequent copies with
rsync -t|a will not update the time stamp because the base modification time and size remain the same.
rsync -I will not preserve the time stamp, but an immediate subsequent
rsync -t|a will restore the time stamp including nanoseconds.
Using the rsync
--checksum option succeeds in correcting the full time.
rsync -ac restores the nanoseconds even if the file and base time stamp are unchanged.
My final solution was to use
--time-style="+%Y-%m-%d %H:%M:%S" for the comparison.
Interesting I never ran into this quirk after almost 20 years of using Linux based systems.
Yet Another Shoulder Shrug moment.