0

I've encountered a weird issue with rsync (without -c, therefore involving rsync's "quick check" algorithm) copying already synchronized files over and over again to an exFAT formatted drive.

The involved source files are stored on an HFS volume and were created by an automated image extraction process from .docx word documents and have a modification time of Jan 1, 1980 00:00:00 which also happens to be the minimum time supported by the exFAT timestamp format.

After a rsync -a those destination files on exFAT have a modification time of Jan 1, 1980 02:00:00.

It looks like the minimum time supported by my OS X exFAT driver is off by two hours.

And this is the reason that causes rsync's quick check algorithm to believe the files are different.

The following test seems to confirm this:

# create a 10Mb exFAT disk image:
hdiutil create -size 10m -fs ExFAT -volname EXFATTEST exfattest.dmg

# attach the diskimage
hdiutil attach exfattest.dmg

# create a test file with a creation time of Jan 1, 1980 00:00:00
touch -t "198001010000.00" /Volumes/EXFATTEST/test1.txt
ls -lT /Volumes/EXFATTEST/test1.txt 
-rwxrwxrwx  1 user  staff  0 Jan  1 02:00:00 1980 /Volumes/EXFATTEST/test1.txt
#                                   ^^ off by two hours

# try a file with 3am
touch -t "198001010300.00" /Volumes/EXFATTEST/test2.txt
ls -lT /Volumes/EXFATTEST/test2.txt
-rwxrwxrwx  1 user  staff  0 Jan  1 03:00:00 1980 /Volumes/EXFATTEST/test2.txt
#                                   ^^ works as expected


# unmount the disk image
hdiutil detach /Volumes/EXFATTEST/

An equivalent test on Linux does not show this error:

truncate -s 10M exfattest.img
mkfs.exfat exfattest.img
mkdir /mnt/EXFATTEST/
mount -o loop exfattest.img /mnt/EXFATTEST/
touch -t "198001010000.00" /mnt/EXFATTEST/test.txt
ls --full-time /mnt/EXFATTEST/test.txt
rwxr-xr-x 1 root root 0 1980-01-01 00:00:00.000000000 +0100 /mnt/EXFATTEST/test.txt
#                                  ^^ correct modification time
umount /mnt/EXFATTEST/

My simple solution is to update the source files' creation time using touch.

However, I'd like to know if this is really a bug and if someone can reproduce this error on newer OS X releases since Apple doesn't allow my still perfectly fine running early 2008 iMac to be be upgraded ;)

I'm still on El Capitan:

sw_vers
ProductName:    Mac OS X
ProductVersion: 10.11.6
BuildVersion:   15G22010

Update:

After inspecting the exFAT table it seems that my El Capitan exFAT driver always uses 0xfc for the LastModifiedTimezoneOffset data field.

This signed byte value specifies the offset from UTC in 15 min increments. 0xfc is -4 in decimal and would be interpreted as a negative UTC offset of one hour.

I don't know why the driver is doing this but if it uses a static timezone offset of -1 it would also always have to subtract 1 hour from the corresponding UTC time stamp when storing the timestamp and adding 1 hour to the stored value when reading the timestamp back from disk.

This basically works, unless the UTC time stamp is younger than Jan 1, 1980 01:00, since the minimum supported time stamp of a file on exFAT is Jan 1, 1980 00:00.

So I consider this a bug, since El Capitan's exFAT driver is not able to set the minimum possible file modification time stamp supported on exFAT.

The following test seems to confirm this. It uses the TZ environment variable to force the usage of the UTC time zone independently of the system's time zone:

# set the minimum possible exFAT file modifification timestamp in UTC
TZ=UTC touch -t '198001010000.00' /Volumes/EXFATTEST/test.txt
# read back the stored timestamp in UTC
TZ=UTC ls -lT /Volumes/EXFATTEST/test.txt
-rwxrwxrwx  1 gollum  staff  0 Jan  1 01:00:00 1980 /Volumes/EXFATTEST/test.txt
#                                     ^^^^^^^^ ERROR: off by one hour

Same error when creating a file with this date on HFS and copying it to the exFAT volume:

# set Jan 1, 1980 00:00:00 on HFS (e.g. file in home directory)
TZ=UTC touch -t '198001010000.00' $HOME/test2.txt
TZ=UTC ls -lT $HOME/test2.txt
-rw-r--r--  1 gollum  staff  0 Jan  1 00:00:00 1980 /Users/gollum/test2.txt
#                                     ^^^^^^^^ CORRECT!

# copy the file to the exFAT volume and check the destination date
cp -a $HOME/test2.txt /Volumes/EXFATTEST/test2.txt
TZ=UTC ls -lT /Volumes/EXFATTEST/test2.txt
-rwxrwxrwx  1 gollum  staff  0 Jan  1 01:00:00 1980 /Volumes/EXFATTEST/test2.txt
#                                     ^^^^^^^^ ERROR: off by one hour

Another interesting test I've done is taking the exFAT disk image created on linux (which contains the test file with the correct time stamp) and mounting it on OS X:

TZ=UTC ls -lT
total 0
-rwxrwxrwx  1 gollum  staff  0 Jan  1 00:00:00 1980 test.txt
#                                     ^^^^^^^^ correct!

However, when duplicating this file the date is off by one hour again. So at least the OS X exFAT driver is able to read the time stamps correctly, even if the files are stored using a non 0xfctimezone offset. Yay!

Now the other way around: Mounting the OS X dmg on Linux reveals a second error of the OS X exFAT driver:

mount -o loop exfattest.dmg /mnt/EXFATTEST/
FUSE exfat 1.0.1
ERROR: bad date 1980-01-00
#                       ^^^ OS X stored the DAY as zero ::facepalm::

Another perfect example of missing corner case tests in development.

This is basically the answer to my question. However, I will accept an answer that proves my findings wrong or confirms them and which also answers my question if this bug also happens on newer than El Capitan OS X releases.

Update 2:

I've come across a blog post by Adam Harrison (a digital forensics investigator from UK): exFAT Timestamp Behavior Associated with Different Operating Systems

Quoting:

Ubuntu 16.04 and Ubuntu 18.04
All timezones are recorded in UTC. The Timezone fields are consistently set to 00, indicating that they are not in use.

OSX 10.13.3
Timezone Fields – Consistently set to “FC” which is UTC-1 and I have no idea why...

So this seems to match my findings.

1

exFAT implementations can optionally store UTC timestamps, instead of local timestamps. (Most modern filesystems use UTC timestamps exclusively, but exFAT likely wanted to retain some compatibility with original FAT.)

The macOS exFAT driver probably was written to use the UTC timestamp mode – this means that when you're specifying 1980-01-01 00:00.00 (local) in touch, you're actually asking the driver to store 1979-12-31 23:00.00 (UTC) on disk.

The Linux exfat-utils package meanwhile uses the local time and stores timestamps with the system timezone offset re-applied, so local 00:00 is also stored as 00:00.

  • nope, look at the 3am example – gollum Jan 3 at 12:59
  • Example looks fine. You gave touch 03:00 local time, it was translated to something like 01:00 or 02:00 UTC when storing on disk. Then it was read back from disk by ls and translated back from UTC into 03:00 local time before displaying it. – grawity Jan 3 at 13:50
  • 1
    It really is timezone related but your assumption is wrong. I've analyzed the FAT table in my hex editor and no matter which timezone the system uses, my OSX always writes 0xFC for all three (create, modify, access) TimezoneOffset. These values are stored as signed bytes (0xFC is -4 in decimal) and interpreted as offset from UTC in 15 min increments. In other words El Capitan always uses a time zone offset of UTC-01:00 when writing exFAT time stamps. – gollum Jan 3 at 16:38
  • And also nope regarding your assumption about linux exfat using local time. I've looked at the images's hex dump and it uses 0x00 for the timzone offsets. That means the linux exfat fuse driver stores the timestamps in UTC. However, +1 for making me check if the bug is time zone related, which seems to be the case. – gollum Jan 4 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.