Linux copy to fat32 filesystem: invalid argument

Question

When I copy files from an ext3 partition to a fat32 one using cp:

cp -R /ext3/stuff /fat32/partition/

I get invalid argument messages for all files with colons and question marks in.

Is there any way to get cp to strip out the invalid characters for the target filesystem?

edit: I've checked through cp's options again, and unless I'm being stupid, there's nothing in there. I'm sure I could write a script, but it feels like there should be a cleaner solution!

Answer 1

The usual suspects when you want complex copies or renames are GNU cp, zmv from zsh, rsync and pax (or cpio). There's no rename feature in cp, nor (I think) in rsync. While zmv can rename, this doesn't mesh well with recursive copies. But pax can do it:

cd /ext3
pax -rw -s '/[*?:]/_/gp' stuff /fat32/partition

This changes each *?: to _. Warning: minimally tested. If there are collisions, whichever file is copied last wins.

Answer 2

Based on post by Gilles I tested the following list:

#!/bin/sh
touch questionmark?
touch less<
touch less\<
touch more\>
touch backslash\\
touch colon:
touch asterisk\*
touch pipe\|
touch inch\"
touch carret\^
touch comma,
touch semicolon\;
touch plus+
touch equals=
touch lbracket[
touch rbracket]
touch quote\'

I tried to copy that onto Android phone MicroSDHC card with vfat filesystem and refined pax command until everything worked. That may still not be enough for Windows and Unicode:

pax -rw -s '/[?<>\\:*|\"]/_/gp' source dest

You may also want to use the -k option to ensure that there are no overwrites (due to collisions in file names). Both lists I gave in the comment were different from Linux vfat behaviour.

Answer 3

I received the "Invalid argument" when copying with cp -r source usbstick and found out the cause was a sourcefilename ending with a space. Removing the space cleared the message. The file with incorrect name happened BTW in this case to be in mailer program directories.

Answer 4

I just had cause to need to do this, and while the pax-based answer was good, it still ran into problems with accented characters.

So I found it simpler to use tar and get it to replace all the non-permitted characters with underscores:

cd /parent-of-source
tar cf - Söurce | (cd /destination; tar xvf - --transform='s/[^A-Za-z0-9\/ ]/_/g')

It is doubtless possible to come up with a better list of permitted characters than the one above, but this works.

Answer 5

After reading the answers to this very interesting question and doing some experimenting with SD card for Android (exfat) and a car entertainment system (vfat), I came up whis this little bash script.

#! /bin/bash

DST=$1
# copy music to FAT media

find music/ Music/ -type f | while read f ; do
    d=$DST/$( echo $f | sed 's/[^-A-Za-z0-9/._ ()]/_/g' )

    echo :$d:
    mkdir -p "$(dirname "$d")"
    cp -n "$f" "$d"

done

It take the destination (mount point) as it argument and use find to locate all files in my music repository.

For each file path, it computes a destination path and file name by prepending the destination and replacing any offending character with an underscore _. I use a white list of characters (letters, digits, -, /, ., . (, ) and _) to remove any unwanted punctuation.

Depending on the locale, this will leave accented letters in the path and file name, which is OK for modern FAT file systems, as it seems.

For each destination file path and name, the directories are created as needed using mkdir -p, then the file is copied, unless it already existed.

Note the quote " chars in various places, they are required to keep paths and names with spaces in them from breaking apart.

Answer 6

I first tried pax, however apparently, it does not correctly resolve name collisions beside it's -k option, see also here. Further, FAT's case insensitivity is a problem:

$ pax -k -rw -s '/[?<>\\:*|\"]/_/gp' foo.txt foo.TXT /media/tycho/exfat/
$ ls /media/tycho/exfat/
foo.txt

Oops. foo.TXT is gone without warning.

Next I tried the fuse filesystem posixovl.

$ mount.posixovl -S /media/tycho/exfat/ mpoint
$ rsync -a foo.txt foo.TXT mpoint/
$ ls mpoint/
foo.TXT

Oops. foo.txt is gone without warning.

So I quickly came up with this script, which solves name conflicts by appending a number before the file suffix:

$ exfat-copy  foo.txt foo.TXT 'bar?.txt' 'bar<.txt' /media/tycho/exfat
NAMECONFLICT_SOLVED: /media/tycho/exfat/foo.TXT >> /media/tycho/exfat/foo1.TXT
REPLACED: bar?.txt >> bar_.txt
REPLACED: bar<.txt >> bar_.txt
NAMECONFLICT_SOLVED: /media/tycho/exfat/bar_.txt >> /media/tycho/exfat/bar_1.txt
$ ls /media/tycho/exfat/
bar_1.txt  bar_.txt  foo1.TXT  foo.txt

Use at your own risk and don't quote me on it.

#!/usr/bin/env python3

import os
import re
import shutil
import sys

from collections import OrderedDict

def eprint(*args, **kwargs):
    print(*args, file=sys.stderr, **kwargs)

# pax -k -rw -s '/[?<>\\:*|\"]/_/gp'
def replace_chars(s):
    # Test with:
    # re.sub(r'[?<>\\:*|\"]', "_", "1?2<3:")
    new_s = re.sub(r'[?<>\\:*|\"]', "_", s)
    if new_s != s:
        print('REPLACED:', s, '>>', new_s)
    return new_s


def find_new_name(filename):
    # already exists
    path, name = os.path.split(filename)
    new_p = "UNSET"
    name, suffix = os.path.splitext(name)
    for i in range(1, 10):
        new_p = '{}/{}{}{}'.format(path, name, i, suffix)
        if not os.path.exists(new_p):
            print('NAMECONFLICT_SOLVED:', filename, '>>', new_p)
            return new_p
    raise ValueError('Failed to find non-conflicting path for {}: last tried {}'.format(filename, new_p))

# based on shutil.copytree
def copytree(src, dst, ignore=None):
    names = os.listdir(src)
    if ignore is not None:
        ignored_names = ignore(src, names)
    else:
        ignored_names = set()

    # conflict due to FAT-based renameing
    if os.path.exists(dst):
        dst = find_new_name(dst)

    os.mkdir(dst)
    errors = []
    for name in names:
        if name in ignored_names:
            continue
        name_cleaned = replace_chars(name)

        srcname = os.path.join(src, name)
        dstname = os.path.join(dst, name_cleaned)
        handle_filenames(srcname, dstname)
    try:
        shutil.copystat(src, dst)
    except OSError as e:
        print('ERROR: shutil.copystat failed on', src, 'and', dst, e)


def handle_filenames(srcname, dstname):
    try:
        if os.path.islink(srcname):
            print('SYMLINK_IGNORED:', srcname)
        elif os.path.isdir(srcname):
            copytree(srcname, dstname, None)
        else:
            if os.path.exists(dstname):
                dstname = find_new_name(dstname)
            # Will raise a SpecialFileError for unsupported file types
            shutil.copy2(srcname, dstname)
    except OSError as e:
        print('ERROR: during handling of', srcname, 'and', dstname, e)

if len(sys.argv) < 3:
    eprint("At least two args required: src dst")
    exit(1)

src_list = sys.argv[1:-1]
dst = sys.argv[-1]


if not os.path.exists(dst):
    eprint(dst, 'must exist')
    exit(1)

if len(os.listdir(dst)) != 0:
    eprint('destination directory must be empty:', dst)
    exit(1)

new_dst = replace_chars(dst)
if new_dst != dst:
    eprint("destination directory must only contain valid chars")
    exit(1)

src_list = list(OrderedDict.fromkeys(src_list))

if dst in src_list:
    eprint('src and dst must be different')
    exit(1)

for src in src_list:
    dst_sub = replace_chars(os.path.basename(src))
    # we allow files and dirs, so cannot start with copytree
    handle_filenames(src, dst+'/'+dst_sub)