6

I tried to tag late onto a question similar to mine on stackoverflow (Find Non-UTF8 Filenames on Linux File System) to elicit further replies, with no luck so far, so here goes again...

I have the same problem as the OP in the link above and convmv is a great tool to fix one's own filesystem. My question is therefore academic, but I find it unsatisfactory (in fact I can't believe) that 'find' is not able to find non standard ascii characters.

Is there anyone out there that would know what combination of options to use to find filenames that contain non standard characters on what seems to be a unicode FS, in my case the characters seem to be 8bits extended ascii rather than unicode, the files come from a Windows machine (iso-8859-1) and I regularly need to fetch them. I'd love to see how find and/or grep can do the same as convmv.

Sample files:

> ls
Abc�def ÉÈéèáà-rest everest éverest

> ls -b
Abc\251def  ÉÈéèáà-rest  everest  éverest

First file comes from Windows (or simulated with touch $(printf "Abc\xA9def")).

> find . -regex '.*[^a-zA-Z./].*'
./ÉÈéèáà-rest

> ls | egrep '[^a-zA-Z]'
ÉÈéèáà-rest

Missing almost all of them (the hyphen saved that file, can be seen with coloured grep). Whatever is happening here is not what I would expect: neither find nor grep is able to take an accented letter as being outside the range provided [^a-zA-Z./].

> find . -regex '.*é.*'
./éverest
./ÉÈéèáà-rest

> ls | egrep 'é'
ÉÈéèáà-rest
éverest

> ls | egrep '[é]'
ÉÈéèáà-rest
éverest

> find . -regex '.*[é].*'
./éverest
./ÉÈéèáà-rest

Bizarrely both are able to pick up a standard accent when provided (including in the range). Any find or grep trial with \xA9, \0251 or \o251 fails (no match).

> ls | fgrep e
Abc�def
ÉÈéèáà-rest
everest
éverest

Looking for a non-controversial character shows all files with grep, as I would have expected.

> find . -regex '.*e.*'
./éverest
./ÉÈéèáà-rest
./everest

> find . -name '*e*'
./éverest
./ÉÈéèáà-rest
./everest

find, however, is very discriminatory: even looking up a normal character, it seems to me that it eliminates filenames that contain characters outside the range of acceptable characters for the filesystem's name encoding schema.

As far as I am concerned if the file is in the filesystem, then find should find it, right? But maybe there's a feature I don't know about?

Any insights would be very much appreciated.

3 Answers 3

9

The GNU tools appear to have code that causes accented letters to be treated like their base letters when matching a regex character class, if supported by the character encoding. This is intended as a "do what I mean" sort of feature to make writing regexes easier, but in this case it's getting in your way.

Try the following modification to your "find" command line:

LANG=C find . -regex '.*[^a-zA-Z./].*'

This sets the LANG environment variable only in the context of the "find" command. Since the "C" language encoding supports only ASCII, the accented letters will no longer be treated as their base letters, and so will be matched properly by your regex.

1
  • 3
    It would be more reliable to run LC_CTYPE=C find … or LC_ALL=C find …. The precedence of environment variable is LC_ALL first, if absent LC_xxx for the desired locale category (here ctype, i.e. character set and encoding), if also absent LANG. So if already you have LC_CTYPE in the environment (common in the 95% of the world whose language can't be written in ASCII), LANG= won't change a thing. Dec 6, 2010 at 20:51
5

Jander's answer does the job perfectly, for those interested in getting more out of this, here is one more tip.

With LANG=C, find displays non-ascii characters with question marks. To convert that back to their normal display with that file system, just pipe the output to cat.

LANG=C find . -regex '.*[^a-zA-Z./-].*'
./??verest
./????????????-rest
./Abc?def

LANG=C find . -regex '.*[^a-zA-Z./-].*' | cat
./éverest
./ÉÈéèáà-rest
./Abc�def
1
  • 1
    What's actually going on is that find converts unprintable characters to ?, but only if it's printing to a terminal. With find … | cat, it's not printing to a terminal so it's printing the bytes as they come. With LANG=C, every non-ASCII character is unprintable. Dec 6, 2010 at 20:47
0
find . | grep -E '.*[^[:print:]].*'

For a list of all the posix character classes, see: http://www.regular-expressions.info/posixbrackets.html

1
  • This looks like an elegant solution however it does not seem to work for me.
    – asoundmove
    Nov 16, 2015 at 22:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .