31

The Joomla .ini files require to be saved as UTF-8.

After editing I'm not sure if the files are UTF-8 or not.

Is there a Linux command like file or a few commands that would tell if a file is indeed UTF-8 or not?

2
  • 5
    You cannot tell the encoding of a file. You can only make a smart guess. You might mostly guess right, but sometimes guesses fail. file is an example of a program doing smart guesses.
    – Marco
    Commented Sep 24, 2013 at 21:17
  • 2
    @Marco: It is possible to verify whether it is valid UTF-8 or not, however. There are some encodings which can mistakenly pass as valid UTF-8, but it almost never happens with ISO-8859-𝒏 or Windows-125𝒏 encodings/charsets. Commented Sep 24, 2013 at 21:40

5 Answers 5

30

You can determine the file encoding with the following command:

file -bi filename
14
  • 4
    @nicolas For MacOS you could try file -I filename (-I is a capital i).
    – Rik
    Commented Apr 24, 2016 at 16:07
  • 6
    Does this read the whole file? Commented Mar 30, 2018 at 15:17
  • 2
    @kojow7 utf-8 has no header. Pure ASCII (7-bit only), is indistinguishable from utf-8 (that is the point of it, a header will cause all sorts of problems). So if you have a file that is ASCII for the first MB then has a single UTF-8 character, then you will not know, unless you read the whole file. Commented Apr 21, 2018 at 16:41
  • 8
    This should not be accepted as the answer. The 'file' command does not do that ; it reads only part of the file and uses magic numbers to take a best guess. On occasion 'file' can and will give you the incorrect answer. To verify if a file passes an encoding such as ascii, iso-8859-1, utf-8 or whatever then a good solution is to use the 'iconv' command.
    – Tim
    Commented Mar 9, 2020 at 14:45
  • 2
    I have tested it, and it can and does fail.
    – Tim
    Commented Mar 10, 2020 at 17:18
23

There is, use the isutf8 command from the moreutils package.

Source: How can you tell if a file is UTF-8 encoded or not?


4
  • @davidpostill I'm curious, is bad practice to cite the author in the reference? Commented Aug 28, 2016 at 20:26
  • 1
    No. However, it is good practice to make the link say where it leads me. Assume I'm reading only the blue text. After the edit, I can tell why and when I should click that. Before, I could not. (It wasn't me who made the edit but I'm like 94% sure that this is what it was about.) Commented Dec 31, 2018 at 0:00
  • Nice, and works nicely with find -type f -exec isutf8 {} +, because it also quotes the filename. (And with using find ... -exec ... + is also fast) Commented Mar 22, 2019 at 13:28
  • ⁺¹, it's better than the answer with file -bi as it tells exactly what is wrong with file and where, up to a line number
    – Hi-Angel
    Commented Jan 16 at 8:37
18

Do not use the file command. It does not inspect the whole file, and it basically guesses. It sometimes gives incorrect answers.

You can verify if a file happens to pass UTF-8 encoding like this:

$ iconv -f utf8 <filename> -t utf8 -o /dev/null

A return code of zero means it passes UTF8. A non-zero return code means it is not valid UTF8.

It is not possible to know if a file was necessarily exported using any particular encoding scheme, as some encoding schemes overlap. To do that would require metadata to be embedded in the file, and even then you would be placing trust in whoever generated that file, rather than validating it yourself... and you should always validate it yourself.

2
  • 1
    I've used this successfully for a while, but I'd recommend reordering the arguments so the filename comes last. This way, it will also work on busybox / alpine linux: iconv -f utf8 -t utf8 <filename>
    – phil294
    Commented Oct 3, 2023 at 19:24
  • iconv prints a warning message instead of returning a non-zero code
    – yegor256
    Commented Feb 26 at 8:52
1

Yet another way is to use recode, which will exit with an error if it tries to decode UTF-8 and encounters invalid characters.

if recode utf8/..UCS < "$FILE" >/dev/null 2>&1; then
    echo "Valid utf8 : $FILE"
else
    echo "NOT valid utf8: $FILE"
fi
0

You can use the TextIntegrityInspector package to validate the integrity of your Joomla .ini files as UTF-8. Here's how you can do it:

Install TextIntegrityInspector:

pip install textIntegrityInspector

Use TextIntegrityInspector to validate files in the command line:

textIntegrityInspector path_to_your_ini_files --extensions ini

This will check all .ini files in the specified directory to ensure they are valid UTF-8.

Alternatively, use TextIntegrityInspector in a docker env:

docker run -it -v path_to_your_ini_files :/data text_integrity_inspector --extensions ini

You can add --language="fr" option to indicate only French characters are accepted ,

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .