Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a couple of C++ source files (one .cpp and one .h) that are being reported as type data by the file command in Linux. When I run the file -bi command against these files, I'm given this output (same output for each file):

application/octet-stream; charset=binary

Each file is clearly plain-text (I can view them in vi). What's causing file to misreport the type of these files? Could it be some sort of Unicode thing? Both of these files were created in Windows-land (using Visual Studio 2005), but they're being compiled in Linux (it's a cross-platform application).

Any ideas would be appreciated.

Update: I don't see any null characters in either file. I found some extended characters in the .cpp file (in a comment block), removed them, but file still reports the same encoding. I've tried forcing the encoding in SlickEdit, but that didn't seem to have an effect. When I open the file in vim, I see a [converted] line as soon as I open the file. Perhaps I can get vim to force the encoding?

share|improve this question
    
Any null characters in them? –  Mehrdad Apr 11 '12 at 15:33
    
What's an easy way to test for nulls within the file? To my knowledge, they don't have any, but that's not to say that one snuck in somewhere... –  Jonah Bishop Apr 11 '12 at 15:41
    
Hmm... you could open them up in SciTE and check for a NUL box (or search for \0 with backslash transformations). –  Mehrdad Apr 11 '12 at 15:45
    
Try 'cat -v' to make control characters visible. –  Brian Swift Apr 11 '12 at 15:58
    
What happens when you run file -e soft filename? –  Daniel Beck Apr 11 '12 at 16:57
show 6 more comments

6 Answers

Vim tries very hard to make sense of whatever you throw at it without complaining. This makes it a relatively poor tool to use to diagnose file's output.

Vim's "[converted]" notice indicates there was something in the file that vim wouldn't expect to see in the text encoding suggested by your locale settings (LANG etc).

Others have already suggested

  • cat -v
  • xxd

You could try grepping for non-ASCII characters.

  • grep -P '[\x7f-\xff]' filename

The other possibility is non-standard line-endings for the platform (i.e. CRLF or CR) but I'd expect file to cope with that and report "DOS text file" or similar.

share|improve this answer
add comment
up vote 2 down vote accepted

I found the issue using binary search to locate the problematic lines.

head -n {1/2 line count} file.cpp > a.txt
tail -n {1/2 line count} file.cpp > b.txt

Running file against each half, and repeating the process, helped me locate the offending line. I found a Control+P (^P) character embedded in it. Removing it solved the problem. I'll write myself a Perl script to search for these characters (and other extended) in the future.

A big thanks to everyone who provided an answer for all the tips!

share|improve this answer
add comment

It could be that the files have been saved with a BOM at the beginning of them, although I would have thought a recent-ish version of the file binary should recognise that too.

Have you tried dumping them through something like "head -2 | xxd" and seeing if there's a BOM present?

*BOM = Byte Order Mark - sometimes present in unicode text files. http://en.wikipedia.org/wiki/Byte_order_mark

share|improve this answer
    
That an interesting tip. I've run both files through xxd, and I see no BOM in the first character position. Each file starts out with a giant comment block, so I see a bunch of slashes to start. –  Jonah Bishop Apr 11 '12 at 15:36
    
Care to share an excerpt? –  GodEater Apr 11 '12 at 15:37
    
This is what I see from the xxd dump: 0000000: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f //////////////// –  Jonah Bishop Apr 11 '12 at 15:39
add comment

It probably is a non-ASCII character from Unicode or some other character set. Since you're using vi, which in most Linux distributions is some version of vim, you can search for that character by typing

/[<Ctrl-V>x80-<Ctrl-V>xff]

and hitting Enter, where <Ctrl-V> means typing v while pressing the Ctrl key. Similarly, you can search for nulls (as Mehrdad suggested) with this:

/<Ctrl-V>x00
share|improve this answer
    
This search resulted in a comment block containing some extended characters in my .cpp file. However, I don't see any similar characters in the .h... –  Jonah Bishop Apr 11 '12 at 15:43
    
I updated my answer to include searching for nulls as Mehrdad suggested. –  garyjohn Apr 11 '12 at 16:00
    
I don't see any null characters in either file. :( –  Jonah Bishop Apr 11 '12 at 16:09
add comment

If you run file -D filename, file displays debugging information, including the tests it performs. Near the end, it will show what test was successful in determining the file type.

For a regular text file, it looks like this:

[31> 0 regex,=^package[ \t]+[0-9A-Za-z_:]+ *;,""]
1 == 0 = 0
ascmagic 1
filename.txt: ISO-8859 text, with CRLF line terminators

This will tell you what it found to determine it's that mime type.

share|improve this answer
    
I don't see a -D option in my file install (v5.04)... –  Jonah Bishop Apr 11 '12 at 16:49
    
Try -d instead. That works with file-5.03 as installed on Fedora 11. –  garyjohn Apr 11 '12 at 17:06
    
Notifying @JonahBishop about garyjohn's comment. My post was written for the file included with OS X. My Debian 6 has neither -d nor -D though... –  Daniel Beck Apr 11 '12 at 17:09
    
The -d flag works for me, but there's so much output I'm not sure what to look for... –  Jonah Bishop Apr 11 '12 at 22:32
add comment

Which charset/encoding/(codepage) are the files in?
Perhaps the files have stray character(s). typically from bad cross-encoding between different platforms. Invalid data in you files may be causing file to report as you have described. You can test the validity of a file for a particular charset encoding by testing it with recode (or iconv).

Follow the link for a list of Common character encodings

This script lists charset encodings (from $my_csets) which aren't valid for your file(s). You can list all charsets via: recode -l

file="$1"    
my_csets="UTF-16 UTF-8 windows-1250 ASCII"

# Use the next lines to test all charsets
# =======================================
# all_csets=$(recode -l |sed -ne "/^[^:/]/p" | awk '{print $1}')
# my_csets=$all_csets

for cset in $my_csets ;do 
  <"$1" recode $cset.. &>/dev/null || echo  "$cset  ERROR: $?"
done 
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.