Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have about 700 files that were the results of a closed caption software process. I can open the files on my PC or Mac in a GUI, but when I open it on the server with nano, I get this:

��0^@0^@;^@0^@0^@;^@0^@0^@;^@0^@0^@ ^@0^@0^@;^@0^@0^@;^@0^@6^@;^@0^@0^@
^@
^@N^@o^@t^@ ^@A^@v^@a^@i^@l^@a^@b^@l^@e^@

Here are the results from a few of the command line things I have tried:

#file infile.txt 
infile.txt: data

# file -bi --mime-encoding  infile.txt 
application/octet-stream; charset=binary

recode ../CR-LF infile.txt
#file -bi --mime-encoding  infile.txt

#iconv -f binary -t uff-8 infile.txt > out.txt
iconv: conversions from `binary' and to `uff-8' are not supported

#recode ../CR-LF infile.txt
#recode UTF8..ISO-8859-15 infile.txt 
# file -bi --mime-encoding  infile.txt 
text/plain; charset=utf-16le

#xxd infile.txt | head -5
0000000: fffe 3000 3000 3b00 3000 3000 3b00 3000  ..0.0.;.0.0.;.0.
0000010: 3000 3b00 3000 3000 2000 3000 3000 3b00  0.;.0.0. .0.0.;.
0000020: 3000 3000 3b00 3000 3600 3b00 3000 3000  0.0.;.0.6.;.0.0.
0000030: 0d00 0d0a 004e 006f 0074 0020 0041 0076  .....N.o.t. .A.v
0000040: 0061 0069 006c 0061 0062 006c 0065 00    .a.i.l.a.b.l.e.

The actual file looks like this (it turns out it wasn't the best choice, I accidentally chose one without much of anything in it):

00;00;00;00 00;00;06;00 Not Available

Then if I nano the file, I get text at the bottom that says it was converted from DOS and Mac format, but it is still all garbage. I need to be able to loop through the files and import the data into a database, but of course I can't do that if the server isn't able to read the format.

I would like to find a long term solution since I will be generating lots of these in the future, and would rather not use Windows powershell or equivalent but incorporate the conversion into my process.

share|improve this question
    
What GUI do you open them in? A text editor like Notepad or TextEdit, or some special application? –  D Schlachter Jul 18 '14 at 16:17
    
On Windows I used Notepad or Editplus, on my mac I used TextEdit –  Alan Jul 18 '14 at 16:20
2  
You were likely going the right direction with the iconv, but you passed invalid parameters. It might be useful if you included a few lines of the original file. Update your question with the output of something like xxd original_file | head -5. BTW, I suspect you have an invalid or missing BOM. –  Zoredache Jul 18 '14 at 16:22
    
Thanks for the suggestions @Zoredache, I updated the question and will look into the BOM possibility. –  Alan Jul 18 '14 at 16:43
    
So your file does have a BOM [see the FFFE at 0x0). It tells me your current file seems to be a little endian UTF-16. Are you sure you want to downgrade change the format? Why are you using nano to view these files? Surely that isn't part of your batch process. I don't think nano actually supports unicode. –  Zoredache Jul 18 '14 at 17:00

1 Answer 1

up vote 0 down vote accepted

So the solution for me ended up being to use recode. There isn't a lot of great documentation, but the trick was to use the -f option to force it to change it.

recode -f UTF-7 *.txt
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.