Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

This is somewhat related to question

http://superuser.com/questions/157207/on-windows-7-dir-or-tree-cant-show-unicode-characters-even-starting-cmd-with-c

Even on Windows 7, I found that the only way I can get unicode to go into a file is by

> cmd /U
> dir /B > files.txt

the file will be in "Unicode" when I open in Notepad and try "Save As", and if I dir /B > files.html and open the HTML file in firefox, it can show using Encoding of UTF-16 (or UTF-16 LE).

but, if I want to see it on the screen instead of having it go to a file, it is still impossible. Is there a way to make it happen? Possibly somehow telling cmd not to show nonprintable characters as "?"

Update: I tried cmd.exe, cygwin's bash on windows, and PowerShell. They are the same. Except if I change the "Properties -> Font" to Consolas or Lucida Console, there is some improvement -- now it is not question mark but is either square border or square with a question mark in it.

The more expensive Mac computers with Mac OS X can do it. The free Ubuntu can do it too.

share|improve this question
add comment

4 Answers

Based on your username I suspect you mainly work with asian languages.

Windows tools operate normally in unicode mode (as you saw by piping the output of dir into a file and opening that file with an editor):

  1. the tool does its stuff
  2. it outputs unicode characters
  3. another program takes this output and has to display it.

to display any character on the screen the program from step 3 has to lookup the glyph appropriate for the given byte sequence. example:

  • 0x65 'a' maps to a different glyph in each font (so the 'a' looks different from font to font)

  • 0x937 'Ω' (greek 'omega') maps to a different glyph in each font as well

this mapping only works IF the font has a glyph for the given byte sequence. otherwise the visual result differs, sometimes you see '?', sometimes diamonds etc.

again: dirproduces bytesequences, which sometimes are purely in the ASCII-range, sometimes they are in the unicode range (depending on what filenames it finds). it sends these sequences to another program which is responsible for actually rendering the bytesequences. to be able to display these sequences, this program has to map the sequence to a glyph. to do that, it has to search in a font for the glyph. if the font does not have a glyph for the given sequence, then the program can not display the byte sequence produced by, for example, dir.

so, the solution to your problem (seeing any unicode-character in the 'console / terminal' of windows) is: use a font for the program which has (almost) every glyph for (almost) any given unicode bytesequence in it.

share|improve this answer
1  
hm, but the cmd, cygwin bash, and PowerShell all are limited to 3 fonts: Raster fonts, Lucida Console, and Consolas... actually Windows usually fall back to a unicode font when it can't display anything with the current font... also, if I redirect the output, like dir > file.txt it is still question mark in the file, even though it is "square box" on the screen. –  動靜能量 Jun 27 '10 at 6:48
    
@Jian Lin: yes, but that is essentially YOUR problem to provide a font which contains these glyphs. and even if windows falls back to "some" font which holds "some" unicode glyphs in it ... that is not enough to display some of your asian glyphs (you have problems with the asian glyphs, right?). –  akira Jun 27 '10 at 6:53
    
according to some websites, "Ascender Uni Duo" seems to be the best font (even for "fixed") ascendercorp.de/fonts/multilingual/ascender-uni but maybe you find something better / cheaper en.wikipedia.org/wiki/Unicode_typefaces –  akira Jun 27 '10 at 6:58
    
@akira there are many fonts on Windows 7 that can display the whole Unicode glyph set. But (1) Cmd window won't let you choose any of them. (2) When windows or the app falls back to the font that can display unicode, such as Lucida Sans Unicode, it can display most any chinese characters. –  動靜能量 Jun 27 '10 at 7:34
1  
@akira: Good answer, I'd just replace ”byte sequences” by “16-bit strings” or “UTF-16 strings” since that is what Windows internally uses. –  Philipp Jun 28 '10 at 9:00
show 9 more comments

http://stackoverflow.com/questions/388490/unicode-characters-in-windows-command-line-how

Use chcp 65001 to change the codepage to UTF8 and use Lucida Console.

share|improve this answer
    
hm... still won't work... cmd /U, chcp 65001, dir, and dir /B with the font already set to Lucida Console, still the same. –  動靜能量 Jun 27 '10 at 8:11
1  
You may want to try adding more fonts to the console: support.microsoft.com/kb/247815 and blogs.msdn.com/b/oldnewthing/archive/2007/05/16/2659903.aspx (the latter for some discussion on the issue). –  ta.speot.is Jun 27 '10 at 8:19
    
it all depends on the fonts you are giving the program to render the text. read the support article, good info in it. –  akira Jun 27 '10 at 9:20
    
@taspeotis: The Windows console always uses Unicode internally, regardless of the codepage setting (which is obsolete anyway and only included for backwards compatibility). It is really just a font problem. –  Philipp Jun 28 '10 at 8:59
    
Can any of the included font on Win 7 be used? such as MingLiU, DFKai-SB –  動靜能量 Jun 28 '10 at 9:39
show 3 more comments

It has nothing to do with encodings since the Windows console always uses Unicode internally. The characters are simply not available in the fonts you use, which are designed for programming and European languages. I don't have access to Windows at the moment, but I remeber that I could print Greek characters after switching to the Lucida Console font. Using a font like DejaVu Sans Mono might work.

share|improve this answer
    
i ve created a russian filename and cmd.exe displayed the glyphs correctly after switching to lucida. for asian fonts i think OP has to pick a "better" or more "unicode complete fixed font" (even if he does not like that answer :)). –  akira Jun 28 '10 at 9:16
add comment

Ok, this is a solution using PowerShell:

1) Click the Start button on Windows 7
2) Now, in the blank line, type in PowerShell
3) Choose PowerShell ISE <-- note it is ISE

Now, if you do ls, you will be able to see unicode characters...

4) if you also use chcp 65001, then if your program prints out UTF-8 characters, they will be nicely displayed as well.

You can also ls > list.txt and then type list.txt and the content shows up in Unicode characters as well.

tree will still not show unicode characters, but what the hell.

also, inside the PowerShell ISE, cmd /U /C dir /B will not work either.

ls -R will... so I am eternally grateful.

share|improve this answer
    
ls in Powershell is actually just an alias for "Get-ChildItem" –  KdgDev Jun 28 '10 at 12:07
    
and then? don't tell me you use Get-ChildItem on the command line every day instead of ls. For example, we usually drink water instead of hydrogen dioxide. –  動靜能量 Aug 29 '10 at 21:12
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.