Questions tagged [unicode]
Unicode is the standard for computer representation of plain text. It encompasses the Universal Character Set, intended to unambiguously represent all characters used in human writing systems in any language, Unicode Transformation Formats (UTFs), defining standardized formats for storing and transmitting Unicode text, and standards for processing and manipulating text.
711
questions
15
votes
3
answers
20k
views
Displaying Unicode on Chrome vs Firefox
Unicode Rendering: Firefox vs Chrome
OS: Windows XP SP3
My question is about the rendering of this post on Firefox vs Chrome. I can see a lot of boxes on Chrome, not so much on Firefox.
Firefox:
...
15
votes
1
answer
38k
views
Saving a file in a CSV type in Excel always removes the BOM
I've been trying to find a reasonable solution/explanation (unsuccessfully) to find out why Excel defaults to removing the BOM when saving a file to the CSV type.
Please forgive me if you find this a ...
14
votes
3
answers
63k
views
Windows 7 UTF-8 and Unicode
Could someone please explain what has changed in Windows 7 (Pro 64-bit)?
Details:
Previously I had Windows XP and had some translations files (UTF-8 encoded) in CSV format. I was able to view the ...
14
votes
4
answers
14k
views
Find files with non-ASCII characters in filenames in Windows XP
Is there any easy way to find all files in a particular directory that have any non-ASCII (ie. Unicode) characters in the filename? I'm running Windows XP x64 SP2, NTFS filesystem.
14
votes
2
answers
19k
views
iconv generating UTF-16 with BOM
Inspired by this question, can I use the iconv command to generate UTF-16 output with a BOM and with specified endianness?
The iconv command converts text from one encoding to another.
For example:
...
13
votes
5
answers
70k
views
Find Alt-code for any character
Sometimes, an alt-code that I find on the internet does not work on my PC, for example Pi alt code (Alt+227) throws Ń isntead of π.
So I need to use the character in clipboard (or in a file or ...
12
votes
1
answer
6k
views
Ubuntu's ls can show unicode characters, but tree cannot, even with tree --charset unicode
Ubuntu 10.04's ls can show unicode characters, but tree cannot, even with tree --charset unicode or utf8. Is there a way to solve it. echo could interpret the escape characters except cat list.txt | ...
12
votes
6
answers
36k
views
Entering Unicode characters in Google Chrome?
I would like sometimes to enter Unicode characters such as a right arrow (→).
Currently, the only way I know is to use the Unicode Input application, but it's about as cumbersome as the Windows ...
12
votes
7
answers
22k
views
Robocopy unilog output is gibberish
I tried to get robocopy in Windows 7 to generate a Unicode log, since I have files with Unicode characters. The command I used:
robocopy C:\mysource D:\mydest /mir /unilog:backup.log /tee
File the ...
12
votes
1
answer
9k
views
Display unicode in tmux?
This is urxvt without tmux
This is urxvt with tmux and utf8 not enabled in the config
This is urxvt with tmux and utf8 enabled in the config
How can I make tmux act consistent with the initial ...
12
votes
2
answers
5k
views
In Unicode, is these a way to superimpose a character over another?
I am looking for a joining character that works a bit like overset in LaTeX. It also works a bit like the many combining Unicode characters, but I would like it to work for any alphabetic letter. For ...
11
votes
7
answers
11k
views
Why does Unicode have big or little endian but UTF-8 doesn't?
UNICODE uses 2 bytes for one character, so it has big or little endian difference. For example, the character 哈 is 54 C8 in hex.
And its UTF-8 therefore is:
11100101 10010011 10001000
UTF-8 uses 3 ...
11
votes
2
answers
69k
views
How do I find this character(by unicode search) in notepad++ ﻁ (\uFEC1 and only that character)
How do I find this character(by unicode search) in notepad++ ﻁ
If I go to charmap
and I pick this character
I type FEC1 in the unicode search box and hit ENTER and it finds the character
I look it ...
11
votes
4
answers
15k
views
Cannot use Alt code for Unicode character insertion any more
I've been using the Alt code for the ellipsis, 8230, for some time now, in several applications. A few days ago it stopped working, and & is displayed instead of … when pressing Alt+8+2+3+0 (on ...
11
votes
3
answers
5k
views
Getting gvim to show unicode
How to get GVim (v. 7.3. on Windows XP, if that matters) to show Unicode characters? Something along the lines of
http://vimcasts.org/episodes/show-invisibles/
(search for listchars to see the ...
11
votes
1
answer
20k
views
What's the name and where can I find this strange RTL character?
See, I've just found out something that impressed and confused me at the same time. There's this strange character which allows you to type everything reversed by default. For example, if I type:
"...
11
votes
3
answers
12k
views
Change Firefox default encoding for text files
Text files do not include character encoding information, so Firefox has no way of knowing which one is correct. Through the menu View → Text encoding I can select the appropriate encoding. I find ...
11
votes
1
answer
23k
views
Where are the other letters in this Unicode block?
Info
On the Wikipedia page, "Unicode Subscripts and Superscripts", it says in the section, Other Superscript and Subscript Characters,
Consolidated, the Unicode standard defines ... a full ...
11
votes
4
answers
87k
views
Permanent fix for unicode characters not displaying correctly (as boxes)
Please read this entire message before replying.
First, I know how to fix the issue on a temporary basis. I am looking for a permanent fix. I work with foreign language files a lot.
Unfortunately ...
10
votes
5
answers
39k
views
UTF-8 Angle brackets up and down?
I'm looking for angle brackets which point up and down as UTF-8 characters.
Just for clarification: I need two characters, an angle bracket pointing up and one pointing down.
10
votes
3
answers
62k
views
Convert Unicode code point to corresponding character in Excel spreadsheet
I use the CHAR() function to convert an extended ASCII value into a character. What do I use to convert a Unicode code point into a character?
I'm building a Hebrew conversion chart. For example, ...
10
votes
1
answer
7k
views
Unicode Characters in Ubuntu Server
I recently migrated from Ubuntu Desktop 13.04 to Ubuntu Server 13.10 and now my filesystem doesn't seem to support unicode characters. I use rsync to backup/sync files and it shows the proper ...
10
votes
4
answers
27k
views
On Windows 7, dir or tree can't show unicode characters, even starting cmd with cmd /U
On Windows 7, dir or tree can't show unicode characters, even starting cmd with cmd /U
So I would press Window Key + R to run something, and type in cmd /U so that the content might handle Unicode.
...
10
votes
2
answers
2k
views
What Font does Notepad use to display e.g. Hebrew, when it claims to use Lucida Console but Lucida Console doesn't have those characters?
Courier New has the character \u05D0 hebrew letter aleph
some other fonts do too such as Miriam and David.
I can paste that into notepad
Lucida Console does not have the character. Character map ...
10
votes
1
answer
2k
views
How to stop Windows 10 Calculator from enclosing copied text in 202D and 202C Unicode characters?
If I use Windows 10 calculator to copy one of its produced values:
What it copies to the clipboard is text enclosed in these two Unicode
characters:
'LEFT-TO-RIGHT OVERRIDE' (U+202D) on the left, ...
10
votes
1
answer
5k
views
Can GNU sed (for Windows) handle Unicode? If so, is it a code-page/locale issue, or a switch?
I've been using GNU SED on and off for a couple of years now. It spins me out a bit sometimes, but it does a good job... for single-byte char sets!
I now and then notice references to GNU SED being ...
9
votes
2
answers
2k
views
Why does VIM show the Unicode code point and not the UTF-8 code value?
Consider this supposed line of code that I found in a PHP blog, note the quotes:
throw new Exception(“That's not a server name!”);
Those quotes are RIGHT DOUBLE QUOTATION MARK (Unicode code point: U+...
9
votes
3
answers
2k
views
fixing mis-encoded Chinese in file names
I'm saddled with a bunch of files whose names are garbled beyond recognition. Even though I more or less know what those names originally contained, fixing them by hand would involve a lot of hassle, ...
9
votes
3
answers
2k
views
Why are there separate fixed-width characters for 0-9 in Japanese, compared to the typical 0-9?
Japanese (日本語) -0123456789
Typical ASCII for everywhere else - 0 1 2 3 4 5 6 7 8 9
Why was there a need to create a separate character set for the same numerals?
9
votes
3
answers
6k
views
Get Supported Unicode Ranges from Font *.ttf file on Ubuntu
I have some fonts in 'ttf' format on my Ubuntu machine.
I want to know some metadata of those fonts, about which unicode ranges and which non-unicode codepages those font files are supporte.
How I ...
9
votes
8
answers
24k
views
Cannot copy non-latin characters from PDF document
I have a pdf file which contains some non-latin european characters. If I copy some text with the highlight tool, and paste it into another program (word, notepad) - the 'special' characters do not ...
9
votes
2
answers
19k
views
Several unicode characters are not displaying on many big sites (like this: )
There are several Unicode charactes I'm just seeing as boxes. This happens in Chrome only. The Character Encoding is set to UTF-8, but Autodetection doesn't help.
Example: []
You will only see two ...
9
votes
2
answers
2k
views
In utf-8 collation, why 11- is less then 1-?
I found that the sort result in ASCII:
Source file test:
1-
11-
1-a
11-a
Sort using ASCII:
$ LANG=en_US.ascii sort test
1-
1-a
11-
11-a
And using UTF-8:
$ LANG=en_US.utf8 sort test
1-
11-
11-...
9
votes
2
answers
26k
views
How do I view unicode characters on command prompt
I read this post on superuser on how to view UTF-8 characters on command prompt in windows. I tried the steps in the answer which were :
Start -> Run -> regedit
Go to [HKEY_LOCAL_MACHINE\...
9
votes
3
answers
6k
views
Vim: How to handle Unicode files with text in multiple (more than two) languages?
What settings do I need to set in Vim/gVim to be able to view Unicode text files which have text in many languages?
You may make these assumptions:
The number of languages is more than two.
Some of ...
9
votes
2
answers
8k
views
Command Prompt "tree" shows Unicode in console but outputs to file in ASCII
In Windows 10's cmd.exe, if I do
tree
C:.
├───a
│ ├───b
│ └───etc...
it clearly display the Unicode bars and dashes correctly on the console.
But if I redirect this output to a file
tree > ...
8
votes
3
answers
10k
views
Why some unicode characters are missing in charmap?
In Windows, Character Map doesn't seem to display all of the unicode characters, even if the selected font supports them, and "Character set" selector is "Unicode".
For example, the small envelope U+...
8
votes
3
answers
14k
views
Using the “Combining Enclosing Circle” character in Word
Is there a way to use the “Combining Enclosing Circle” UNICODE character (U+20DD) together with some other character in Word 2010 to actually form a circled character? There are glyphs for e.g. ...
8
votes
2
answers
3k
views
Insert a character with multiple accents on an extended Mac keyboard layout
I can't seem to figure out how to apply both an Umlaut (¨) and the Grave (`) to the the letter 'u'. I'll type one, then the other, then the letter, but only the latter accent is applied, with the ...
8
votes
3
answers
8k
views
Unicode characters in bibtex
I am struggling with unicode chars in bibtex (Debian). I have the following entry in my .bib file:
@Book{bjork2009,
author = {Tomas Björk},
title = {Arbitrage Theory in Continuous Time},
publishe = {...
8
votes
4
answers
4k
views
How to stop unicode glyphs displaying as "emoji" in MS Word 2016
In Word 2010, typing (e.g.) Alt+1 or Alt+4 (using the Numpad) produces Unicode symbols for a smiley face or diamond symbol in the selected font – e.g. ☺ ♦
In Word 2016, these are now replaced with ...
8
votes
2
answers
14k
views
How do I remove non-ASCII characters from filenames?
I have several files with names containing various Unicode characters.
I'd like to rename them to only contain the "printable" ASCII characters (32-126).
E.g,
Läsmig.txt //Before
...
7
votes
2
answers
4k
views
Why are Craigslist posts full of question marks?
There's no craigslist presence in my city, but sometimes I browse the "best of" because it's funny or interesting.
Why are some posts full of question marks? It looks like all punctuation except ...
7
votes
3
answers
7k
views
What is a quick way to write "dagger" sign in MS Word equation mode?
In MS Word equation mode with the Unicode converter, I write symbols by putting the "\" sign and then typing the specific Unicode, i.e., for the summation sign I write "\sum" and ...
7
votes
1
answer
19k
views
Windows tool to convert large text file from UTF-8 to "Windows Unicode" (UTF-16)
I need to view large Unicode text files (current version is 2,379,415,348 bytes) on Windows 7.
Normally I prefer UTF-8 but, but after looking on SuperUser it seems the best Windows large file viewer ...
7
votes
1
answer
2k
views
Any chance to get a superscript directly on top of subscript in Unicode?
Unicode provides subscripts and superscripts, so I can do this:
x²
And this:
x₅
However, combining these two I get:
x²₅ or x₅²
Which looks badly.
Any chance to get the superscript directly on ...
7
votes
2
answers
15k
views
rsync on Windows + non-ASCII file names
Is there a version of rsync that runs on current Windows versions and can handle non-ASCII file names? Every version I checked (including cwRsync and rsync.net client) fail miserably on files with ...
7
votes
4
answers
2k
views
Unicode in an Emacs regular expression
I'm using emacs 24.
How can I replace all occurrences of the non-printable Unicode bidi character RTL, whose hexadecimal number is 202e? I want to replace it with the non-printable Unicode bidi ...
7
votes
2
answers
82k
views
Unicode characters being replaced by question marks after copy and paste on Windows
I have this odd issue that I've been having for years now since Windows 8 (I'm now on Windows 10, if I remember right. The problem only seems to be on my work computer. My other personal computers don'...
7
votes
1
answer
1k
views
Why do text editors think this file is UTF-8?
I have two text files, which I'm giving download links to rather than a pastebin to preserve their contents precisely:
field1.txt
field2.txt
Both of these text files consist only of spaces, carriage ...