Questions tagged [unicode]
Unicode is the standard for computer representation of plain text. It encompasses the Universal Character Set, intended to unambiguously represent all characters used in human writing systems in any language, Unicode Transformation Formats (UTFs), defining standardized formats for storing and transmitting Unicode text, and standards for processing and manipulating text.
693
questions
2
votes
1
answer
77
views
Convert UTF-16 LE to UTF-16 LE BOM or to UTF-8 in windows via command line
I am writing a batch script which pipes a program's output to a text file, the contents of that text file are then later meant to be read back into the script as a variable, which will be used for ...
- 33
8
votes
3
answers
841
views
fixing mis-encoded Chinese in file names
I'm saddled with a bunch of files whose names are garbled beyond recognition. Even though I more or less know what those names originally contained, fixing them by hand would involve a lot of hassle, ...
- 83
0
votes
0
answers
31
views
In Excel, how do I handle someone else's workbook that uses UNICODE?
I have been sent an Excel workbook in which I find that a cell that appears to contain the letter "A" produces FALSE if I compare it with "A". That is, if the cell in question were ...
- 420
0
votes
0
answers
17
views
How to customize fonts in Chrome by Unicode block (or even Unicode character)?
On its default, Chrome chooses which fonts to use on its own and utterly ignores the fact the user has installed fonts that would support much more characters. For example, it ignores all CJKV ...
- 81
0
votes
0
answers
24
views
Is the dataset that underpins the Windows Character Map utility accessible somewhere on the system?
The Windows Character Map utility displays the glyphs available in the chosen font, along with the Unicode codepoint and a name for a selected character. Is the dataset that underpins this utility ...
- 635
1
vote
0
answers
57
views
tmux inside Cygwin mintty - unicode characters broken depending on command
Please help me understand what's going on with some unicode characters looking right inside mintty + tmux in some command outputs, but not in others.
Sample commands:
for c in 00AE 1F007 1F32D 1F603; ...
- 171
0
votes
1
answer
63
views
Missing some Unicode glyphs on Windows 7, is there way to fix/patch it?
I have an arrowsbe.txt file, with the following content (in hex):
FE FF 00 41 00 42 00 43 2B A1 00 20 2B A2
You see, it is UTF-16BE encoded, with BOM at file start. There are two not-so-often seen ...
- 5,563
0
votes
0
answers
21
views
How to pass custom unicode characters to xrdb?
I use xrdb to set a the window title of my terminal (Zutty). I use the following line in the file ~/.config/X11/xresources (UTF-8 encoded):
Zutty.title: 端末
Then I load it using xrdb -load ~/.config/...
- 63
0
votes
0
answers
25
views
Unicode entity SHADDA WITH FATHA MEDIAL FORM — faulty rendering in firefox
Unicode code point 64754 decimal represents arabic script entity 'shadda with fatha' (within word context, not isolated). Thus the official Unicode name 'SHADDA WITH FATHA MEDIAL FORM'.
(There is also ...
- 1
1
vote
0
answers
21
views
Unicode letters replaced by ***** (letter n )
the letter n is being replaced with **** in my console when I run dbt.
In text format:
11:59:24 Fou*****d 397 models, 15 tests, 0 s*****apshots, 0 a*****alyses, 639 macros, 0 operatio*****s, 11 seed ...
- 11
0
votes
1
answer
32
views
unicode-escaping with a standard Unix tool
I have cooked up the following python script, it does 1 thing and it does it well. It does, whatever Unicode magic unicode-escape does in python3 to some text.
#!/usr/bin/env python3
import sys
text ...
- 111
1
vote
1
answer
119
views
Convert Korean files that are showing up incorrectly to utf-8 - character shows Çѱ¹Ÿî
I was just about to ask this after a long time of searching so decided to answer my own question...
I downloaded Korean subtitles in an .smi file that was in zip archive. When I extracted it, the ...
- 251
7
votes
3
answers
3k
views
What is a quick way to write "dagger" sign in MS Word equation mode?
In MS Word equation mode with the Unicode converter, I write symbols by putting the "\" sign and then typing the specific Unicode, i.e., for the summation sign I write "\sum" and ...
- 197
2
votes
0
answers
586
views
How can I set my system's default encoding to UTF-16?
My daily activity involves usage of English, French, Spanish, and when I save personal copies of web pages or other documents, the full character range of those languages finds its way into filenames ...
- 222
0
votes
0
answers
153
views
How to encode to base64 Unicode UTF-16LE in batch script with VBS or certutil and without powershell
I know I can do below with powershell in batch, but for some reason, I need to do it without powershell. Is it possible?
@echo off
set "string=Everything will be fine"
for /f &...
1
vote
1
answer
99
views
zmv to rename Unicode characters (e.g. u0308 'COMBINING DIAERESIS') on Mac/zsh-shell
While cleaning a file server I found unwanted or non-ASCII characters in many filenames.
To rename unwanted filenames, I normally use the comfortable zmv command in a zsh shell on an OSX machine.
To ...
- 11
0
votes
0
answers
191
views
Better calligraphy in word equations
So I want to type calligraphy letters in word's equation editor, i.e., \mathcal{L} in TeX and \scriptL in word's unicode. Currently, I'm using Latin Modern Math as my default math font for MS word, ...
- 101
0
votes
0
answers
277
views
Problems with certain Unicode characters
I am using word, and I needed to insert the name of the Chinese Nüshu script in Nüshu (𛆁𛈬), there wasn't a font that supported it, so I downloaded every font I could find that supported it. Noto ...
- 1
2
votes
0
answers
50
views
Is it possible to write the Lovecraftian Sigil of the Gateway in unicode?
It isn't technically copyrighted and doesn't seem to be subject to trademark, and is kind of a cultural symbol, so for all I know there's a hidden symbol assignment for it somewhere... but the Sigil ...
- 135
0
votes
1
answer
351
views
Can't copy chinese text from PDF
I tried to open it with google docs, and tried the site i2orc to convert the file. But in both cases the Chinese characters are missing from the output, just as when i copy/paste them from the ...
2
votes
0
answers
80
views
Not able to copy Korean characters from PDF file using Acrobat Reader
I decided to study Korean and I found a book that I want to copy some vocabulary from, the problem is when I paste the copied text, it turns into gibberish characters. Any idea how to solve this? I ...
- 121
1
vote
1
answer
4k
views
Unicode backspace key symbol ⌫
Is there a Unicode symbol for the backspace key ⌫, the x inside a left-pointing arrow?
I know that Unicode has a "BACKSPACE" control character (U+0008) that it inherited from ASCII, and it ...
- 4,000
0
votes
0
answers
26
views
How to eliminate a space between two unicodes effectively layering one on top of the other
joiner u + 0008 only moves one space what moves more than one basically what allowing you to overlap accents an spaces between some Unicode more effectively
0
votes
1
answer
133
views
Binding unicode λ symbol (code+glyph) to key combination
I would like to type Unicode λ in a clear way (key L and alt modifier, for example) and not pressing SHIFT+CTRL+u03bb each time (or doing a poor trick like copy/paste).
I've tried a lot of ways to ...
- 123
1
vote
1
answer
290
views
How can I type rough breathing marks in Classical Greek using the Windows 11 Greek Polytonic keyboard?
I'm learning to use the Greek Polytonic keyboard in Windows 11 to type Classical Greek.
I can successfully type the following characters (using α as an example): ᾶ, ἀ, ἄ, ἂ, ά, ά, ἆ, ᾱ. For example, ἄ ...
- 31
0
votes
1
answer
63
views
Why can't I print this in windows ( ه҈҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉ ')?
I got this from the following question: What are these ? ه҈҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉҉
I can't get this to print in windows powershell. I tried using chcp 65001. BTW I can print 北京 but not ه҈҉҉҉҉҉҉҉҉҉҉҉...
- 101
0
votes
0
answers
614
views
Why is it so HARD to have Windows 10 CMD console display Unicode correctly?
Before showing the hard(difficult) example of Windows 10 (Win10.21H2 in this case), let me show an easy example by Ubuntu 22.04 .
I place file names with Chinese and Korean characters on a USB flash ...
- 5,563
2
votes
1
answer
176
views
Large Green Circle Emoji is not being displayed
My browser and apps are not displaying the Large Green Circle Emoji 🟢 [unicode 1F7E2].
Instead, the following is being displayed .
I would appreciate some help in fixing this.
I am using a MacBook ...
- 21
1
vote
1
answer
128
views
Counterclockwise Arrows emoji displays wrong on Windows 11
The Counterclockwise Arrows emoji (U+1F504) is supposed to look like this:
Instead, it displays as the end arrow emoji (U+1F51A):
The following list are the unicode characters in question:
🔄 (...
- 533
0
votes
0
answers
247
views
Python hangs when printing some Unicode c1 control characters, not in a consistent manner
For testing purposes, I created a file with bytes 0xc2 0x90, representing Unicode code point 0x90 in UTF-8 encoding.
I can cat this file without an issue, but trying to print the string written from ...
- 105
1
vote
1
answer
290
views
Unicode in batch
My question is simple.
I need to put fancy unicode into a batch file. I already put chcp 65001 at the start.
Here is my example code:
@echo off
chcp 65001 >NUL
echo [✓] This is a correct line
echo [...
- 23
1
vote
1
answer
20
views
IDN U-Label to A-Label wrong with many services
I own the Unicode Domain κλαρα-σωλις.ευ, which should be converted to xn----ylba7abgd9bnh0e.xn--qxa6a, but many services convert the URL to xn----ylba7abgd9buc7d.xn--qxa6a.
Websites like https://www....
0
votes
1
answer
129
views
Two seemingly identical VS Code Linux installations display fonts differently
I have two laptops, both running the same version of latest Manjaro. The same version of visual-studio-code-bin from AUR is installed. Both use the same font for code. Yet, they render Unicode symbols ...
- 131
1
vote
1
answer
530
views
Unicode character is invisible on Windows but shows up on Linux
The character in question is the Braille Pattern Blank: "⠀"
On machines with Windows and even on my phone, this character is rendered as invisible (as seen here). This is helpful because it ...
- 13
1
vote
0
answers
31
views
If a ligature/digraph has separate collation order, should it be considered different from constituent letters?
There are many digraphs and ligatures in common use like ß, w, ffi, st, æ, dž, ch etc.
Some of these are completely different from its constituents. For eg, search for u in a string containing w should ...
3
votes
0
answers
9k
views
Can I enable Unicode UTF-8 WorldWide Support in WIndows 11, but set another encoding for a specific app?
generally, I like to enable UTF-8 worldwide support in Windows 11, because most of apps I use support UTF-8 and I am using Linux OS with UTF-8 at the same time.
But the problem I had so far is one of ...
- 525
0
votes
0
answers
279
views
when running "LC_ALL=C.UTF-8 egrep -axv '.*'" to detect non-UTF8 characters how can I determine the exact character that caused the detection?
I frequently use this command:
LC_ALL=C.UTF-8 egrep -laxv '.*' filename
this tells me if the file contains any non-UTF8 characters (actually I usually use this in conjunction with find to scan many ...
1
vote
2
answers
1k
views
How do you open/import a CSV file with Unicode characters encoded as numeric character references?
In Excel, I tried to import (using Data > Import) a CSV file with Chinese characters. The characters are represented as Unicode numeric character references (NCR); for example 香&#...
- 113
1
vote
1
answer
70
views
password recovered in weird format
i have been trying for years to recover my old data from very old DS cards " MMC "
recently i found out it is possible to get to mmcstore file that stores the password in plaints text
all ...
- 11
1
vote
0
answers
90
views
Reproducing folder named *
The question Folder named * was brought to my attention. I thought I could easily reproduce this by taking Git Bash and entering
mkdir \*
At first I didn't look closely, I just saw a folder appear in ...
- 5,615
0
votes
1
answer
46
views
Confusing Unicode phenomenon
I have an irritating phenomenon in my home network.
My NAS is a Synology, 3 workstations are Apple based and 1 is Windows 10.
Lately I had a problem copying files using the Apple finder and it croaked ...
- 175
0
votes
2
answers
797
views
Do web browsers have Unicode character files in them or they inherit these from the operating system?
If we create at least one new character with at least one typography set we would have to include any character file and any font file, in the browser file tree and also declare these files in the ...
- 3
1
vote
1
answer
466
views
How can you type characters from U+0100 and up with Numpad?
It appears that the Alt + Num shortcuts end at U+00ff (decimal 0255); typing Alt + 0255 yields ÿ which indeed is at U+ff. I have failed in finding out what the system for the hotkeys in excess of ...
- 149
0
votes
1
answer
2k
views
Simplified Chinese characters are not displayed correctly in Windows command prompt
When I enter or paste a simplified Chinese character, it is displayed weirdly. It looks like a 'p' character enclosed in a rectangle. Please see the picture attached.
But I copy the character and ...
- 1,850
0
votes
2
answers
453
views
Trouble with perl and UTF-8
If I interpret documentation ci=orrectly, perl --CSD is suppose to ensure that input and output, processed or commands, are using UTF-8 encoding.
But if I replace two hyphens -- with an em-dash — (U+...
- 715
-1
votes
1
answer
1k
views
Which Unicode characters cannot be used for NTFS file names?
Typically, my programs are creating files with Extended Latin, CJK or Arabic characters in file names without issues and according to input data.
I was under impression that only officially named ...
- 12.9k
1
vote
0
answers
82
views
Old Hangul (옛한글) on Safari not rendering
I have a Mac(MacOS 12.0.1) and use Safari as my default browser. Across my Mac, despite having several fonts that support it, Old Hangul (옛한글) does not render how it is supposed to.
In more detail, ...
- 11
0
votes
1
answer
389
views
Printf in gawk with the correct encoding?
I'm wondering: can gawk printf in any format besides ASCII?
Currently, I'm using gawk match() to search through some UTF-8 text. When I go ahead and print out the matches gawk finds, it ends up like ...
- 1
1
vote
1
answer
126
views
Is there a way to set up characters with 5 digits of unicode as dead key code points in MSKLC?
Is there any way to overcome this software limitation?
As seen here
it only takes inputs of 4 digits long while i need to input in unicode keys 5 digits long, when I try to do it, an error message ...
1
vote
1
answer
40
views
What is the point of "extended" digits like ۱
This stack exchange comment uses the ۱ character, which I had never seen before. Using babelstone, I've identified the character:
U+06F1 : EXTENDED ARABIC-INDIC DIGIT ONE
Now, the interesting part ...
- 131