Questions tagged [character-encoding]
A character encoding system consists of a code that pairs each character from a given repertoire with something else — such as a bit pattern, sequence of natural numbers, octets, or electrical pulses — in order to facilitate the transmission of data (generally numbers or text) through telecommunication networks or for data storage.
299
questions
0
votes
1
answer
120
views
How to read Linux text files in Windows system?
For example, I run the top command and store it to a file in Linux, after that I open that file in Windows it contains some gibberish. Here is the file viewed in Notepad++:
The option to convert to ...
- 11
1
vote
1
answer
37
views
does batch program support any ascii characters or is there's a way to create encoder and decoder program with another tools?
i wrote batch encoder and decoder from adrianvdh and customize some of the text string input, but the decoder one aren't working, because i put special symbols inside there.
here's the string of the ...
Arip NH
0
votes
1
answer
43
views
AWK: "invalid regexp: Invalid collation character" -- how do I make it valid?
I have an awk script that must process millions of records, but I need to remove any containing a multibyte character.
In one environment where I work, the following simplified shell sequence ...
- 181
0
votes
0
answers
21
views
How to pass custom unicode characters to xrdb?
I use xrdb to set a the window title of my terminal (Zutty). I use the following line in the file ~/.config/X11/xresources (UTF-8 encoded):
Zutty.title: 端末
Then I load it using xrdb -load ~/.config/...
- 63
0
votes
0
answers
60
views
How to use ISO8859-9 encoding in terminal?
I maked a file containing "ırmak" with a text editor via encoding ISO8859-9.
Then, I tried to print the content with "cat" command in the terminal. But I could not.
I use the ...
- 1
1
vote
1
answer
114
views
Convert Korean files that are showing up incorrectly to utf-8 - character shows Çѱ¹Ÿî
I was just about to ask this after a long time of searching so decided to answer my own question...
I downloaded Korean subtitles in an .smi file that was in zip archive. When I extracted it, the ...
- 251
0
votes
2
answers
194
views
Restoring corrupted UTF-8 files
After my PC broke down I managed to make a backup of the relevant files before reinstalling Windows.
Now that I'm restoring those files and setting the system up I noticed that some of the files got ...
0
votes
0
answers
444
views
Redirect the Windows console output of a Python script without raising UnicodeEncodeError
I have a Python script called scratch_1.py which can be simplified for the sake of the question to:
s = "∞"
print(s)
If I run the script in the Windows console, I get the expected output:
...
- 187
2
votes
0
answers
569
views
How can I set my system's default encoding to UTF-16?
My daily activity involves usage of English, French, Spanish, and when I save personal copies of web pages or other documents, the full character range of those languages finds its way into filenames ...
- 222
3
votes
1
answer
3k
views
My text file is riddled with question marks. How can I make it readable?
When I open one of my text files in Visual Studio Code,
the text contains a lot of question marks where I had expected to see
Swedish letters, such as å, ä, ö :
^ click to enlarge
Down to the right (...
- 751
1
vote
0
answers
47
views
Mounting .iso image in Linux Mint with incorrect filename encoding
I am trying to mount some ISO files but have filename encoding issue. The .iso file in question is here: https://archive.org/download/cpcfan-200510b/200510B.iso
Mount command and incorrect filename:
$ ...
- 111
2
votes
1
answer
233
views
How to restore Arabic text of very old emails
I have an old Yahoo mail account, and I have very old emails since 2006 written in Arabic, but the encoding of these emails looks very weird, something like:
Óæì Ø *Ý æÚæÏß ßáÇã Ý * ßáÇã. ÇÍÈß ÊÕæÑ ...
- 298
1
vote
1
answer
275
views
How does Git for Windows' cat.exe deal with charset encoding?
I'm testing the behaviour of Windows terminal (cmd.exe) in relation to charset encodings. I have some test files in several encodings (Win1252, CP437, UTF-8, etc) with the Spanish text: "qué tal&...
- 645
0
votes
1
answer
1k
views
Using iconv and file - how do I change an incorrect character encoding setting?
I have a folder full of ASCII text files that have the file information set so text editors on the mac believe they are Turkish. The original notes in the folder claim it is Windows Latin 1 (Turkish ...
- 135
0
votes
1
answer
382
views
Printf in gawk with the correct encoding?
I'm wondering: can gawk printf in any format besides ASCII?
Currently, I'm using gawk match() to search through some UTF-8 text. When I go ahead and print out the matches gawk finds, it ends up like ...
- 1
2
votes
1
answer
502
views
How to decompress an encrypted zip file with ANSI encoded password?
I need to decompress zip files generated in Windows with Japanese language. I'm using unzip.
If I use unzip files.zip I will get bad file names. So, according to this question, I used unzip with -O ...
- 21
2
votes
1
answer
693
views
Change default text encoding in Excel 2019?
How can I change the default text encoding in Excel 2019?
I already tried this hack/workaround using the registry editor https://superuser.com/a/1179248/1468612
( based on this: http://www.lukemiller....
- 21
9
votes
6
answers
8k
views
Why does Unicode have big or little endian but UTF-8 doesn't?
UNICODE uses 2 bytes for one character, so it has big or little endian difference. For example, the character 哈 is 54 C8 in hex.
And its UTF-8 therefore is:
11100101 10010011 10001000
UTF-8 uses 3 ...
- 2,840
1
vote
0
answers
59
views
How to fix accentuation encoding with cmd.exe running inside bash?
I installed https://www.msys2.org/ and setup an ssh server for it. With this I can connect to my machine and work remotely. The problem is that some application as Visual Studio tools or windows ...
- 378
1
vote
2
answers
1k
views
Does clipboard change character encoding?
I am having a UTF8-related issue in a piece of software - it complains some text containing special characters isn't valid UTF8. But whenever I copy-paste it into an online validation tool or into an ...
- 4,966
0
votes
1
answer
503
views
Why does copying text between Notepad++ files create files with different bytes?
I've created a simple pdf [hi.pdf] with the word hi and when I open it in Notepad++, its encoding is ANSI, which I assume is Notepad++'s best guess, with it opening successfully when I Save as ...
- 111
0
votes
1
answer
205
views
convert character set (German)
I have a text file that uses various characters in the 128+ range in currently non-standard ways.
The file command just says Non-ISO extended-ASCII.
From the context I can recognise these:
Octal 201: ...
- 169
1
vote
2
answers
472
views
Incorrect display of Chr() function return
I have got a new work PC (it has a higher build of Windows 10, the same highest version of O365), configured it pretty much the same as my previous one (exactly like that in terms of region and ...
- 204
0
votes
0
answers
474
views
What is this for a file name encoding and how to fix it?
On my Linux machine I found old files (at least from 2004 if not older), so possibly Win9x days. Maybe they came over some old FAT drive on my disk or some old Samba share.
Umlaute are very weirdly ...
- 565
1
vote
1
answer
53
views
How to prevent Safari from implicitly converting character in XHR request?
I picked this character 〉 as a separator for my combo-key-field for my DynamoDb database.
That character surfaces in the browser as part of a next-page-query token. (in an endless scroll list view)
...
- 113
0
votes
1
answer
227
views
WinSCP Displays Weird Characters in script output file
I'm viewing the outputs of a bash script that logs into juniper devices and runs commands.
When I view the output of the script via the unix server using cat <filename> it appears just fine.
...
1
vote
2
answers
10k
views
How to display special non-ASCII Unicode characters in Notepad++? (regardless if file saved and reopened)
When bookmarking a URL like:
https://addons.mozilla.org/firefox/addon/youcare-search/
You get this bookmark title:
YouCare - The charitable search engine – Get this Extension for 🦊 Firefox
Then, if ...
0
votes
2
answers
1k
views
Finding the encoding of a text file containing weird characters
I recently received a file, of Turkish origin, where the file has some English words which I can easily read, and some weird characters. I wonder if this file is encoded, encrypted or sth else. I ...
- 101
1
vote
0
answers
50
views
Word 365 character encoding annoying
When I open some text files with Microsoft Word 365 (v2001 build 12430.20184 - 16.0.12430.20172) it asks me to choose the characters encoding.
My question is: How to clean the original file to let ...
- 35
0
votes
0
answers
48
views
Is URL encoding guarranteed to output printable characters only?
I refer to the URL encoding used in Burp and in the browsers
Is URL encoding guaranteed to output printable characters only?
- 131
1
vote
2
answers
1k
views
Windows 10 Missing Left Double Quotation Mark Glyph in Character Map
This is so weird.
I am simply trying to type an open double quotes symbol in Windows 10 that should look like this:
Actually, I'm trying to replace wrongly interpolated quotes symbols in an ANSI ...
0
votes
0
answers
270
views
Wireshark - Don't mask non-printable characters (Windows)
Is there any way to have Wireshark running under Windows not mask non-printable characters in the packet view on the bottom? Currently, any characters < 0x20 and >= 0x7f are shown as . This ...
- 1,165
0
votes
1
answer
153
views
BER decode SubjectAltName and CHOICE?
I'm having trouble working out the syntax when decoding a SubjectAltName in a TLS self-signed certificate. I believe the certificate is well formed. The trouble is, I don't understand how to decode ...
- 11.8k
5
votes
0
answers
312
views
Can I tinker with the encoding when using pdftotext to convert PDF to text?
Sometimes when I do pdftotext it results in perfect text. I assume this is because the actual unicode text data is embedded directly in the PDF itself, and simply read out.
But other times (around ...
- 351
0
votes
0
answers
359
views
Readable text when copy-pasted from PDF becomes completely unreadable
What is this that when I select a completely readable, Latin (no language-specific characters, actually a company name and street address) text in PDF document and then copy and paste it to any ...
- 10.6k
1
vote
4
answers
227
views
Which ANSI standard is Joel referring to?
I was rereading Joel Spolsky's classic blog post The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) and noticed this passage:
...
- 3,090
1
vote
1
answer
1k
views
In cmd.exe with codepage 437, why are characters whose "number" is greater than 127 prepended with a ┬ when using type?
I have a file that contains some characters whose "number" is greater than 127. If I use type file.txt to display the content of the file in a cmd.exe console whose code page (chcp) is set to 437, ...
- 2,219
5
votes
1
answer
974
views
How to make Chrome URL display spaces instead of %20
Browsers like Firefox display URLs containing spaces (including nbsp's) with an actual space (); Chrome always displays spaces as %20 (and nbsp's as %C2%A0) in the address bar. (ie, Firefox displays ...
- 151
0
votes
1
answer
187
views
NetTerm shows characters instead of lines
My issue is almost the same as Why does YaST now show lines as lqqqqqqqqqqqqqqq? , but instead of PuTTY and YaST, I'm using NetTerm and MLS (= WMS). My lines are also shown in characters as you can ...
1
vote
1
answer
2k
views
How to fix the character encoding problems after using filezilla?
I have the following problem:
I used the filezilla application to get the files of a page through FTP to backup them in case it needs to be restored later.
If it were to restore the site with the ...
- 133
2
votes
1
answer
153
views
How to send an e‑mail to an address with Latin9/iso‑8859‑15 characters inside the username part of the address?
As part of finding a job, I need to send an e‑mail to an address which contains latin letters with accents inside the username.
I know this is not standard, but they did it and there’s less than 1000 ...
- 1,770
0
votes
1
answer
137
views
CodeLite replacing accent by special characters
I'm using CodeLite to develop my PHP applications, but this one has the annoying tendency to replace all my accents with special characters (this one: � or others like é).
But UTF-8 encoding is ...
1
vote
0
answers
599
views
NFS mounted to drive fails with Hebrew file names
I have an NFS mounted to a drive on windows.
With English filenames all is working as expected.
Whit Hebrew I get this error:
“The file name(s) would be too long for the destination folder. You can ...
- 123
2
votes
1
answer
8k
views
Arabic characters do not display correctly when saving and reopening text file
When I save a text file with arabic letters from Notepad, they do not show correctly after reopening. Instead it shows ??????.
Example, this is my file in Notepad:
Then after saving, closing and ...
- 21
0
votes
1
answer
2k
views
How does notepad.exe determine character encoding?
I have a .txt file saved in UTF-8 format without a BOM. It contains an 'é' character.
How does notepad.exe determine that it is UTF-8 encoded?
Other .txt files containing only < 0x80 characters ...
- 869
0
votes
0
answers
888
views
How do you create custom zalgo text?
I know what zalgo text is and that there are a few websites that can make it for you. But I'm looking for how I can make it however I want." HͥAͣQͫ" is an example, how can I make it so I can choose ...
- 11
2
votes
1
answer
2k
views
Stop VS Code from auto guessing encoding
I use VS Code to make a German site.
I use a German special character in a style.css file.
After restarting VS Code and changing the file encoding from UTF-8 to
Windows-1252, I get what is shown in ...
5
votes
1
answer
1k
views
Windows 10 All alt codes and accentuated characters replaced by �
Since this morning, all alt codes I type in (like alt 256, alt 26, alt 144, ...), are replaced by the "�" character.
It also happens sometimes in Windows with accentuated characters (like é à ï ...). ...
- 221
1
vote
3
answers
6k
views
How to type directly with keyboard the U+200C character?
How to type directly with keyboard the U+200C character in Notepad or other places... in Windows 8.1? (without depending on a certain font)
I don't want to copy and paste ways.
I don't want any ...
- 11
0
votes
1
answer
610
views
Which type of character encoding consumes the most memory?
I will store a long (45132 character) string in a Postrgres database whilst preserving every character (including really rare ones).
Postgres can store strings up to 1GB (see here).
In terms of the ...
- 573