I'm doing an export from command line on ms-dos with mysqldump:
& mysqldump -u root -p --default-character-set=utf8 -W -B dbname
> C:\mysql_backup.sql
My database/tables are encoded with UTF-8 and I specify the same encoding when I did the dump. But when I open the file with Notepad++ or Scite I see an encoding of UTF-16 (UCS-2). If I don't convert the file with iconv to UTF-8 before running the import I got an error.
It seems that MS-DOS / CMD.exe is redirecting by default with UTF-16. Can I change this ?
A side note: I use Powershell to call mysqldump.
UPDATE: it seems that it occurs only when calling mysqldump from Powershell. I change the command line with the one I use in my PS script
By default PowerShell represents text as Unicode and when you save it to a file it saves as Unicode by default. You can change the file save format by using the Out-File cmdlet instead of the > operator e.g.:
... | Out-File C:\mysql_backup.sql -Encoding UTF8
You may also need to give PowerShell a hint on how to interpret the UTF8 text coming from the dump utiltiy. This blog post shows how to handle this scenario in the event the utility isn't outputting a proper UTF8 BOM.
Related
I've just inherited a legacy project in which my predecessor pushed incorrectly encoded files.
The comments, in French, should include special characters as é,è,ç etc.
But, for instance here, a 'é' is shown as 'é'.
I'm looking for a command line tool to handle all files of the project. I'm pretty sure iconv should to the trick, but what I tried so far did not work :
Here are some initial informations:
# problematic file example
$ file Parametres.cpp
Parametres.cpp: C source, ISO-8859 text
# check that my OS handles utf8
$echo "éè" > test.tmp
$ file test.tmp
test.tmp: UTF-8 Unicode text
$ cat test.tmp
éè
I tried whithout success (meaning in Parametres.cpp.utf8 i still got 'é') :
iconv -f ISO-8859-1 -t UTF-8 Parametres.cpp -o Parametres.cpp.utf8
iconv -f ISO-8859-1 -t UTF-8//TRANSLIT Parametres.cpp -o Parametres.cpp.utf8
iconv -f ISO-8859-1//TRANSLIT -t UTF-8 Parametres.cpp -o Parametres.cpp.utf8
My guess is that the original encoding was not ISO-8859-1 but something else. And due to misconfigured IDE, chars 'Ã' and '©' got definitly encoded in ISO-8859-1. From what I understood, TRANSLIT should to the job, but it seems not.
So, here are my questions :
is there a better tool than iconv to do this job in CentOS7.2 (yes, I know. Legacy is legacy...) ?
Or, How to determine (or guess) the original encoding to make iconv solve my problem ?
Any help or ideas are appreciated :-)
I have a bash script which extract data from an oracle database. I use spool to extract data. After extraction I format the file by removing and replacing some characters. My problem is after formating the files are in ANSI encoding instead of ut8.
Extraction with spool. The file is utf8
Format with cat and tr command and redirect in another file. This file is ansi.
The same process works fine on Aix system. I try iconv but it doesnt work. Do you please have an idea why the encoding changes from utf8 to ansi ? How to correct it please ?
You should consequently use either ISO-8859-1 or UTF-8. In the latter case, don't use tr as it doesn't (yet?) support multi-byte characters, use sed instead (e.g sed 's/deletethis//g').
ISO-8859-1:
export LC_CTYPE=fr_FR.ISO-8859-1
export NLS_LANG=French_France.WE8ISO8859P1
# fetch data from Oracle, emulated by the following line
echo 'âêîôû' >test.latin1 # 5 bytes (+lineend)
# perform formatting, eg:
sed 's/ê/[e-circumflex]/g' test.latin1
# or the same with hex-codes:
sed $'s/\xea/[e-circumflex]/g' test.latin1
UTF-8:
export LC_CTYPE=fr_FR.UTF-8
export NLS_LANG=French_France.AL32UTF8
# fetch data from Oracle, emulated by the following line
echo 'âêîôû' >test.utf8 # 10 bytes (+lineend)
# perform formatting, eg:
sed 's/ê/[e-circumflex]/g' test.utf8
# or the same with hex-codes:
sed $'s/\xc3\xaa/[e-circumflex]/g' test.utf8
Note: no conversion (iconv, recode, etc) is required, just make sure NLS_LANG and LC_CTYPE are compatible. (Also, your terminal(emulator) should be set accordingly; for PuTTY it is Configuration/Category/Window/Translation/Remote-character-set.)
Original answer:
I cannot tell what's wrong with the formatting you perform, but here is a method to damage the utf8-encoded text:
$ echo 'ÁRVÍZTŰRŐ TÜKÖRFÚRÓGÉP' | iconv -f iso-8859-2 -t utf-8 | xxd
00000000: c381 5256 c38d 5a54 c5b0 52c5 9020 54c3 ..RV..ZT..R.. T.
00000010: 9c4b c396 5246 c39a 52c3 9347 c389 500a .K..RF..R..G..P.
$ echo 'ÁRVÍZTŰRŐ TÜKÖRFÚRÓGÉP' | iconv -f iso-8859-2 -t utf-8 | tr -d $'\200-\237' | xxd
00000000: c352 56c3 5a54 c5b0 52c5 2054 c34b c352 .RV.ZT..R. T.K.R
00000010: 46c3 52c3 47c3 500a F.R.G.P.
Here the tr -d $'\200-\237' part deleted half of the utf8-sequences (c381 became c3, c590 became c5), rendering the text unusable.
I have a set of md files, some of them are utf-8 encoded, and others are not (windows-1256 actually).
I want to convert only non-utf-8 files to utf-8.
The following script can partly do the job:
for file in *.md;
do
iconv -f windows-1256 -t utf-8 "$file" -o "${file%.md}.🆕.md";
done
I still need to exclude the original utf-8 files from this process, (maybe using file command?). Try the following command to understand what I mean:
file --mime-encoding *
Notice that although file command isn't smart enough to detect the right character set of non-utf-8 files, It's enough in this case that it can distinguish between utf-8 and non-utf-8 files.
Thanks in advance for help.
You can use for example an if statement:
if file --mime-encoding "$file" | grep -v -q utf-8 ; then
iconv -f windows-1256 -t utf-8 "$file" -o "${file%.md}.🆕.md";
fi
If grep doesn't find a match, it returns a status code indicating failure. The if statement tests the status code
Does mongoimport cli command support only UTF-8 format files?
Is there a way to provide encoding format so that, it can accept non-utf-8 files, without we manually converting each file to UTF-8?
This is one way of doing it on Linux/Unix. You could use iconv to convert non-utf8 to utf8 and then use mongoimport on the converted file:
iconv -f ISO-8859-1 -t utf-8 myfile.csv > myfileutf8.csv
man iconv should give you more details about options
Also, Import CSV file (contains some non-UTF8 characters) in MongoDb
discusses some options for windows.
I want to use iconv to convert files on my Mac. The goal is to go from "Windows ANSI" to "whatever Windows Notepad saves, if you tell it to use UFT8".
This is what I want:
$ file names.csv
names.csv: UTF-8 Unicode (with BOM) text, with CRLF line terminators
This is what I use:
$ iconv -f CP1252 -t UTF-8 names.csv > names.utf8.csv
This is what I get (not what I want):
$ file names.utf8.csv
names.utf8.csv: UTF-8 Unicode text, with CRLF line terminators
How do I get the BOM?
You can add it manually by first echoing the bytes into the file:
echo -ne '\xEF\xBB\xBF' > names.utf8.csv
and then concatenating your required information at the end:
iconv -f CP1252 -t UTF-8 names.csv >> names.utf8.csv
Note the >> rather than >.
Note that "Windows ANSI" may not be CP1252 - that is configured by users.
The BOM is not necessary for UTF-8.
And Windows Notepad can save UTF-8 with or without BOM.
I needed the opossite. (covert german text from UTF-8 to ANSI)
So command I used:
1. iconv -l (check available formats)
2. iconv -f UTF8 -t MS-ANSI de.txt > output.txt
and now if I open output.txt it is already in ANSI. Job done.