I am trying to insert a huge(~831M) file into mongo collection using mongoimport
/Library/mongodb/bin/mongoimport --port 12345 -d staging -c collection < out.all.1
and see some errors like
exception:Failure parsing JSON string near: , 'Custome
and there are instances where I found some weird characters
'CustomerCity': u'Wall \xa0'
'CustomerCity': u'La Ca\xc3\xb1ada Flintridge'
'CustomerCity': u'La Ca\xf1ada Flintridge'
How do I resolve these issues?
Thank you
I struck a similar problem where mongoimport gave errors about non-UTF8 characters in a flat file I'd asked it to import. This google groups thread led me to try putting my source data file through iconv on the unix command line to 'correct' non-UTF-8 characters, thus:
iconv -f ISO-8859-1 -t UTF-8 inputfile.txt > outputfile.txt
That solved the issue for me. I wonder if that approach might help you? While the error you're seeing is different, it's the odd characters that are messing up the JSON parsing, no?
One wonders, however, how those odd characters are ending up in your output data if you're generating it yourself. Perhaps you could filter in the code that generates the output?
Related
I have a postgresql .sql dump file created by pg_dump on another windows 10 box. I am trying to restore it on my windows 10 laptop with
"psql -U user -d database -1 -f filename.sql". I created the database, but when I run the command to do the restore I get an error from psql after I give it my password:
psql:filename.sql:1:1: ERROR: syntax error at or near "ÿ_"
LINE 1: ÿ_;
The file looks like straight ascii (I only see two dashes on line one. I don't see a 'y' with an umlaut anywhere). I did a file on the .sql file with cygwin bash, and it says:
Little-endian UTF-16 Unicode text, with very long lines, with CRLF, CR line >terminators
I really don't want to recreate the database by hand. I am looking for any suggestions.
I tried psql with and without the '-1' option; no luck. I tried putting a ';' at the top of the sql file, which I found suggested somewhere; again no luck.
I did a psql -l on my postgresql installation and the encoding on all my databases (including the one to which I am trying to do the restore) shows UTF8.
There really is no code. It is just that I can't seem to restore this dump file because it errors out.
I think that captures my problem. The windows box that I got the dump from is not available to me now; so I'm just hoping there is a way to get around this problem. Recreating the database by hand table by table is something I would prefer to avoid.
Thanks--
Al
In my case , this exact thing happened because I was taking the dump using windows Powershell , due to which other characters got included in the dump file.
Simply using command prompt to take the solved my problem.
I can only give you leads how to debug the problem, because the cause is not immediately obvious.
First, there should be a line close to the beginning of the dump file that sets client_encoding. The dump file should be in that encoding.
I can see two possibilities:
The file got mangled during transfer. To test for that, calculate a checksum for both files and compare.
Always use binary mode to transfer PostgreSQL dumps.
some editor or something else sneaked a BOM (byte order mark) into the file at the very beginning.
That's my prime suspect, since the problem is at line 1.
Use a hex editor or od (in Cygwin) to verify that. If this is the problem, simply replace the BOM with spaces.
I'm trying to copy a large data set from Postgresql to ScyllaDB, which is supposed to be compatible with Cassandra.
This is what I'm trying:
psql <db_name> -c "COPY (SELECT row_number() OVER () as id, * FROM ds.my_data_set LIMIT 20) TO stdout WITH (FORMAT csv, HEADER, DELIMITER ';');" \
| \
CQLSH_HOST=172.17.0.3 cqlsh -e 'COPY test.mytable (id, "Ist Einpöster", [....]) FROM STDIN WITH DELIMITER = $$;$$ AND HEADER = TRUE;'
I get an obscure error without a stack trace:
:1:'ascii' codec can't decode byte 0xc3 in position 9: ordinal not in range(128)
My data, and column names, including the ones already in the created table in ScyllaDB, contain values with German text. It's not ASCII, but I haven't found anywhere to set the encoding, and everywhere I looked it seemed to be using utf-8 already. I tried this as well, and saw in the vicinity of line 1135 that, and changed it in my local cqlsh (using vim $(which cqlsh)), but it had no effect.
I'm using cqlsh 5.0.1, installed using pip. (weirdly it was pip install cqlsh==5.0.4)
I also tried the cqlsh from the docker image that I used to install ScyllaDB, and it has the exact same error.
<Update>
As suggested, I piped the data to a file:
psql <db_name> -c "COPY (SELECT row_number() OVER (), * FROM ds.my_data_set ds) TO stdout WITH (FORMAT csv, HEADER);" | head -n 1 > test.csv
I thinned it down to the first row (CSV header). Piping it to cqlsh made it cry with the same error. Then, using python3.5 interactive shell, I did this:
>>> with open('test.csv', 'rb') as fp:
... data = fp.read()
>>> data
b'row_number,..... Ist Einp\xc3\xb6ster ........`
So there we are, \xc3 in the flesh. Is it UTF-8?
>>> data.decode('utf-8')
'row_number,....... Ist Einpöster ........`
Yes, it's utf-8. So how does the error happen?
>>> data.decode('ascii')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 336: ordinal not in range(128)
Same error text, so it's probably Python as well, but without a stack trace, I have no idea where this is happening, and default encodings are utf-8. I tried overriding the default with utf-8 but nothing changed. Still, somewhere, something is trying to decode a stream using ASCII.
This is the locale on the server/client:
LANG=
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8
Someone on Slack suggested this answer UnicodeDecodeError: 'ascii' codec can't decode byte 0xd1 in position 2: ordinal not in range(128)
Once I added the last 2 lines in cqlsh.py at the beginning, it got past the decoding issue, but the same column was reported as invalid with another error:
:1:Invalid column name Ist Einpöster
side note:
I lost interest in this test at this point, and I'm just trying to not have an unanswered question, so please excuse the wait time. As I was trying it out as an analytical engine, coupled with Spark, as a data source for Tableau, I found "better" alternatives, like Vertica and ClickHouse. "Better" because both of them have limitations.
</Update>
How can I complete this import?
What was it?
The query passed in as an argument, contained the column list, which contained that column with a non-ASCII character. At some point, cqlsh parsed those as ascii and not utf-8, which lead to this error.
How it was fixed?
First attempt was to add these 2 lines in cqlsh:
reload(sys)
sys.setdefaultencoding('utf-8')
but that still made the script unable to work with that column.
Second attempt was to simply pass the query from a file. If you can't, know that bash supports process substitution, so instead of this:
cqlsh -f path/to/query.cql
you can have
cqlsh -f <(echo "COPY .... FROM STDIN;")
And it's all great, except that it doesn't work either. cqlsh understands stdin as "interactive", from a prompt, and not piped in. The result is that it doesn't import anything. One could just create a file, and load it from the file, but that's an extra step that might take minutes or hours, depending on the data size.
Thankfully, POSIX systems have these virtual files like '/dev/stdin', so the above command is equivalent to this:
cqlsh -f <(echo "COPY .... FROM '/dev/stdin';")
except that cqlsh now thinks that you actually have a file, and it reads it like a file, so you can pipe your data and be happy.
This would probably work, but for some reason I got the last kick:
cqlsh.sql:2:Failed to import 15 rows: InvalidRequest - Error from server: code=2200 [Invalid query] message="Batch too large", will retry later, attempt 4 of 5
I think it's funny that 15 rows are too much for a distributed storage engine. And it's likely that it's again some limitation from the engine related to unicode and just a wrong error message. Or I'm wrong. Nevertheless, the initial question was answered, with some BIG help from the guys in Slack.
I don't see that you ever got an answer to this. UTF-8 should be the default.
Did you try --encoding?
Docs: https://docs.scylladb.com/getting-started/cqlsh/
If you didn't get an answer here, would you wish to ask it on our slack channel?
I would try to eliminate all the extra complexity you have in there first. Try to dump a few rows into a CSV, and then load it into Scylla using COPY
Update: utf8: Print invalid UTF-8 character position
Add new validate_with_error_position function
which returns -1 if data is a valid UTF-8 string
or otherwise a byte position of first invalid
character. The position is added to exception
messages of all UTF-8 parsing errors in Scylla.
validate_with_error_position is done in two
passes in order to preserve the same performance
in common case when the string is valid.
https://github.com/scylladb/scylla/commit/ffd8c8c505b92a71df7e34d5196c7545f11cb12f
I tried to import a simple json file using mongoimport and i get the following error
PER-MacBook-Pro:/AJ$ mongoimport --db test --collection samplePM --file /users/AJ/Documents/Development/ETLwork/Dummydata/Penguin_Players.json
2015-06-16T09:53:57.291-0400 connected to: localhost
2015-06-16T09:53:57.293-0400 Failed: error processing document #1: invalid character '\\' looking for beginning of object key string
2015-06-16T09:53:57.293-0400 imported 0 documents
Sample json file is as follows:
{
"position":"Right Wing",
"id":8465166,
"weight":200,
"height":"6' 0\"",
"imageUrl":"http://1.cdn.nhle.com/photos/mugs/8465166.jpg",
"birthplace":"Seria, BRN",
"age":37,
"name":"Craig Adams",
"birthdate":"April 26, 1977",
"number":27
},
{
"position":"Right Wing",
"id":8475761,
"weight":195,
"height":"6' 2\"",
"imageUrl":"http://1.cdn.nhle.com/photos/mugs/8475761.jpg",
"birthplace":"Gardena, CA, USA",
"age":23,
"name":"Beau Bennett",
"birthdate":"November 27, 1991",
"number":19
}
Am I doing something wrong here?
I was able to get away with using --jsonArray tag, giving the file full path, and modified it adding bracket at the beginning and at the end,
[{"my":"json","file":"imported"},{"my":"cool","file":"succeeded"}]
mongoimport --db myCoolDb --collection myCoolColl --file /path/to/my/imported/file.json --jsonArray
The comment about non "UTF-8" characters was helpful.
It seems like there is a problem with creating json documents using textedit in Mac. I could not find these non UTF-8 characters but i created the same file using vi test.json in mac shell. I pasted the contents, saved the file and used mongoimport. It works now.
Thanks
I got the same error while importing json data. Instead use the .bson data using mongorestore command.
mongorestore -d <db> -c <collection> <.bson-file>
Use --drop if you want to drop the existing data in the collection.
I was getting Failed: error processing document #112783: invalid character ',' looking for beginning of value because one of my objects was formatted improperly. Notice how "psychosurgery" is missing curly braces:
{
"word": "psychosurgeons",
"firstLetter": "p"
}
" psychosurgery",
{
"word": "psychosurgical",
"firstLetter": "p"
}
Since there are over 600,000 lines in the file I'm trying to import, this would have been tough to find manually.
So I ran the same mongoimport command with full verbosity (-vvvvv) enabled, and the script stopped right on the problematic object. See mongoimport --help for more info.
Hope this helps someone.
I got the same problem because I used texteditor on the mac. The solution was to convert the file to plain text. Make sure the extension ends in .json because texteditor wants to put .txt at the end.
Just open a text file, copy all the data to the newly created text file. While saving the text file select the option 'UTF-8' in the Encoding drop down and later change the text file to JSON or CSV by renaming.
Then import the file as usual as it is.
if you used mongoexport to download use mongoimport to upload
or if you use mongodump to download use mongorestore to upload
because i used to download with mongodump and tried to upload with mongoimport i was getting error processing document #1: invalid character '\u008c'error after i tried with mongorestore it was fine
I'm using WikiPrep to process the latest wiki dump enwiki-20121101-pages-articles.xml.bz2. Instead of "use Parse::MediaWikiDump;" I replaced that by "use MediaWiki::DumpFile::Compat;" and did the proper changes in the code. Then, I ran
perl wikiprep.pl -f enwiki-20121101-pages-articles.xml.bz2
I got an error
enwiki-20121101-pages-articles.xml.bz2:1: parser error : Document is empty
BZh91AY&SY±H¦ÂOÿ~Ð`ÿÿÿ¿ÿÿÿ¿ÿÿÿÿÿÿÿÿÿÿ½ÿýþdß8õEnÞ¶zëJ¨Eà®mEÓP|f÷Ô
^
I guess there are some non-utf8 characters contained in the dump. So I ran
iconv -f utf8 -t utf8 enwiki-20121101-pages-articles.xml.bz2
And indeed, I got some errors
BZh91AY&SYiconv: illegal input sequence at position 10
So, my question is what's the encoding format of wiki dump and if I wish to convert it to utf-8, what shall I do? Or how should modify wikiprep.pl to avoid such problems.
Many thanks
-- [solved] I should first unzip the file first.
You are running iconv on the compressed (bz2) version of the file, rather than the XML file itself. Uncompress it first.
(Posting borrible's answer so that this resolved question is not listed as unanswered.)
i Convert my database from this tutorial
http://en.gentoo-wiki.com/wiki/Convert_latin1_to_UTF-8_in_MySQL
but i didn't notice the arabic characters INSIDE the database is encrypted , like
اوÙاµ ®ØµØ… „Ù‡ Øكلق§Ø‡Ø°Ù…ا؄مشٳÙÙ‹ ÙÙ„...
through the php script connect with the database everything GOOD , but inside the database the arabic characters looks like that
i try to return the database to the old encoding which is WINDOWS-1256 using iconv by the following command
# iconv -f UTF-8 -t WINDOWS-1252 database.sql > database_1252.sql
i got this error
iconv: illegal input sequence at position
so i try to run the command again using -c option
# iconv -c -f UTF-8 -t WINDOWS-1252 database.sql > database_1252.sql
it's worked and i can see the arabic characters inside the database as well, but alot of characters missing , for example :
i would like to go shopping
after the converting
i would like to
i want to know how could i fix the Arabic Characters to be read as normal inside the database complete not missing anything
thanks
Wait wait .... you say your database was in WINDOWS-1256 (or WINDOWS-1252?) and you converted it based on tutorial latin1 -> utf8? No wonder the characters are malformed.
I wouldn't trust to the tutorial solution at all. I would recommend that you return to your former version of the database and use mysql alter table command to change the encoding.