I want to upload data into Google cloudsql instance from csv file stored on GCS bucket. I am using postgresql database and to import csv files using gcloud sql import csv command in my shell script. There is an issue as some csv files contains " characters and to ignore that I want to add " as Escape character but gcloud sql import csv command doesn't have any fields to add escape character. Does anybody have anyidea on that?
As per documentation, to import CSV files to Cloud SQL PostgreSQL, the CSV file has to follow a specific format.
We can also see that for the command you're using there isn't any parameter that fits your requirements.
Instead, as an alternative, I'd use some sort of lint or text editor and try to massively remove the characters that conflict you, if possible.
I'm new to pgAdmin and GIS DB in general. I want to upload a CSV file to pgAdmin v4.1 and I'm trying to understand the logic to do so. I am able to do this by creating a new table under the desired DB and then manually defined the column (name, type etc.), only then I am able to load the CSV into pgAdmin using the GUI. This seems a bit cumbersome way to import a CSV file, because let's say I have a CSV file with 200 columns, it is not practical to define them all manually, and there must be a way to tell pgAdmin: this is the CSV file, now get the columns by yourself and get (or at least assume) the columns type, ad create a new table, much similar to how pandas reads CSV in python. As I'm new to this topic, please elaborate your answer\comment as much as possible.
NO: Unfortunately, we can only import CSV after the table is created.
YES: There is no GUI method, but:
There is a utility called pgFutter which will do exactly what you want. This is a command line utility. Here are the binaries.
You can write a function that does that. Here is an example.
I would look into using GDAL to upload your CSV file into postgis.
I used this recently to do a similar job.
ogr2ogr -f "PostgreSQL" -lco GEOMETRY_NAME=geometry -lco FID=gid PG:"host=127.0.0.1 user=username dbname=dbname password=********" postgres.vrt -nln th_new_data_2019 -t_srs EPSG:27700
Code used to upload a csv to postgis and transform the coordinate system.
-f = file format name
output file format name, some possible values are:
-f "ESRI Shapefile"
-f "TIGER"
-f "MapInfo File"
-f "GML"
-f "PostgreSQL
-lco = NAME=VALUE:
Layer creation option (format specific)
-nln name:
Assign an alternate name to the new layer
-t_srs srs_def:
target spatial reference set. The coordinate systems that can be passed are anything supported by the OGRSpatialReference.SetFromUserInput() call, which includes EPSG PCS and GCSes (i.e. EPSG:4296), PROJ.4 declarations (as above), or the name of a .prj file containing well known text.
The best and simplest guide for installing GDAL that I have used is :
https://sandbox.idre.ucla.edu/sandbox/tutorials/installing-gdal-for-windows
I am trying to insert this database into MongoDB using Studio 3T. I can import the bson without any issues (the countries and timezones) by selecting the parent folder and using BSON - mongodump folder option. However I cannot figure out how to import the split cities dataset.
I have tried all the options available on Studio3T and attempted to change the filename to gz however it always fails to import. I don't know what file format the cities are in.
Normally I do not have any issue importing but I cannot figure out how to do this. How would I achieve this?
The source DB is here https://github.com/VinceG/world-geo-data
This data is nothing but a big .bson file that has been gzipped up and split into various parts. I was not able to import the .bson file successfully. However, I could unzip the file at least without an error using the following commands and GZip for Windows
copy /b city_split_aa+city_split_ab+city_split_ac+city_split_ad+city_split_ae cities.bson.gz
gzip -d cities.bson
I tried to import a simple json file using mongoimport and i get the following error
PER-MacBook-Pro:/AJ$ mongoimport --db test --collection samplePM --file /users/AJ/Documents/Development/ETLwork/Dummydata/Penguin_Players.json
2015-06-16T09:53:57.291-0400 connected to: localhost
2015-06-16T09:53:57.293-0400 Failed: error processing document #1: invalid character '\\' looking for beginning of object key string
2015-06-16T09:53:57.293-0400 imported 0 documents
Sample json file is as follows:
{
"position":"Right Wing",
"id":8465166,
"weight":200,
"height":"6' 0\"",
"imageUrl":"http://1.cdn.nhle.com/photos/mugs/8465166.jpg",
"birthplace":"Seria, BRN",
"age":37,
"name":"Craig Adams",
"birthdate":"April 26, 1977",
"number":27
},
{
"position":"Right Wing",
"id":8475761,
"weight":195,
"height":"6' 2\"",
"imageUrl":"http://1.cdn.nhle.com/photos/mugs/8475761.jpg",
"birthplace":"Gardena, CA, USA",
"age":23,
"name":"Beau Bennett",
"birthdate":"November 27, 1991",
"number":19
}
Am I doing something wrong here?
I was able to get away with using --jsonArray tag, giving the file full path, and modified it adding bracket at the beginning and at the end,
[{"my":"json","file":"imported"},{"my":"cool","file":"succeeded"}]
mongoimport --db myCoolDb --collection myCoolColl --file /path/to/my/imported/file.json --jsonArray
The comment about non "UTF-8" characters was helpful.
It seems like there is a problem with creating json documents using textedit in Mac. I could not find these non UTF-8 characters but i created the same file using vi test.json in mac shell. I pasted the contents, saved the file and used mongoimport. It works now.
Thanks
I got the same error while importing json data. Instead use the .bson data using mongorestore command.
mongorestore -d <db> -c <collection> <.bson-file>
Use --drop if you want to drop the existing data in the collection.
I was getting Failed: error processing document #112783: invalid character ',' looking for beginning of value because one of my objects was formatted improperly. Notice how "psychosurgery" is missing curly braces:
{
"word": "psychosurgeons",
"firstLetter": "p"
}
" psychosurgery",
{
"word": "psychosurgical",
"firstLetter": "p"
}
Since there are over 600,000 lines in the file I'm trying to import, this would have been tough to find manually.
So I ran the same mongoimport command with full verbosity (-vvvvv) enabled, and the script stopped right on the problematic object. See mongoimport --help for more info.
Hope this helps someone.
I got the same problem because I used texteditor on the mac. The solution was to convert the file to plain text. Make sure the extension ends in .json because texteditor wants to put .txt at the end.
Just open a text file, copy all the data to the newly created text file. While saving the text file select the option 'UTF-8' in the Encoding drop down and later change the text file to JSON or CSV by renaming.
Then import the file as usual as it is.
if you used mongoexport to download use mongoimport to upload
or if you use mongodump to download use mongorestore to upload
because i used to download with mongodump and tried to upload with mongoimport i was getting error processing document #1: invalid character '\u008c'error after i tried with mongorestore it was fine
I am trying to insert a huge(~831M) file into mongo collection using mongoimport
/Library/mongodb/bin/mongoimport --port 12345 -d staging -c collection < out.all.1
and see some errors like
exception:Failure parsing JSON string near: , 'Custome
and there are instances where I found some weird characters
'CustomerCity': u'Wall \xa0'
'CustomerCity': u'La Ca\xc3\xb1ada Flintridge'
'CustomerCity': u'La Ca\xf1ada Flintridge'
How do I resolve these issues?
Thank you
I struck a similar problem where mongoimport gave errors about non-UTF8 characters in a flat file I'd asked it to import. This google groups thread led me to try putting my source data file through iconv on the unix command line to 'correct' non-UTF-8 characters, thus:
iconv -f ISO-8859-1 -t UTF-8 inputfile.txt > outputfile.txt
That solved the issue for me. I wonder if that approach might help you? While the error you're seeing is different, it's the odd characters that are messing up the JSON parsing, no?
One wonders, however, how those odd characters are ending up in your output data if you're generating it yourself. Perhaps you could filter in the code that generates the output?