I want to do some data analysis from NYCopendata. The file is ~900 MB. So I am using postgresql database to store this file. I am using pgadmin4 but could not figure out how to directly store the csv in postgresl without first downloading in my machine. Any help is greatly appreciated.
Thanks.
You can use:
pgAdmin to upload a CSV file from import/export dialog
https://www.pgadmin.org/docs/pgadmin4/4.21/import_export_data.html
COPY statement on the database server
\copy command from psql on any client
Related
I have been trying for days to copy a large CSV file to a table in PostgreSQL I am using PGadmin4 to access the database. I have a file on my system the file is 10 GB so I am getting starting error when trying to upload it via UI or \copy command.
When talking about 10 GB CSV file, then you may use as well different options
I believe \copy should work, you did not provide any more information about the issue
I'd personally use the AWS Glue - an ETL service which could read from an S3 file
I have taken full backup of postgresql database which consists of 100 databases. The backup format is .sql (eg pg_dumpall.exe -U postgres > D:\Backup\fullbkp.sql) now one of my database got crashed and I want to extract this file to get that database backup only for restoration.
I have searched a lot but couldn't find any way to decompress so that I can get that particular database from full backup file.
Please suggest !!!!
Regards
Sadam
Such a backup is not compressed. Also, it contains a backup of all databases in th cluster, and there is no easy way to extract a single database.
Create a new PostgreSQL cluster with initdb, restore the dump there using psql, then use pg_dump to extract the single database you need.
I'm trying to export my database created in Google Cloud Sql and import it into a new external server.
I tried to create a sql backup through the google console, downloaded it and copied it to the new server via filezilla and then launched the following command:
psql -U postgres -d ciclods-db -1 -f Backup-db_Cloud_SQL_Export_2019-03-23\ \(17_01_19\)
but i get this output:
ERROR: role "cloudsqladmin" does not exist
REVOKE
ERROR: role
"cloudsqlsuperuser" does not exist GRANT
what is the right procedure to follow in these cases?
I have resolved the same problem by locating and deleting the two lines from the exported sql file with "cloudsqladmin". My app does not use it anyway.
to do this task you can follow the official GCP guide about How to export data from Cloud SQL[1] in that document they give you the option to export the data into a dump file or csv files which can be used for other tools.
https://cloud.google.com/sql/docs/mysql/import-export/exporting
In order to create the export file, you have to do it from a command line and use additional flags. As per documentation‘s “Exporting data to a SQL dump file”, there is a section on Exporting data from an externally-managed database server.
As well you can find there the option to export the data into a CSV file.
I have remote postgresql database and a local csv file which I need to add to the database. I'm trying to do it with PyCharm.
Thus, I'm trying to copy data from a local file to a remote database.
If the database local is, then this command works:
COPY master_relationsextra(code, serial_number, member_type, characteristic, price_list)
FROM '/Users/name/Desktop/AUTOI.csv' with CSV HEADER delimiter ';' encoding 'ISO-8859-1';
But for the remote database it doesn't working.
Any advice how can I do that?
I'm using PyCharm thus I did with PyCharm's help. All queries and commands did PyCharm for me. I did it as follows:
I connected to the remote database from PyCharm database pane
Right click on table and then import from file
Choose all rules and import
That did the trick for me.
I've been using JDBC with a local Postgres DB to copy data from CSV files into the database with the Postgres COPY command. I use Java to parse the existing CSV file into a CSV format matches the tables in the DB. I then save this parsed CSV to my local disk. I then have JDBC execute a COPY command using the parsed CSV to my local DB. Everything works as expected.
Now I'm trying to perform the same process on a Postgres DB on a remote server using JDBC. However, when JDBC tries to execute the COPY I get
org.postgresql.util.PSQLException: ERROR: could not open file "C:\data\datafile.csv" for reading: No such file or directory
Am I correct in understanding that the COPY command tells the DB to look locally for this file? I.E. the remote server is looking on its C: drive (doesn't exist).
If this is the case, is there anyway to indicate to the copy command to look on my computer rather than "locally" on the remote machine? Reading through the copy documentation I didn't find anything that indicated this functionality.
If the functionality doesn't exist, I'm thinking of just populating the whole database locally then copying to database to the remote server but just wanted to check that I wasn't missing anything.
Thanks for your help.
Create your sql file as follows on your client machine
COPY testtable (column1, c2, c3) FROM STDIN WITH CSV;
1,2,3
4,5,6
\.
Then execute, on your client
psql -U postgres -f /mylocaldrive/copy.sql -h remoteserver.example.com
If you use JDBC, the best solution for you is to use the PostgreSQL COPY API
http://jdbc.postgresql.org/documentation/publicapi/org/postgresql/copy/CopyManager.html
Otherwise (as already noted by others) you can use \copy from psql which allows accessing the local files on the client machine
To my knowledge, the COPY command can only be used to read locally (either from stdin or from file) from the machine where the database is running.
You could make a shell script where you run you the java conversion, then use psql to do a \copy command, which reads from a file on the client machine.