I generate a .sql file, on my laptop, that contains around 11 million insert statements into several tables.
Locally I run a MySQL database, into which I import this file. It takes a while, but it succeeds without any problems. The local MySQL version is:
mysql Ver 14.14 Distrib 5.6.16, for osx10.7 (x86_64) using EditLine wrapper
I want to import this file into a Google Cloud SQL instance. To do so, I first gzip the .sql file and upload it to a bucket in Google Cloud Storage.
Then I create a D0 pay-per-use instance (the least powerful / cheapest). I click 'Import' and enter the name of the file on cloud storage.
The import starts, but after a while (around a day) the import fails, stating: An unknown error occurred.
I tried this using a MySQL 5.5 and an experimental 5.6 instance, both fail at different inserts. (I can see what the latest successful insert was).
My problem is, I cannot find out what MySQL thinks is the problem.
How can I ask the Google developer console to show me a log? I tried on the Google APIs page which has a 'Logs' tab, but it gives me An error has occurred. Please retry later.
Maybe Google Cloud SQL has some limits on the insert statements that my local MySQL does not have?
One of the fields is a MEDIUMTEXT, which I believe can be larger than 65.536 bytes.
Any advice is appreciated.
---------- UPDATE -----------
I mailed with the cloud-sql team and they confirmed the problem was that the import timed out.
So indeed, 24 hours is the maximum time an import may take on Cloud SQL.
Solutions are: use a more powerful instance for the import (and use asynchronous replication), or split up the .sql in multiple parts.
Another approach is to use several values per insert statement, just make sure the line does not exceed 4MB. This is what the value of max_allowed_packet is on cloud sql. It speeds up the insert greatly.
In fact, this makes it possible to have the D0 instance import the file in a few hours, so I don't need to bump it to a more powerful one.
Related
I have the official Postgres ODBC drivers installed and am using IBM SPSS to try and load 4 million records from a MS SQL data source. I have the option set to bulk load via ODBC, but the performance is REALLY slow. When I go SQL-SQL the performance is good, when I go Postgres-Postgres the performance is good, but when I try and go SQL-Postgres it takes about 2.5 hours to load the records.
It's almost as if it's not bulk loading at all. Looking at the output it seems like it's reading the batched record count from the source very quickly (10,000 records), but the insert into the postgres side is taking forever. When I look at the record count every few seconds it jumps from 0 to 10,000 but takes minutes to get there, whereas it should be seconds.
Interestingly I downloaded a third party driver from DevArt and the load went from 2.5 hours to 9 minutes. Still not super quick, but much better. Either Postgres ODBC does not support bulk load (unlikely since postgres to postgres loads so quickly) or there's some configuration option at play in either the ODBC driver config or SPSS config.
Has anybody experienced this? I've been looking at options for the ODBC driver, but can't really see anything related to bulk loading.
IBM SPSS Statistics uses the IBM SPSS Data Access Pack (SDAP). These are 3rd party drivers from Progress/Data Direct. I can't speak to performance using other ODBC drivers. But if you are using the IBM SPSS Data Access Pack "IBM SPSS OEM 7.1 PosgreSQL Wire Protocol" ODBC driver, then there are resources for you
The latest Release of the IBM SPSS Data Access Pack (SDAP) is version 8.0. It is available from Passport Advantage (where you would have downloaded your IBM SPSS Statistics Software) as "IBM SPSS Data Access Pack V8.0 Multiplatform English (CC0NQEN )"
Once installed, see the Help. On Windows it will be here:
C:\ProgramData\Microsoft\Windows\Start Menu\Programs\IBM SPSS OEM Connect and ConnectXE for ODBC 8.0\
Looking for any thoughts on what is going on here:
Environment:
Java 11 GCP Function that copies data into table
Postgres 11 Cloud SQL using JDBC driver
(org.postgresql:postgresql:42.2.5)
No changes to any code or configuration in 2 weeks.
I'm connecting to the private SQL IP address, so similar to
jdbc:postgresql://10.90.123.4/...
I'm not requiring a SSL cert
There is Serverless VPC Access set up between the Function and SQL.
This is happening across two different GCP projects and SQL servers.
Prior to this Saturday (2/22), everything was working fine. We are using Postgres' CopyManager to load data into a table: copyManager.copyIn(sql, this.reader);
After 2/22, this started failing with "SSL error: DATA_LENGTH_TOO_LONG" as seen in the SQL server log. These failures are 100% consistent and still happen. I can see that SQL was restarted by Google a few hours before the issue started and I'm wondering if this is somehow related to whatever maintenance happened, SQL version upgrade? I'm unclear what version we had before Saturday, but it's now 11.6.
Interestingly enough, I can avoid the error if the file loaded into the table is under a certain size:
14,052 bytes (16 KB on disk): This fails every time.
14,051 bytes (16 KB on disk): This works every time.
I'd appreciate if someone from Google could confirm what took place during the maintenance window that might be causing this error. We are currently blocked by this as we load much larger datasets into the database than ~14 000 bytes.
FYI this was caused by a JDK issue with TLS v1.3, addressed in JDK 11.05. Google will likely upgrade the JDK used for the Cloud Functions JVMs from 11.04 to something newer next week. See https://bugs.openjdk.java.net/browse/JDK-8221253
I am new in the form and also new in postgresql.
Normally I use MySQL for my project but I’ve decided to start migrating towards postgresql for some valid reasons which I found in this database.
Expanding on the problem:
I need to analyze data via some mathematical formulas but in order to do this I need to get the data from the software via the API.
The software, the API and Postgresql v. 11.4 which I installed on a desktop are running on windows. So far I’ve managed to take the data via the API and import it into Postgreql.
My problem is how to transfer this data from
the local Postgresql (on the PC ) to a web Postgresql (installed in a Web server ) which is running Linux.
For example if I take the data every five minutes from software via API and put it in local db postgresql, how can I transfer this data (automatically if possible) to the db in the web server running Linux? I rejected a data dump because importing the whole db every time is not viable.
What I would like is to import only the five-minute data which gradually adds to the previous data.
I also rejected the idea of making a master - slave architecture
because not knowing the total amount of data, on the web server I have almost 2 Tb of hard disk while on the local pc I have only one hard disk that serves only to take the data and then to send it to the web server for the analysis.
Could someone please help by giving some good advice regarding how to achieve this objective?
Thanks to all for any answers.
I have a google storage bucket to which I want to export my google cloud sql database. I go to the Export tab, select a location on my bucket, give a filename, and choose the database I want to export. But I'm always greeted with Could not complete the operation.
It has been happening to me for the last 2 days. This flow had worked a couple of weeks ago, and I haven't tweaked around with the settings since then.
Is there a way I can get a more descriptive response, so I can identify the error? Also, how do I export my cloud sql db until then? Do I connect with the psql client and figure out a way from there?
I have a script that works correctly on a server machine when running within FileMaker Pro 13, but raises errors when run within FileMaker Server 13. Both are running under Windows. The portion that is raising the error is an Import Record script step that imports from one table within a file into another table within the same file.
The error returned is 100, "File is missing," so I'm wondering if this is something not supported when running a script within FileMaker Server. If that's the case, I'm thinking that perhaps exporting the records to a temporary file and importing from that might be a workaround, but before I start down that road, I want to check and see if I'm missing something.
The short answer is: no, Server Side Scripts can't import from a FileMaker file. From FM's help site: http://help.filemaker.com/app/answers/detail/a_id/7035/~/import%2Fexport-script-on-filemaker-server
Importing/exporting directly to and from another FileMaker Pro file is not supported via a FileMaker Server scheduled script.
Yes, exporting to an .xlsx, .csv or .txt file in the temporary directory is a common work-around. I use it frequently. If you want to avoid a temporary file, you can also grab all of the indices to a variable and loop through them, creating records. HyperLists come in handy for this.