I am using Azure PostgreSQL, in my web project, I save files(such as images,.csv) in table as bytea data type.
Now I can get files from Azure PostgreSQL successfully, but the performance is not so good. it will take more than 20s to retrieve files if I request for multiple files, even sometimes I got timeout error from my web server. So is there anyone could give me some advice to solve this? Any suggestion is appreciated.
As #eshirvana said, saving files in databases is not a good idea. As we know,the performance that querying blobs or files in SQL is awful and it also greatly cost DB server memory. Saving URLs in DB only is recommended.
If you are using Azure Cloud, you can save your files in Azure blob storage. And you can save your file storage blob URL in your DB table.
Your client could get the file URL from the webserver by SQL query and access files in Azure storage by the file URL. This way could ease the IO and memory pressure of your web server and greatly improve the query performance.
What's more, creating your SQL service in the region that you are in or near could also reduce network latency. You can see all the Azure datacenter regions here.
Related
I would like to create an automated data pulling from our PostgreSQL database to a Google sheet. I've tried JDBC service, but it doesn't work, maybe incorrect variables/config. Does anyone already try doing this? I'd also like to schedule the extraction every hour.
According the the documentation, only Google Cloud SQL MySQL, MySQL, Microsoft SQL Server, and Oracle databases are supported by Apps Script's JDBC. You may have to either move to a new database or develop your own API services to handle the connection.
As for scheduling by the hour, you can use Apps Script's installable triggers.
I am trying to follow quickstart to setup SQL Server (not LocalDb version of SQLServer that comes with Visual Studio) as my data store. Looks like that two databases will be needed - one for configuration and the other for operation. But my problem is that I couldn't figure out what db names I should use. I created two databases using names I came up with and ran the scripts I downloaded from quickstart to create all the tables. Now, when I try to make connection, I think I will need to specify db names in my connection string, don't I? What should I use to replace the original connection string provide by quickstart - "Data Source=(LocalDb)\MSSQLLocalDB;database=IdentityServer4.Quickstart.EntityFramework-4.0.0;trusted_connection=yes;" ?
You can have one database for both. But in general I would keep the configuration part in memory if the number of clients is small. Why spend hours keeping the config in a database for just a few clients and resources?
Better to just keep the users and persisted grants in a database.
I am new in the form and also new in postgresql.
Normally I use MySQL for my project but I’ve decided to start migrating towards postgresql for some valid reasons which I found in this database.
Expanding on the problem:
I need to analyze data via some mathematical formulas but in order to do this I need to get the data from the software via the API.
The software, the API and Postgresql v. 11.4 which I installed on a desktop are running on windows. So far I’ve managed to take the data via the API and import it into Postgreql.
My problem is how to transfer this data from
the local Postgresql (on the PC ) to a web Postgresql (installed in a Web server ) which is running Linux.
For example if I take the data every five minutes from software via API and put it in local db postgresql, how can I transfer this data (automatically if possible) to the db in the web server running Linux? I rejected a data dump because importing the whole db every time is not viable.
What I would like is to import only the five-minute data which gradually adds to the previous data.
I also rejected the idea of making a master - slave architecture
because not knowing the total amount of data, on the web server I have almost 2 Tb of hard disk while on the local pc I have only one hard disk that serves only to take the data and then to send it to the web server for the analysis.
Could someone please help by giving some good advice regarding how to achieve this objective?
Thanks to all for any answers.
After I proceed to restore database from automated backup file generated on Mar 13, 2019, the SQL instance stuck in this state forever:"
Restoring from backup. This may take a few minutes. While this operation is running, you may continue to view information about the instance."
The database size is very small, less than 1MB.
For future users that experience problems like this is in the future, here is how you can handle it:
If you have a Google Cloud support package, file a support ticket directly with support for the quickest response.
Otherwise please file a private GCP issue describing the problem, remembering to include the project id and instance name.
However - Cloud SQL instances are monitored for stuck states like this, so often the issue will resolve itself within a few hours.
I'm building a customer management system using Rails that requires CSV files containing customer information to be imported into/diffed with a Postgres database. I'm hosting the application on Heroku. I moved the database to the background with Sidekiq but need advice on where to upload the file to in the first place for importing. Is hosting the file on S3 really the best solution or is there a simpler solution without using a third party storage service? The application will be used daily but up 10 employees and the larges CSV file being upload is around 100,000 rows.
Thanks.
Yes, I do think S3 is the best solution
We faced same problem at Storemapper (we use Resque instead of Sidekiq, but that's not a problem). The limiting factor here is the Heroku request timeout. You only have 30s to finish your upload to Heroku, which put hard limit on how big your csv can be. This is where S3 come. Basically what we do is:
User upload csv directly to S3 via javascript, bypassing our app server on Heroku.
Once the upload complete, the javascript makes a request to app server that will launch background worker, telling the worker where the file is at S3
The worker download the csv from s3, then process it as necessary
I found carrierwave_direct gem to be very helpful for step 1 and 2. For step 3, I use smarter_csv gem. Checkout our complete story here:
https://tylertringas.com/very-large-csv-import-in-rails-on-heroku/