fluentD plugin is used to insert csv to redshift table, but i dont see update is possible with this? If yes, could you please help me point the steps/configuration.
I have already asked this question here, and waiting for the response.
If there is no fluentD support then is it good idea to write custom notification listener service where the service will listen for sql update request events in the queue and update the same using jdbc batch update ..?
Please suggest.
https://forums.aws.amazon.com/message.jspa?messageID=816020#816020
Related
I wanted to check if there is a way to copy Druid datasource data (segments) from one server to another. Our ask is to load new data to prod Druid (using SQL queries), and copy the same data to qa Druid server. We are using hive druid storage handler to load the data, and HDFS as deep storage.
I read Druid documentations, but did not find any useful information.
There is currently no way to do this cleanly in druid.
If you really need this feature, please request this by creating a github ticket on : https://github.com/apache/druid/issues .
The workaround way is documented here : https://docs.imply.io/latest/migrate/#the-new-cluster-has-no-data-and-can-access-the-old-clusters-deep-storage
Full disclosure: I work for imply.
I have setup a postgres database inside the Kubernetes cluster and now I would like to backup the database and don't know how it is possible.
Can anyone help me to get it done ?
Thanks
Sure you can backup your database. You can setup a CronJob to periodically run pg_dump and upload the dumped data into a cloud bucket. Check this blog post for more details.
However, I recommend you to use a Kubernetes native disaster recovery tool like Velero, Stash, Portworx PX-Backup, etc.
If you use an operator to manage your database such as zalando/postgres-operator, CrunchyData, KubeDB, etc. You can use their native database backup functionality.
Disclosure: I am one of the developer of Stash tool.
Is there any way of getting merged logs from more than one deployments on Kybernetes? What's the best way of logging events for all deployments?
Look for Elasticsearch , Logstash and Kibana (ELK) stack with Filebeats or FluentD to ship log data from individual deployments/pods onto your Elasticsearch DB. Once data is in your DB , use Kibana to visualize and search your merged logs. Logstash can be used to modify your data inflight. A simple google search should yield you lot of resources on doing the same.
I am currently working in a project where we are using CDC Transaction stage which will automatically capture the change data from source.This data is used to create xml which will inturn pass to MQ where MDM will pickup this from MQ.
Now I got a requirement for automating the deployment process in datastage.Currently I am looking for an architecture how to achieve this. So when we did automation of deployment process, there should be some mechanism where I need to stop the CDC transaction process, stop the MQ process and deploy the jobs.Once the deployment is successfull I need to restart the CDC transaction process and MQ process from the point where it stopped. Also I need a rollback mechanis in case if the deployment went wrong. Please let me know all your thought process on this, so that I can create a good solution
NewRelic gives nice database analyses, however it seems to track only the web app's transactions.
I have independently managed servers which query and load my Heroku postgresql database. Is there a way I can get diagnostics and analysis of the database activity so that it will include all connections to it?
New Relic application monitoring will only collect data on database queries that are part of a web transaction or background task that is being monitored. If you're using one of New Relic's supported languages to query your database, you may be able to track that code as a background task (see https://newrelic.com/docs/features/monitoring-background-processes). If you would like a general monitoring plugin for your postgresql database, you could check out the postgresql plugin for New Relic (created and supported by Boundless): http://newrelic.com/plugins/boundless/109.
You should also try Heroku PG Extras: https://github.com/heroku/heroku-pg-extras. That will give info about cache hit, indexes, long queries, etc.