Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 1 year ago.
Improve this question
I have a bunch of scripts collecting data from internet and local services, writing them to disk, scripts transforming the data and writing it into a database, scripts reading data from the database and generating new data, etc, written in bash, Python, SQL, ... (Linux).
Apart from a few time-triggered scripts, the glue between the scripts is currently me, running the scripts now and then in a particular order to update everything.
What is the simplest way to replace me by a tool that observes dependencies and triggers the next step as soon as the preconditions are met?
I've found many ETL and data warehousing tools, but these seem too heavy weight for my simple setting. I'd prefer a CLI solution with text-based configuration (maybe able to visualise the graph of dependencies). Any suggestions?
Try airflow: airflow.apache.org
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I connected to an analysis server using (32-bit) Ms Excel, but it crashed since the amount of data was big.
Then I used SQL Server Management Studio, even that did not worked for me since it seems like SSMS is an administrator tool and you need to have administrator rights for the DB you like to view (Which I do not and cannot have).
Please suggest me a software that can allow me to work with the large amount of data (without crashing) coming from an OLAP server?
SQL Server Management Studio is the right s/w. We need to use the query editor to access the db where we got no admin rights and then use MDX query to pull data.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
We've fed up with instability and unpredictability of ELK stack but still in love with the Kibana dashboards.
Hence I'm looking for some potential migration paths. MongoDB looks very promising: huge track record, lots of docs, ability to cope with json easily etc.
Is there some equivalent to Kibana working on top of MongoDB? Some web app which lets you easily run search queries over indexed data, make them into dashboards, add nice maps and diagrams etc.
I've looked into https://docs.mongodb.org/ecosystem/tools/administration-interfaces/ but this seems to be more about managing MongoDB itself rather than playing with data in it.
you could have a look at mongodb-compass click here
if you would want more, the new mongodb 3.2 has features to connect to any BI tool, like talend. Read more here
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
I am fairly new to golang, and trying to identify the best tools for the job. Currently I am evaluating the following packages:
https://github.com/mattes/migrate
https://github.com/DavidHuie/gomigrate
https://bitbucket.org/liamstask/goose/
I was wondering if anyone had any experience with these (or other packages) and could provide some comments.
We use mattes/migrate at work and are very happy with it. It works with plain SQL files, handles file naming by itself and can easily be automated via CLI. It doesn't do anything Go specific.
With gomigrate you need to create the files yourself and write code for executing the migrations.
Take a look at https://github.com/pressly/goose, a maintained fork of https://bitbucket.org/liamstask/goose/.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
I have a lot of logged data stored into a database by a data logger. Basically i have a lot of rows with a timestamp and some values. I want to store this data into a db that has performance and can scale on a multi node structure to support fault tolerance behaviour (and balance requests). Typically i use MySQL but i find its scalability not simple for this type of application. This time, i want offer other db scenarios.
So: Mongo, Redis, Couchdb?
Thanks all.
This is a hard question to answer and not something we can really give answers to on SO.
Redis is quick for getting the data in, but you can not query on the values of the keys so searching would be harder.
MongoDB & CouchDB would both work well as they are document stores and can be used to store any format for the logs.
There are other options. I know Cassandra is used a lot for this task, but there is also ElasticSearch as in (ElasticSearch, Log Stash, Kibana) which is a great solution for central logging.
In the end it probably down to what you want to do with the data.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
I really like Fabric for personal projects, but I am working in a Perl-oriented environment in my day-to-day job. I’d like to stay away from introducing another language to the workflow, so I’m wondering if there exists something similar to Fabric for doing automated deployments in Perl.
Have a look at Rex. However its more based on Capistrano than Fabric.
Also check out this older SO question Is there a Perl or Lua alternative to Capistrano?
As noted in this answer if you can log in with SSH, you can use Fabric, no matter the language used in the running app. Unless there is some Perl specific code that can't be run by issuing a command in a shell.