Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
Is there an open source tool to extract data from a T24 system to a SQL database? Or maybe a connector for Talend?
Thank you
Short answer is no and no. There is no such open source tool and there is no widely available connector for Talend that I am aware of.
The main issue you have in translating jbase data to SQL is that jbase is not a relational database, and has multi valued fields or groups of fields that do not fit in a relational database, as they are not normalized. Now, in order to connect to jbase and extract info to store in a RDBMS, you would need some kind of api or connector that understands the JEDI. If you check link jbase you will find there is a odbc and a jdbc driver that you can use to connect. But I don't think they are provided free of charge. You have to own jbase database to gain access to them. Alternatively, you could use T24 tool, called DW.EXPORT, to extract information to text format such as csv and then find a way to import that csv to your RDBMS. This is the process used by T24 to export its data to its data warehouse system which uses a relational database.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 1 year ago.
Improve this question
I have a bunch of scripts collecting data from internet and local services, writing them to disk, scripts transforming the data and writing it into a database, scripts reading data from the database and generating new data, etc, written in bash, Python, SQL, ... (Linux).
Apart from a few time-triggered scripts, the glue between the scripts is currently me, running the scripts now and then in a particular order to update everything.
What is the simplest way to replace me by a tool that observes dependencies and triggers the next step as soon as the preconditions are met?
I've found many ETL and data warehousing tools, but these seem too heavy weight for my simple setting. I'd prefer a CLI solution with text-based configuration (maybe able to visualise the graph of dependencies). Any suggestions?
Try airflow: airflow.apache.org
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
I have been successfully used the Postgresql Debezium plugin for Kafka connect. This connector hooks up directly to the relational database's Write Ahead Log(WAL) which vastly improves performance, compared to the normal JDBC connector, which continuously polls the database via an sql query.
Is something similar possible with Redshift as the source, instead of Postgresql? I know there are major differences between Redshift and Postgresql, in that Redshift is columnar based, cluster-based, doesn't have secondary indexes, and it has different use cases. I could not find definitive information if Redshift has anything similar with a write ahead log or it uses a completely different approach.
Is there a write-ahead-log based approach to stream data changes from a redshift table directly into kafka, via debezium or some other way, or its not technically possible? If not, what about some alternative which achieves the same thing?
To answer your question in one line - No, it's not supported, and Im sure that AWS(or any modern DW) will never even think of enabling this feature.
Here are two strong reasons from my point of view:
RedShift itself getting the data from a different database(like your Postgres) and the main purpose is Read not write(so less concurrent writes).
For analytical purposes we bring all the data into a DW. From there it'll go to BI tool or any ML related works. But I never have seen any place where DW data will go to another database in Real or near realtime.
(You might already know this option)If you still need to do this, then you are getting the data from some sources, right? use that same source to send the data where you want to use from RedShift CDC.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 10 months ago.
Improve this question
I would like to know which is the best data catalog available which serves the following requirements.
Open source
Highly available behind load balancer
Describes the dataset
Can describes the data inside the datasets, Ability to have option to describe the fields inside the data
Data set searchable
I looked at ckan and was really impressed but it did not offer describing fields inside the datasets
Is there any other good tool for the same
You can actually extend the CKAN and mold it as per your needs. CKAN is a very versatile and flexible product.
e.g. for describing fields inside CKAN you can use
https://github.com/ckan/ckanext-scheming
or you can create your own extension by following the documentation.
https://docs.ckan.org/en/2.8/extensions/
Have you looked at Lyft's open source data catalog and discovery tool called "Amundsen"?
https://github.com/lyft/amundsen
https://eng.lyft.com/open-sourcing-amundsen-a-data-discovery-and-metadata-platform-2282bb436234
I have used the data catalog part of the engrafo-solution (engrafo.eu)
Open source (free plan for 10 users)
Highly available behind load balancer (?)
Describes the dataset (yes)
Can describes the data inside the datasets, Ability to have option to describe the fields inside the data (yes)
Data set searchable (yes)
a demo on the data catalog
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
Realm Database or MySQL or MongoDB or the standard Firebase.
I am curious why would anyone choose one over another. I am in a group project and one of the members in the group suggested that we should use Realm Database over Firebase. He tried to explain to me the reason but I was not able to comprehend.
What has your experience been like? Is one database more user-friendly over another?
Firebase seems to have very nice documentation.
Which one would you guys suggest?
Those are three different things.
Realm is a database for storing content locally on a device.
MySQL is usually used together with a web server. If you have a server and need to store data, you can use MySQL.
Firebase is Google's alternative for building your own server. It gives you tools that allow you to avoid having to build your own web server, such as Firestore, which lets you do CRUD operations directly from devices without needing to send that information through a server first.
If you works on small project you must going with MYSQL database. Its very simple and easy to understand. But if your project are big like that organisation type projects I recommend you to going with MongoDB.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
I have a lot of logged data stored into a database by a data logger. Basically i have a lot of rows with a timestamp and some values. I want to store this data into a db that has performance and can scale on a multi node structure to support fault tolerance behaviour (and balance requests). Typically i use MySQL but i find its scalability not simple for this type of application. This time, i want offer other db scenarios.
So: Mongo, Redis, Couchdb?
Thanks all.
This is a hard question to answer and not something we can really give answers to on SO.
Redis is quick for getting the data in, but you can not query on the values of the keys so searching would be harder.
MongoDB & CouchDB would both work well as they are document stores and can be used to store any format for the logs.
There are other options. I know Cassandra is used a lot for this task, but there is also ElasticSearch as in (ElasticSearch, Log Stash, Kibana) which is a great solution for central logging.
In the end it probably down to what you want to do with the data.