How to import MySQL metadata to Apache Atlas using sqoop or Spark,
need to show actual RDBMS data types and its column in Apache Atlas
i tried MySQL to hive using sqoop
MySQL to Hive using Spark
I don't want to show Mysql to Hive, its only showing hive datatypes
is there any way to connect MySQL to Apache Atlas like JDBC connection, etc.
Related
I am trying to archive data from mongodb in hive using Spark. Intention is to connect to mongodb, read data and dump it as hive table. Problem is that both mongodb and hive are Kerberized and have different principals. I am not able to connect to mongo from Spark after successfully connecting to Hadoop. Having a cloudera cluster.
Did a kinit with hdfs credentials. Executed Spark submit and passed mongo credentials as Java options. I get an error during mongo authentication from executor node, gssapi failed.
I'm trying to connect to Denodo using Python SQLAlchemy, and create a DataFrame from a table. My environment is Cloudera Data Science Workbench. Can I use psycopg2 and the PostgreSQL connection string?
This won't work as Denodo dabbled with PostgreSQL schemas and is thus not compatible with it (or any other dialect). I use engine = sqlalchemy.create_engine("mssql+pyodbc://DenodoODBC"). DenodoODBC settings are taken from my ODBC Data Sources.
EDIT
In April, Denodo added a python connection guide:
By default, PostgreSQL JDBC reads all records into momery and then starts to process them.
If it is MySQL JDBC and MySQL database, I know how to do the reading and processing simultaneously. However, I don't know how to do it in PostgreSQL JDBC and PostgreSQL database.
I am trying to connect Vertica using Spring batch but getting excception from Vertica DB.
Can I connect to Vertica DB using Spring batch. Tried to search on on net but didn't find any example.
You need to initialize your database using jdbc:initialize-database: use your custim vertica db script contains springbatch metadata tables creation and all should be fine.
I'm using Hadoop to store the data of our application. How can I synchronize data between PostgreSql and Hadoop? I 'm using SymmetricDS as the replication tool.
If hadoop only copies data from PostgreSQL and no updates are done on the hadoop site, try using sqoop - simple database into hadoop import tool.
If you want to continue to use SymmetricDS you can implement an IDatabaseWriter. Here is an example of writing to MongoDB. https://github.com/JumpMind/symmetric-mongo