Can you please provide an example of flyway.conf settings for Redshift?
I tried using:
flyway.url=jdbc:Redshift://name.redshift.amazonaws.com:5439/DBName
flyway.user=user
flyway.password=pass
but that produced this error:
ERROR: Unable to autodetect JDBC driver for url: jdbc:Redshift:
There are many issues here:
redshift should be lower case in the jdbc url
You also need to put the Redshift JDBC driver on the classpath (/drivers directory for Flyway command-line)
additionally you need to set flyway.driver to the AWS redshift driver class name (Flyway defaults to the standard PG driver: http://flywaydb.org/documentation/database/redshift.html)
Related
I need to poll DB2 database table until a record is created and once it is successful,i need to trigger a task in dag.
For this I have to do below:
Create a DB2 connection in airflow
Use SQL sensor to poll db table using the connection created in above step.
But I don't see the connection type option for DB2 (screenshot attached). Do I have to install something? Please help.
PS: I am new to Airflow.
There is no DB2 specific connection.
For general connections you can just use the Generic connection:
For older Airflow versions that doesn't have Generic connection you can use any other connection type (HTTP/MySql for example). It doesn't really matter. Airflow looks for connection by the connection id. The type is almost meaningless in that perspective.
Yes the airflow connection does not appear to be a specific connection for db2, so you have to choose generic in connection Type.
I also recommend to install the security mechanism for connection.
the next package: pip install airflow-provider-db2
I have tried to connect to DB2 successfully on Airflow 2.2.5.
Airflow doesn't has specific provider for DB2. You need to use JDBC to connect.
So you need to install some libraries of Airflow and download DB2 JDBC driver to fulfill this task.
List of all libs and provider need to install below:
Apache-airflow-provider-jdbc/Version:2.1.3
JayDeBeApi/Version:latest
JPype1/Version:latest
You also need to download DB2 JDBC drivers. This driver include some packages and license file like:
db2jcc4.jar
sqlj4.zip
jdbc4_LI_en
Because it will use JDBC to connect,openJDK also need to install on your docker.
Setting environment variables JAVA_HOME, PATH and CLASSPATH are needed.
All of above actions can be done on your docker file.
When you have completed to build your image, You can start your airflow to set connection.
Connection URL syntax: jdbc:db2://Host:Port/Database
You can replace HOST,PORT and Database.
Driver Path: You can input where your class file(db2jcc4.jar) located.
Hope these tips can help you. I am New to Airflow too.
I'm trying to setup an easily-replicable (or even manual and I do it once a month or so) process for moving data from a large Azure PostgreSQL database to a more manageable Azure SQL database for end users that are most familiar with SQL Server. I've successfully connected to the PostgreSQL database via PGAdmin, so I know all my connection string info.
I started by installing the latest ODBC driver from here.
I then used a connection string which was given to me from the Azure portal, filled in the proper database name and password, and attempted to use the following drivers:
PostgreSQL ODBC Driver(UNICODE)
PostgreSQL ODBC Driver(ANSI)
I am getting the following error with either of them:
ERROR [IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified
What step am I missing in this process? Or how best can I troubleshoot this?
After more research, I attempted to add the ODBC driver here:
And got the following error (I'm not sure why Tableau is relevant to this?):
Thank you.
I had configured the JDBC connection configuration in the pipeline.
and when the application executes i get the following error on the logs.
"java.sql.SQLSyntaxErrorException: Table 'databaseName.aim_table' doesn't exist"
The databaseName is not what I have set.
I have tried many times. it shows the same message that could not find the table in different database, and the question is all the db occurred in the sdc.log are that I had never configured ,and the correct database is never used ,so I want to know how could it find the wrong db and I had checked before start the pipeline and it shows successful:
Do you have anything set in the Schema Name configuration for JDBC
Producer? This should be blank for MySQL, since you're setting the
database/schema name in the connect URL.
Check that your MySQL driver matches the server. In particular, using
the current version 8.0.x JDBC driver with a 5.x.x server seems to
result in this problem. Download the older 5.1.x driver (currently
5.1.46) and it should work.
refer this
This problem is indeed caused by the wrong version of the driver package. I found the correct driver package and successfully wrote the data to the target table. add aonther point, I have set the SCHEMA NAME to blank and defined the database name in the connect URL for mysql.
My English is not good. Please forgive me.
Problem:
Can't connect to Redshift using Zeppelin. When I try to connect I get the following error:
java.sql.SQLException: No suitable driver found for jdbc:redshiftl://redshift_db_address.redshift.amazonaws.com:5439/db_name
Background:
I am using the dylanmei/zeppelin docker image.
I created the Redshift interpreter according to the instructions here: https://zeppelin.apache.org/docs/0.7.0/interpreter/jdbc.html#redshift
I downloaded this JDBC driver: https://s3.amazonaws.com/redshift-downloads/drivers/RedshiftJDBC42-1.2.1.1001.jar
--> installed to /usr/zeppelin/interpreter/jdbc/RedshiftJDBC42-1.2.1.1001.jar
Set the artifact path to: com.amazonaws:aws-java-sdk-redshift:1.11.51
Set driver.default to: com.amazon.redshift.jdbc42.Driver
in icCube 5.1 there is no Redshift as list of supported JDBC connections.
How to create a data source in icCube on Amazon Redshift ?
A first solution is using the Postgres jdbc driver. Redshift is based on Postgres so it also works (for how long is a good question).
The second is a bit more complicated as you need to add Reshift jdbc driver to icCube. First download jdbc driver from amazon from here, after follow this instructions to add a library to icCube.
Once done you've to configure a new data-source :