I need to poll DB2 database table until a record is created and once it is successful,i need to trigger a task in dag.
For this I have to do below:
Create a DB2 connection in airflow
Use SQL sensor to poll db table using the connection created in above step.
But I don't see the connection type option for DB2 (screenshot attached). Do I have to install something? Please help.
PS: I am new to Airflow.
There is no DB2 specific connection.
For general connections you can just use the Generic connection:
For older Airflow versions that doesn't have Generic connection you can use any other connection type (HTTP/MySql for example). It doesn't really matter. Airflow looks for connection by the connection id. The type is almost meaningless in that perspective.
Yes the airflow connection does not appear to be a specific connection for db2, so you have to choose generic in connection Type.
I also recommend to install the security mechanism for connection.
the next package: pip install airflow-provider-db2
I have tried to connect to DB2 successfully on Airflow 2.2.5.
Airflow doesn't has specific provider for DB2. You need to use JDBC to connect.
So you need to install some libraries of Airflow and download DB2 JDBC driver to fulfill this task.
List of all libs and provider need to install below:
Apache-airflow-provider-jdbc/Version:2.1.3
JayDeBeApi/Version:latest
JPype1/Version:latest
You also need to download DB2 JDBC drivers. This driver include some packages and license file like:
db2jcc4.jar
sqlj4.zip
jdbc4_LI_en
Because it will use JDBC to connect,openJDK also need to install on your docker.
Setting environment variables JAVA_HOME, PATH and CLASSPATH are needed.
All of above actions can be done on your docker file.
When you have completed to build your image, You can start your airflow to set connection.
Connection URL syntax: jdbc:db2://Host:Port/Database
You can replace HOST,PORT and Database.
Driver Path: You can input where your class file(db2jcc4.jar) located.
Hope these tips can help you. I am New to Airflow too.
Related
Objective: Set up a connection with PostgreSQL db in PDI. I'm a noob in all this stuff, so each step forward requires long resolution time.
I'm trying to set up the connection to a PostgreSQL database in order to prepare some queries for an ETL process in PDI. But there's no way to succeed with the connection.
I've tried with PostgreSQL connection type and with Generic database connection type and the error is always related with the drivers.
The version of the PostgreSQL db I'm trying to connect is 9.5.25 and I'm working in Ubuntu 20.04 and Java 8 as demands Pentaho Data Integration.
Below the error message after failed connection:
"Connection failed. Verify all connection parameters and confirm that the appropriate driver is installed. Connection to [IP]:[Port] refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections."
I am literally in despair, don't know what I'm doing wrong. Will be very grateful with anyone who can help me.
You need to add the jdbc driver to connect to postgresql DB in the pdi-installation/lib directory, you just put the *.jar file there and restart spoon to have the library available.
That driver can be downloaded from postgresql downloads. The Community Edition of PDI only comes with a small number of jdbc drivers, for each type of database you need to connect to, you should check if the jdbc driver is available in that directory, and if it's up to date or if you need a more recent version of the driver.
This is a link for the official Pentaho documentation, looking for information in Hitachi Vantara documentation is becoming more and more difficult, they don't always clarify what steps apply only for the Enterprise Edition and are not available for the Community Edition: https://help.hitachivantara.com/Documentation/Pentaho/9.2/Setup/JDBC_Drivers_Reference
I need to implement migration tool like Flyway in order to use Jenkins to deploy DB changes.
I tried to add jt400.jar file and added configuration as follows:
flyway.url=jdbc:as400://192.168.171.251:446/DBDEV
flyway.driver=com.ibm.as400.access.AS400JDBCDriver
as a driver and it would not connect with this message:
ERROR: No database found to handle jdbc:as400://192.168.171.251:446/DBDEV
I also tried with using IBM DB2 driver and had configuration
flyway.url=jdbc:db2://192.168.171.251:50000/DBDEV
flyway.driver=com.ibm.db2.jcc.DB2Driver
this time I am getting this kind of refusal message
ERROR:
Unable to obtain connection from database (jdbc:db2://192.168.171.251:50000/DBDEV) for user 'DEVUSER':
[jcc][t4][2043][11550][4.26.14] Exception java.net.ConnectException: Error opening socket to server
/192.168.171.251 on port 50,000 with message: Connection refused (Connection refused).
ERRORCODE=-4499, SQLSTATE=08001
With this test migration I am trying to create a simple table by executing this sql
CREATE TABLE PERSON (
ID INT NOT NULL,
NAME VARCHAR(100) NOT NULL
);
Anyone had this situation and solved it?
I believe that at present there is no support for flyway to work with IBM i (as/400) regardless of whether you use jt400.jar or an IBM jdbc driver.
You can either use a different database-schema versioning tool, or find a fork of flyway that supports i-series (or pay someone to create and support such a fork, it is open source...).
It seems that currently (flyway 7.7.2) does not recognize a URL that starts with "jdbc:as400:" as a Db2 URL, so it throws an exception, which is the reason that the jt400.jar style URL is rejected with exception:
"No database found to handle ..."
The github history tells a story (see: https://github.com/flyway/flyway/issues/105).
Looks like the devs did not succeed to get the AS400 support added due to the lack of a suitable available i-series testing/dev environment (and also available to travis ci) . There may have been at least one PR for such support in the past, although it seems to be removed.
If you try to use the IBM db2jcc4.jar driver to connect to i-series (as400) with a url similar to: jdbc:db2://hostname/dbname, and you explicitly use an IBM jre , and have the relevant license file (e.g. db2jcc_license_cisuz.jar on the CLASSPATH), then flyway will connect and then report the exception similar to:
Unsupported Database: AS 7.4
The flyway source code shows that flyway does not recognize this database product-name and version, at current flyway version 7.7.2.
Are you sure DBDEV is the name of your Db2 data base on the IBM i?
Use the Work with RDB Directory Entry (WRKRDBDIRE) from the green screen, and look for the *LOCAL entry.
Or use the Access Client Solutions (ACS) "Schemas" tool to see a list of DB on your system.
The above shows 2 DB's, UT29p63 and Dbtest
I'm fairly new to Oracle SQL developer. As I'm trying to make a new connection via TNS, I'm getting this error down below which I've been looking to solve for a while fruitlessly.
Error Message:
Statut : échec -Echec du test : no ocijdbc18 in java.library.path
Assuming MySQL is a typo (You are using port 1521, which is the default port for Oracle database, also its giving you ODBC error, so I assume its an Oracle database. Remember that you can connect SQL developer to a MySQL database, so please clarify if you want to connect to a MySQL database rather than Oracle) your problem is due to lack of Oracle Data Base Connector driver (ODBC). From this link download the latest version of Oracle Instant client and install it on your machine. Then define ORACLE_HOME to the path of Instant client installation (To the actual installation folder, the one that contains folders like bin, network and so on) and restart SQL Developer, then you should be good to go.
P.S. : as #thatjeffsmith has mentioned correctly in the comments, it's not necessary to have Oracle Client and/or ODBC (or similar) drivers in order to connect to an Oracle database. Using basic as the connection type and adding the correct connection info would suffice. This Article in Oracle Magazine has extensively talked about the different connection types.
I'm trying to migrate a PostgreSQL DB persisted on cloud (on DO droplet) to RDS using AWS Database Migration Service (DMS).
I've successfully configured the replication instance and endpoints.
I've created a task with Migrate existing data and replicate ongoing changes. When I start the task it shows some error ERROR: could not access file "test_decoding": No such file or directory.
I've tried to create a replication slot manually on my DB console it throws the same error.
I've followed the procedures which was suggested on the DMS documentation for Postgres
I'm using PostgreSQL 9.4.6 on my source endpoint.
I presume that the problem is the output plugin test_decoding was not accessible to do the replication.
Please assist me to resolve this. Thanks in advance!
You must install postgresql-contrib additional supplied modules on Your source endpoint.
If it is installed, make sure, directory where test_decoding module located is the same with directory where PostgreSQL expect it.
In *nix, You can check module directory by command:
pg_config --pkglibdir
If it is not the same, copy module, or make symlink, or some other solution You prefer.
After painful installation of hadoop_fdw into our running pgsql 9.3.4, I am trying to connect it to cloudera cluster 5.2.0 with no luck.
Is there a way of debugging the fdw? After creating the foreign table and selecting from it, I just got an error - ERROR: failed to connect to Hive: No more data to read.
btw.: Some old version of hadoop_fdw was capable of using url (jdbc://server:port/args), but not the recent version, there's just address & port.
Hadoop_fdw didn'd make it. There's probably something wrong/old/obsolete in hive.c. But with even more effort we managed to make jdbc_fdw work with cloudera jdbc drivers. The steps were as follows:
1) install jdbc_fdw extension
2) merge all driver jar files into one
3) CREATE SERVER cloudera2 FOREIGN DATA WRAPPER jdbc_fdw OPTIONS(drivername 'com.cloudera.hive.jdbc4.HS2Driver',url 'jdbc:hive2://fqdn:10000;user=hive',querytimeout '15', jarfile '/opt/cloudera/combined.jar');
mental note: set client_min_messages to debug5; can help you identify where is the problem e.g.:driver not found etc