We have a four-node Hadoop cluster with HDP 2.4 and Kerberos installed in it. As this is our production cluster, We wanted to have HA for all the services including the PostgreSQL database which is used by Hive, Ambari, and Oozie for storing the metadata. However, the version of our postgreSQL, which is 8.4.2 doesn't support the inbuilt feature(stream replication) of Postgres.
So, we have decided to upgrade PostgreSQL to a version(9.3) ambari supports.
I followed this link to upgrade the Postgres. Everything went well. Expect that, we are getting the following error when restarting ambari server.
Ambari Server running with administrator privileges.
Running initdb: This may take upto a minute.
Data directory is not empty!
[FAILED]
Could someone help?
Thanks.
Your server want's to initilize the Database. I guess your Server does not see the Ambari DB. Use ambari-server setup zu restore the database Connection. Than the sever should start perfectly.
I found the fix for the issue here.
Related
I'm trying to connect to an RDS cluster in AWS that's an Aurora PostgreSQL database. It's a brand-new database that I created along with the instances that I have Jira deployed to. However, when I try to connect to the instance from the Jira configuration screen I get this error
You have specified a database that is not empty, please specify an empty database.
I haven't touched this database at all, why is it giving me this error? I have one read and one write database in my cluster and the "hostname" is the endpoint for my write database, which is what the docs say. Could this be an issue with the Jira version I'm using?
This is the download link I'm using in my user-data script to install Jira. I'm also using PostgreSQL version 12.11
https://www.atlassian.com/software/jira/downloads/binary/atlassian-servicedesk-4.19.1-x64.bin
I switched to a different PostgreSQL version and now it's working.
PostgreSQL version 12.11 was giving me the error and switching to version 13.7 works as expected.
I need to poll DB2 database table until a record is created and once it is successful,i need to trigger a task in dag.
For this I have to do below:
Create a DB2 connection in airflow
Use SQL sensor to poll db table using the connection created in above step.
But I don't see the connection type option for DB2 (screenshot attached). Do I have to install something? Please help.
PS: I am new to Airflow.
There is no DB2 specific connection.
For general connections you can just use the Generic connection:
For older Airflow versions that doesn't have Generic connection you can use any other connection type (HTTP/MySql for example). It doesn't really matter. Airflow looks for connection by the connection id. The type is almost meaningless in that perspective.
Yes the airflow connection does not appear to be a specific connection for db2, so you have to choose generic in connection Type.
I also recommend to install the security mechanism for connection.
the next package: pip install airflow-provider-db2
I have tried to connect to DB2 successfully on Airflow 2.2.5.
Airflow doesn't has specific provider for DB2. You need to use JDBC to connect.
So you need to install some libraries of Airflow and download DB2 JDBC driver to fulfill this task.
List of all libs and provider need to install below:
Apache-airflow-provider-jdbc/Version:2.1.3
JayDeBeApi/Version:latest
JPype1/Version:latest
You also need to download DB2 JDBC drivers. This driver include some packages and license file like:
db2jcc4.jar
sqlj4.zip
jdbc4_LI_en
Because it will use JDBC to connect,openJDK also need to install on your docker.
Setting environment variables JAVA_HOME, PATH and CLASSPATH are needed.
All of above actions can be done on your docker file.
When you have completed to build your image, You can start your airflow to set connection.
Connection URL syntax: jdbc:db2://Host:Port/Database
You can replace HOST,PORT and Database.
Driver Path: You can input where your class file(db2jcc4.jar) located.
Hope these tips can help you. I am New to Airflow too.
I'm trying to upgrade an RDS database cluster engine from Aurora PostgreSQL 9.6.19 before its end of life, I made copy and tried to upgrade to 9.6.21 and 10.16 but everytime the same problem happens:
Database cluster is in a state that cannot be upgraded: Postgres cluster is in a state where pg_upgrade can not be completed successfully.
The status of the database is Available so maybe it refers to something else but I don't know what and how to fix it, I've tried looking for answers to no avail.
Has anyone fixed this?
The pg_upgrade_internal log file will usually contain details on any failures/errors.
You can take a look on these logs using the command line:
aws rds describe-db-log-files --db-instance-identifier my-db-instance
Or via console, or RDS API.
For more information take a look on these links: Upgrading the PostgreSQL DB engine for Amazon RDS, Viewing and listing database log files
I have PostgreSQL 9.5 (yes I know it's not supported anymore) installed on Ubuntu Server 18.04 using this instructions https://www.postgresql.org/download/linux/ubuntu/
I want to change path and separate log for every database. But it's configuret by package maintainer in such a way that it ignores log* settings in PostgreSQl configuration and uses some other way to log everything to files and I can't find out how. Currently it logs to /var/log/postgresql/postgresql-9.5-clustername.log. I want it to be /var/log/postgresql/clustername/database.log but I don't know where to configure it. In PostgreSQL log_destination is set to stderr
The Ubuntu packages have logging_collector disabled by default, so the log is not handled by PostgreSQL, but by the startup script.
However, there is no way in PostgreSQL to get a separate log file per database, so the only way to get what you want is to put the databases in individual clusters rather than into a single cluster.
I'm trying to migrate a PostgreSQL DB persisted on cloud (on DO droplet) to RDS using AWS Database Migration Service (DMS).
I've successfully configured the replication instance and endpoints.
I've created a task with Migrate existing data and replicate ongoing changes. When I start the task it shows some error ERROR: could not access file "test_decoding": No such file or directory.
I've tried to create a replication slot manually on my DB console it throws the same error.
I've followed the procedures which was suggested on the DMS documentation for Postgres
I'm using PostgreSQL 9.4.6 on my source endpoint.
I presume that the problem is the output plugin test_decoding was not accessible to do the replication.
Please assist me to resolve this. Thanks in advance!
You must install postgresql-contrib additional supplied modules on Your source endpoint.
If it is installed, make sure, directory where test_decoding module located is the same with directory where PostgreSQL expect it.
In *nix, You can check module directory by command:
pg_config --pkglibdir
If it is not the same, copy module, or make symlink, or some other solution You prefer.