I trying to enable hive streaming by following https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest#StreamingDataIngest-StreamingRequirements
I had changed all configuration properties to enable hive streaming, but hive metastore service running with below error,
18/02/09 12:22:51 ERROR compactor.Initiator: Caught an exception in the main loop of compactor initiator, exiting MetaException(message:Unable to connect to transaction database org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does not exist
Note: I am using PostgreSQL for JDBC metastore and hive version 2.0.1
Help me to solve this error and start working with Hive Streaming.
The definition of this table (and others related to ACID tables/streaming ingest) can be found in https://github.com/apache/hive/blob/branch-2.0/metastore/scripts/upgrade/postgres/hive-txn-schema-2.0.0.postgres.sql. All of these are required for streaming to function properly.
Related
I have created a visual job in AWS Glue where I extract data from Snowflake and then my target is a postgresql database in AWS.
I have been able to connect to both Snowflak and Postgre, I can preview data from both.
I have also been able to get data from snoflake, write to s3 as csv and then take that csv and upload it to postgre.
However when I try to get data from snowflake and push it to postgre I get the below error:
o110.pyWriteDynamicFrame. null
So it means that you can get the data from snowflake in a Datafarme and while writing the data from this datafarme to postgres, you are failing.
You need to check was glue logs to get more understanding why is this failing while writing the data into postgres.
Please check if you have the right version of jars (needed by postgres) compatible with scala(on was glue side).
I am using scala and akka stream for my application and finally want to insert the record to snowflake.
Is it possible to connect to snowflake using slick jdbc or alpakka slick .
Please assist
You can't, Snowflake is not in the list of supported databases:
https://scala-slick.org/doc/3.3.2/supported-databases.html
I'm using Kafka connect HDFS.
When I'm trying to run my connector I'm got the following exception:
ERROR Failed creating a WAL Writer: Failed to create file[/path/log] for [DFSClient_NONMAPREDUCE_208312334_41] for client [IP] because this file is already beign create by [DFSClient_NONMAPREDUCE_165323242_41]
Any suggestions please?
Aside from the workaround to cleanup the wal logs and connector restart. We are also testing on upgrading the hadoop local version 2.73 to 2.7.7 and the kafka-hdfs version to 5.5 of our sink connector from Its currently on testing and so far the wal issue is not recoccuring.
As per further testing, upgrading the version to 2.7.7 didn't help. We still ran into WAL issue. We ended using a different sink s3SinkConnector.
I'm using AWS Glue and have a crawler to reflect tables from a particular schema in my Redshift cluster to make those data accessible to my Glue Jobs. This crawler has been working fine for a month or more, but now all of the sudden I'm getting the following error:
Error crawling database reporting: SQLException: SQLState: 42P01 Error Code: 500310 Message: [Amazon](500310) Invalid operation: relation "{table_name}" does not exist
But, I can query the relevant schema & table with the exact same credentials used for the connection that Glue is using. I am able to subset to particular tables in the schema and have Glue reflect those, but not the full schema or the problematic tables it runs into.
Any ideas on how Glue reflects tables from Redshift and what might be going on here? The crawlers are all pretty black-box so I've pretty quickly run out of debugging ideas and not sure what else to try.
Can you please provide an example of flyway.conf settings for Redshift?
I tried using:
flyway.url=jdbc:Redshift://name.redshift.amazonaws.com:5439/DBName
flyway.user=user
flyway.password=pass
but that produced this error:
ERROR: Unable to autodetect JDBC driver for url: jdbc:Redshift:
There are many issues here:
redshift should be lower case in the jdbc url
You also need to put the Redshift JDBC driver on the classpath (/drivers directory for Flyway command-line)
additionally you need to set flyway.driver to the AWS redshift driver class name (Flyway defaults to the standard PG driver: http://flywaydb.org/documentation/database/redshift.html)