I am trying to connect to Crate as a Streamsets Data collector pipeline origin ( JDBC Consumer ). However I get this error: "JDBC_00 - Cannot connect to specified database: com.streamsets.pipeline.api.StageException: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.PoolInitializationException: Exception during pool initialization: ERROR: SQLParseException: line 1:13: no viable alternative at input 'CHARACTERISTICS' "
Why am I getting this error ? The Crate JDBC Driver version is 2.1.5 and Streamsets Data collector version is 2.4.0.0.
#gashey already solved the issue. Within Streamsets DC uncheck Enforce Read-only Connection on the Advanced tab of my JDBC query consumer configuration
(see https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/crateio/hBexxel2KQw/kU34mrsJBgAJ).
We will update the streamsets-documentation with the workaround. https://crate.io/docs/tools/streamsets/
Related
I am facing below exception while trying to connect to snowflake to pyspark:
py4j.protocol.Py4JJavaError: An error occurred while calling o117.load.
: net.snowflake.client.jdbc.SnowflakeSQLException: !200051!
at net.snowflake.client.core.SFBaseSession.getHttpClientKey(SFBaseSession.java:321)
at net.snowflake.client.core.SFSession.open(SFSession.java:408)
at net.snowflake.client.jdbc.DefaultSFConnectionHandler.initialize(DefaultSFConnectionHandler.java:104)
at net.snowflake.client.jdbc.DefaultSFConnectionHandler.initializeConnection(DefaultSFConnectionHandler.java:79)
at net.snowflake.client.jdbc.SnowflakeConnectionV1.initConnectionWithImpl(SnowflakeConnectionV1.java:116)
at net.snowflake.client.jdbc.SnowflakeConnectionV1.<init>(SnowflakeConnectionV1.java:96)
at net.snowflake.client.jdbc.SnowflakeDriver.connect(SnowflakeDriver.java:172)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at net.snowflake.spark.snowflake.JDBCWrapper.getConnector(SnowflakeJDBCWrapper.scala:209)
It looks like you are behind a firewall or a proxy server. I suggest using the Snowflake connectivity diagnostic tool SnowCD to make sure that all Snowflake URLs are reachable. If you see any errors, then you might want to check your firewall configuration or add a proxy configuration to spark the connection.
I can connect to IBM DB2 inside the IBM Cloud Pak for Data, but when I try to run the exact same %sql connection it errors out. What am I missing?
'''%sql ibm_db_sa://un:pw#host:port/db?security=SSL'''
(ibm_db_dbi.Error) ibm_db_dbi::Error: [IBM][CLI Driver] SQL5005C The operation failed because the database manager failed to access either the database manager configuration file or the database configuration file.\r SQLCODE=-5005
(Background on this error at: http://sqlalche.me/e/dbapi)
Connection info needed in SQLAlchemy format, example:
postgresql://username:password#hostname/dbname
or an existing connection: dict_keys([])
IBM DB2 SQL
Try loading the package ibm_db
Issue using MongoDb with AWS glue - I've created a connection to the database (using the MongoDb connection option) and run a crawler against it and it all worked fine, but when I try to use this as a datasource in a basic ETL job (script- Glue version 2.0Python version 3) it throws the exception
py4j.protocol.Py4JJavaError: An error occurred while calling o70.getDynamicFrame.
: java.lang.RuntimeException: Mongo/DocumentDB connection URL is not supported
Has anyone had any success using MongoDb as a datasource in glue ETL jobs?
i am trying to set up a connector that fetches data from an SQL server to use with apache kafka. I've set up all of the kafka services with a docker-compose file, however the SQL server is on another server.
This is the configuration of my debezium connector in ksqldb:
create source connector sql_connector with
('connector.class'='io.debezium.connector.sqlserver.SqlServerConnector',
'database.server.name'='sqlserver',
'database.hostname'= 'xxxxx',
'database.port'='1433',
'database.user'= 'xxxx',
'database.password'= 'xxxxxx',
'database.dbname'='xxxxxxxxx',
'database.history.kafka.bootstrap.servers'='broker:29092',
'database.history.kafka.topic'='dbz_dbhistory.sqlserver.asgard-01');
When i do this, i get a response that the connector is succesfully created however when i query ksqldb by using 'show connectors' for my connectors i get the following error message:
io.confluent.ksql.util.KsqlServerException: org.apache.http.conn.HttpHostConnectException: Connect to connect:8083 [connect/172.18.0.6] failed: Connection refused (Connection refused)
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to
connect:8083 [connect/172.18.0.6] failed: Connection refused (Connection
refused)
Caused by: Could not connect to the server.
Caused by: Could not connect to the server.
When i inspect my kafka connect logs i can see that its issueing select statements to the server but after a while i get the following error and my kafka connect shuts down:
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000b5400000, 118489088, 0) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 118489088 bytes for committing reserved memory.
# An error report file with more information is saved as:
any ideas on how to fix this? Other then just giving my server more ram.
Your machine has less than ~118MB of free memory:
Native memory allocation (mmap) failed to map 118489088 bytes for committing reserved memory
You will need to increase or free up memory on the machine to get the JVM to start. If it's running, you can change the heap memory settings of the JVM using the following environment variable:
KAFKA_HEAP_OPTS="-Xms256M -Xmx2G"
I am using kafka source connector for capturing CDC from RDS Aurora Postgres. Getting this error.
Please assist if someone know this issue.
Caused by: org.postgresql.util.PSQLException: Database connection failed when reading from copy
at org.postgresql.core.v3.QueryExecutorImpl.readFromCopy(QueryExecutorImpl.java:1074)
at org.postgresql.core.v3.CopyDualImpl.readFromCopy(CopyDualImpl.java:37)
at org.postgresql.core.v3.replication.V3PGReplicationStream.receiveNextData(V3PGReplicationStream.java:158)
at org.postgresql.core.v3.replication.V3PGReplicationStream.readInternal(V3PGReplicationStream.java:123)
at org.postgresql.core.v3.replication.V3PGReplicationStream.readPending(V3PGReplicationStream.java:80)
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.readPending(PostgresReplicationConnection.java:397)
at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:119)
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:99)
... 5 more
Caused by: java.net.SocketException: Socket is closed
at java.base/java.net.Socket.setSoTimeout(Socket.java:1155)
at java.base/sun.security.ssl.BaseSSLSocketImpl.setSoTimeout(BaseSSLSocketImpl.java:639)
at java.base/sun.security.ssl.SSLSocketImpl.setSoTimeout(SSLSocketImpl.java:73)
at org.postgresql.core.PGStream.setNetworkTimeout(PGStream.java:589)
at org.postgresql.core.PGStream.hasMessagePending(PGStream.java:139)
at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1109)
at org.postgresql.core.v3.QueryExecutorImpl.readFromCopy(QueryExecutorImpl.java:1072)
... 12 more
Debezium 1.1.0.CR1 already handles auto-reconnects in these cases https://debezium.io/blog/2020/03/13/debezium-1-1-c1-released/
yes Jiri Pechanec. Actually the problem was with debezium version. older version does not support Postgres auto-connect facility if connection lost for some temporary issue. Now new version of debezium (1.1.0) support auto-connect facility.