I am facing below exception while trying to connect to snowflake to pyspark:
py4j.protocol.Py4JJavaError: An error occurred while calling o117.load.
: net.snowflake.client.jdbc.SnowflakeSQLException: !200051!
at net.snowflake.client.core.SFBaseSession.getHttpClientKey(SFBaseSession.java:321)
at net.snowflake.client.core.SFSession.open(SFSession.java:408)
at net.snowflake.client.jdbc.DefaultSFConnectionHandler.initialize(DefaultSFConnectionHandler.java:104)
at net.snowflake.client.jdbc.DefaultSFConnectionHandler.initializeConnection(DefaultSFConnectionHandler.java:79)
at net.snowflake.client.jdbc.SnowflakeConnectionV1.initConnectionWithImpl(SnowflakeConnectionV1.java:116)
at net.snowflake.client.jdbc.SnowflakeConnectionV1.<init>(SnowflakeConnectionV1.java:96)
at net.snowflake.client.jdbc.SnowflakeDriver.connect(SnowflakeDriver.java:172)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at net.snowflake.spark.snowflake.JDBCWrapper.getConnector(SnowflakeJDBCWrapper.scala:209)
It looks like you are behind a firewall or a proxy server. I suggest using the Snowflake connectivity diagnostic tool SnowCD to make sure that all Snowflake URLs are reachable. If you see any errors, then you might want to check your firewall configuration or add a proxy configuration to spark the connection.
Related
I am trying to import data from my local PostgreSQL database to neo4j
Firstly, I load the JDBC driver into memory
CALL apoc.load.driver("org.postgresql.Driver")
Then, I run this query to ingest data from postgresql
WITH "jdbc:postgresql://localhost:5432/graph-test?user=kt" as url
CALL apoc.load.jdbc(url,"os.operating_systems") YIELD row AS line
MERGE (o:Os {name: line.name})
MERGE (of:OsFamily {name: line.familly})
MERGE (o)-[:FROM]->(of)
Unfortunately, I received this error.
Failed to invoke procedure `apoc.load.jdbc`: Caused by: java.net.ConnectException: Connection refused (Connection refused)
Could this error be caused by the incorrect URL, jdbc plugin version, or something else?
Issue using MongoDb with AWS glue - I've created a connection to the database (using the MongoDb connection option) and run a crawler against it and it all worked fine, but when I try to use this as a datasource in a basic ETL job (script- Glue version 2.0Python version 3) it throws the exception
py4j.protocol.Py4JJavaError: An error occurred while calling o70.getDynamicFrame.
: java.lang.RuntimeException: Mongo/DocumentDB connection URL is not supported
Has anyone had any success using MongoDb as a datasource in glue ETL jobs?
i am trying to set up a connector that fetches data from an SQL server to use with apache kafka. I've set up all of the kafka services with a docker-compose file, however the SQL server is on another server.
This is the configuration of my debezium connector in ksqldb:
create source connector sql_connector with
('connector.class'='io.debezium.connector.sqlserver.SqlServerConnector',
'database.server.name'='sqlserver',
'database.hostname'= 'xxxxx',
'database.port'='1433',
'database.user'= 'xxxx',
'database.password'= 'xxxxxx',
'database.dbname'='xxxxxxxxx',
'database.history.kafka.bootstrap.servers'='broker:29092',
'database.history.kafka.topic'='dbz_dbhistory.sqlserver.asgard-01');
When i do this, i get a response that the connector is succesfully created however when i query ksqldb by using 'show connectors' for my connectors i get the following error message:
io.confluent.ksql.util.KsqlServerException: org.apache.http.conn.HttpHostConnectException: Connect to connect:8083 [connect/172.18.0.6] failed: Connection refused (Connection refused)
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to
connect:8083 [connect/172.18.0.6] failed: Connection refused (Connection
refused)
Caused by: Could not connect to the server.
Caused by: Could not connect to the server.
When i inspect my kafka connect logs i can see that its issueing select statements to the server but after a while i get the following error and my kafka connect shuts down:
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000b5400000, 118489088, 0) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 118489088 bytes for committing reserved memory.
# An error report file with more information is saved as:
any ideas on how to fix this? Other then just giving my server more ram.
Your machine has less than ~118MB of free memory:
Native memory allocation (mmap) failed to map 118489088 bytes for committing reserved memory
You will need to increase or free up memory on the machine to get the JVM to start. If it's running, you can change the heap memory settings of the JVM using the following environment variable:
KAFKA_HEAP_OPTS="-Xms256M -Xmx2G"
I am trying to connect to Crate as a Streamsets Data collector pipeline origin ( JDBC Consumer ). However I get this error: "JDBC_00 - Cannot connect to specified database: com.streamsets.pipeline.api.StageException: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.PoolInitializationException: Exception during pool initialization: ERROR: SQLParseException: line 1:13: no viable alternative at input 'CHARACTERISTICS' "
Why am I getting this error ? The Crate JDBC Driver version is 2.1.5 and Streamsets Data collector version is 2.4.0.0.
#gashey already solved the issue. Within Streamsets DC uncheck Enforce Read-only Connection on the Advanced tab of my JDBC query consumer configuration
(see https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/crateio/hBexxel2KQw/kU34mrsJBgAJ).
We will update the streamsets-documentation with the workaround. https://crate.io/docs/tools/streamsets/
I am upgrading my heroku database from a hobby dev to Standard 0 (using the official instructions https://devcenter.heroku.com/articles/upgrading-heroku-postgres-databases#upgrade-with-pg-copy-default).
All went well, until I promoted the new database and restarted the app. I then get the following error:
o.s.boot.SpringApplication : Application startup failed
...
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Unable to obtain Jdbc connection from DataSource
...
Caused by: org.flywaydb.core.api.FlywayException: Unable to obtain Jdbc connection from DataSource
...
Caused by: org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for host "54.xxx.xx.xxx", user "u94bf9vxxxxxx", database "d2mqk0b6xxxxxx", SSL off
...
If I swap back to the old database again, everything works again. The only thing that I am changing is the promoted database.
Is there a difference between connecting to hobby and standard databases that I need to be aware of?
The relevant part of my application.yml looks as follows:
spring:
datasource:
driverClassName: org.postgresql.Driver
url: ${JDBC_DATABASE_URL}
username: ${JDBC_DATABASE_USERNAME}
password: ${JDBC_DATABASE_PASSWORD}
flyway:
enabled: true
locations: classpath:db/migrations
Any suggestions on how I can debug this would be very welcome too.
Looks like you aren't connecting with SSL where it is required by Heroku PostgreSQL installs.
See Herokus documentation on SSL for PostgreSQL.
See also Herokus documentation for enabling SSL on JDBC connections.
You will need to add something like &ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory to your JDBC URL.