please help if you have any idea:
I am trying to connect kerberized Hortonworks hadoop clusters livy server with jupyter, I have 401 error when connecting
Is it possible to connect sparkmagic to kerberized spark livy server? if it is then I think I have some misconfiguration in the sparkmagic config json.
username and password is the technical user which runs server and have right of impersonation in the hadoop cluster(proxy user) not actual username when I login in the jupyterhub
its part of my config.json:
"kernel_python_credentials" : {
"username": "username",
"password": "password",
"url": "http://mylivy.server:8999",
"auth": "Kerberos"
}
"logging_config": {
"version": 1,
"formatters": {
"magicsFormatter": {
"format": "%(asctime)s\t%(levelname)s\t%(message)s",
"datefmt": ""
}
},
"handlers": {
"magicsHandler": {
"class": "hdijupyterutils.filehandler.MagicsFileHandler",
"formatter": "magicsFormatter",
"home_path": "~/.sparkmagic"
}
},
"loggers": {
"magicsLogger": {
"handlers": ["magicsHandler"],
"level": "DEBUG",
"propagate": 0
}
}
},
"wait_for_idle_timeout_seconds": 15,
"livy_session_startup_timeout_seconds": 600,
.................................................etc............................
this is error message when I try some "hello world" in the spark or pyspark notebook or shell in the jupyter:
print("Hello World")
The code failed because of a fatal error: Invalid status code '401'
from http://mylivy.server:8999/sessions with error payload:
Error 401
HTTP ERROR: 401 Problem accessing
/sessions. Reason: Authentication required Powered by Jetty://
9.3.24.v20180605 .
Some things to try: a) Make sure Spark has enough available resources
for Jupyter to create a Spark context. b) Contact your Jupyter
administrator to make sure the Spark magics library is configured
correctly. c) Restart the kernel.
(UPDATE)
I just have found the reason why the error occurred: problem was that there was not ticket on the system where the notebook app was launched, kinit command resolved the issue. P.S. also username and password was not needed when use kerberos in the config.json
Related
I am trying to ingest data from MySQL to BigQuery. I am using Debezium components running on Docker for this purpose.
Anytime I try to deploy the BigQuery sink connector to Kafka connect, I am getting this error:
{"error_code":400,"message":"Connector configuration is invalid and contains the following 2 error(s):\nFailed to construct GCS client: Failed to access JSON key file\nAn unexpected error occurred while validating credentials for BigQuery: Failed to access JSON key file\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"}
It shows it's an issue with the service account key (trying to locate it)
I granted the service account BigQuery Admin and Editor permissions, but the error persists.
This is my BigQuery connector configuration file:
{
"name": "kcbq-connect1",
"config": {
"connector.class": "com.wepay.kafka.connect.bigquery.BigQuerySinkConnector",
"tasks.max" : "1",
"topics" : "kcbq-quickstart1",
"sanitizeTopics" : "true",
"autoCreateTables" : "true",
"autoUpdateSchemas" : "true",
"schemaRetriever" : "com.wepay.kafka.connect.bigquery.retrieve.IdentitySchemaRetriever",
"schemaRegistryLocation":"http://localhost:8081",
"bufferSize": "100000",
"maxWriteSize":"10000",
"tableWriteWait": "1000",
"project" : "dummy-production-overview",
"defaultDataset" : "debeziumtest",
"keyfile" : "/Users/Oladayo/Desktop/Debezium-Learning/key.json"
}
Can anyone help?
Thank you.
I needed to mount the service account key in my local directory to the Kafka connect container. That was how I was able to solve the issue. Thank you :)
I have set up Stepzen as described in the Getting Started Docs. I get the following error when running stepzen start:
{
"data": {
"getProfilesList": null
},
"errors": [
{
"message": "database error: can't prepare the statement for execution failed to connect to `host=postgres user=root database=`: hostname resolving error (lookup postgres on 10.97.0.10:53: no such host)",
"locations": [
{
"line": 2,
"column": 3
}
],
"path": [
"getProfilesList"
]
}
]
}
The message shows wrong hostname, user and empty database, although my config file has the correct credentials:
configurationset:
- configuration:
name: postgresql_config
uri: postgresql://postgres:[My Password]#[My Supabase DB Host]/postgres
I have also tried to connect to the DB directly from a GUI client and the connection was successful.
Why this issue is coming up when trying to connect from Stepzen Graphiql explorer running on localhost?
Postgres requires the password to be URL Encoded when using some characters that have a special meaning.
You can learn more about percent-encoding to handle special characters in Postgres passwords here: How to handle special characters in the password of a Postgresql URL connection string?
I'm quite new to Strapi and I'm following the Strapi deployment documentation at https://strapi.io/documentation/3.0.0-beta.x/guides/deployment.html#configuration. I have setup strapi using mongodb and it seems to work both in production and dev on my server. I can create content types and add data...
Now I'm trying to start Strapi using the PM2 Runtime. I have setup the ecosystem.config.js file (see below) and I run pm2 start ecosystem.config.js. The Strapi app seems to start just fine, but now what happens in the browser is that I am prompted with a new admin user. Seems like all users and data is lost... Mongo db not accessed or whats going on?
this is my ecosystem.config.js file
module.exports = {
apps : [{
name: 'cms.strapi',
cwd: '/var/www/domain/public_html',
script: 'server.js',
env: {
NODE_ENV: 'production',
DATABASE_HOST: '127.0.0.1',
DATABASE_PORT: '28015',
DATABASE_NAME: 'db-name',
DATABASE_USERNAME: 'db-u-name',
DATABASE_PASSWORD: 'pw',
},
}],
};
What am I missing?
Hi Jim and thanks for your reply! I believe the problem was a mixup between the prod and the dev environment. Sorry, my bad. I thought I was in one environment when I was really in the other. I guess it should be obvious when you start the server from the prompt whether your starting dev or prod, but once the web server is up and running in the browser I guess you can't tell from the gui whether it's the one or the other. At least I can't find one other than that the admin usernames (and possibly data) are different... Hmm..
Anyway my production/database.json file looks like this:
{
"defaultConnection": "default",
"connections": {
"default": {
"connector": "mongoose",
"settings": {
"uri": "mongodb://localhost:27017/db-prod",
"database": "db-prod",
"host": "127.0.0.1",
"srv": false,
"port": 27017,
"username": "u-name-prd",
"password": "pw"
},
"options": {
"ssl": false
}
}
}
}
PM2 Runtime seems to be working correctly with Strapi and Mongo now :-)
I am trying to setup the loopback starter app on Bluemix with MongoDB. I have set up the MongoDB instance in Compose. However, when I keep getting a connection error even though I have followed all instructions and can even connect using the mongo shell.
{ [MongoError: connect ECONNREFUSED] name: 'MongoError', message: 'connect ECONNREFUSED' }
Take a look here: http://www.technicaladvices.com/2015/10/06/deploying-your-existing-strongloop-application-to-ibm-bluemix/
It shows the details of deploying a "StrongLoop" app in IBM Bluemix.
If the issue is still there open a support request directly from your Bluemix console or you can open a new ticket here: https://support.ng.bluemix.net/gethelp/
I was able to solve the problem by using the following configuration format in datasources.json:
"mongoDs": {
"host": "candidate.53.mongolayer.com",
"port": 10852,
"database": "SiteRite",
"username": "xxxx",
"password": "xxxx",
"name": "mongoDs",
"connector": "mongodb"
}
NOT using the 'url' key and using 'host' and 'port' with a separate 'username' and 'password' for the database is what seemed to have fixed it.
I'm using the Cloud 9 IDE to develop an application using MongoDB. I created a database called "appdata" at MongoLab and the following user:
{
"_id": "appdata.admin",
"user": "admin",
"db": "appdata",
"credentials": {
"SCRAM-SHA-1": {
"iterationCount": 10000,
"salt": "K/WUzUDbi3Ip4Vy59gNV7g==",
"storedKey": "9ow35+PtcOOhfuhY7Dtk7KnfYsM=",
"serverKey": "YfsOlFx1uvmP+VaBundvmVGW+3k="
}
},
"roles": [
{
"role": "dbOwner",
"db": "appdata"
}
]
}
Whenever I try connecting to the database through Cloud 9 Shell using the following command (given by MongoLab with my newly created user):
mongo ds057244.mongolab.com:57244/appdata -u admin -p admin
I get the following error message:
MongoDB shell version: 2.6.11
connecting to: ds057244.mongolab.com:57244/appdata
2015-11-22T05:23:49.015+0000 Error: 18 { ok: 0.0, errmsg: "auth failed",
code: 18 } at src/mongo/shell/db.js:1292
exception: login failed
Also, on my javascript file running on Cloud 9, while following this tutorial (which uses mongoose to access the DB) I got stuck on the post route for bears. Whenever I send a post request through postman with the specified fields set, the route doesn't return anything, neither a bear created nor an error message, which makes me think the problem is also failing to login to the database. The previous get request is working just fine and my code is the exactly same as the tutorial.
Does anyone know what the problem in any of the cases and what I need to do to solve them?
The shell problem was fixed updating it to the Database version (which was 3.0.3).
For the javascript files, I restarted the tutorial and made sure I downloaded all necessary dependencies with the most recent stable version (not the ones shown on the tutorial), after that the problem was solved.