I'm having some difficulty with using Kerberos with sparkmagic on a jupyter notebook. I installed jupyter via anaconda on a windows box, and I was able to set it up to access a test HDInsight cluster. That was just to prove that I could get it to access anything via sparkmagic. So I know the general jupyter/sparkmagic setup works.
What I really needed it to do was to talk to the on-prem cluster. That is running livy and secured via kerberos. My difficulty is in getting sparkmagic to use the windows kerberos tickets. I can't find any directions on setting up the sparkmagic config file to authenticate. Furthermore, when I run the following on windows, it gives me an authentication error when I connect to livy:
curl --negotiate --user myusername:mypassword --url http://livyserver:8998/sessions
If I run the same line when I ssh to the edge node, it authenticates properly. I use the same username and password to authenticate Windows, the edge node and Hue.
In the sparkmagic config file, I tried putting in my username and password like so:
{
... stuff ...
"kernel_scala_credentials":{
"username":"myusername",
"password":"mypassword",
"url": "http://livyserver:8998",
"auth": "Kerberos"
}
... the rest is following example_config.json ...
}
So, how do I set up winkerberos and sparkmagic to honor the windows tickets? Or is it that, despite using the same credentials, I need Windows to get a kerberos ticket from the authentication method used by the linux boxes?
Related
Problem Summary:
I can SSH to remote host using Kerberos and PuTTY on Windows 10, but I can't connect using VS Code.
Steps I have tried:
I used MIT Kerberos Ticket Manage to generate a Kerberos API key.
Then, in PuTTY I selected both “Attempt GSSAPI authentication” and “Allow GSSAPI credential delegation” . After entering the host name and my username, I can successfully connect to the remote host in PuTTY.
However, I cannot connect to the remote host in VS Code using the following SSH config file:
Host my-host
GSSAPIAuthentication yes
GSSAPIDelegateCredentials yes
User my-name
My error looks like this:
I learned from this Stack Overflow answer that Windows "has two Kerberos libraries (MIT KfW & Windows SSPI)", so my suspicion is that VS Code is not looking for GSSAPI libraries in the correct order, like the PuTTY screen shot. But I don't know how to specify the order like in PuTTY.
Please help! Thanks!
I used to connect remote servers provided by Linode and GCP and I use Visual Studio Code via Sftp extension and all works just fine. However, with the same set of configurations, I couldn't connect to my Aws EC2 server.
The following is the sftp.json (with sensitive information changed)
And the aws security setting is as following
The strange thing is that I can ssh connect to the server on the terminal and I can use filezilla to send file with the same credential as well. But somehow I can't connect to it via Visual Studio Code sftp or SSH remote extension.
I searched the anwser on the internet and found an answer that worked for me.
The problem has to do with the Ubuntu 22.04 default server default key setting and not with the VSCode.
The solution is adding
PubkeyAcceptedKeyTypes=+ssh-rsa
in /etc/ssh/sshd_config
and then restart the service on the server with the following command
sudo systemctl restart sshd
The source of the answer comes from https://github.com/liximomo/vscode-sftp/issues/37 under the user windware-ono's answer.
Is it possible to run docker without elevated priv ex. (docker version).
Im trying to run a command on another machine (windows server with docker as service) with powershell invoke command but it seems as long as the docker insists on elevated priv i cannot.
So if i can get "docker verison" to work im all set.
The error i get is
docker.exe: error during connect: Post http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.39/containers/create: open //./pipe/docker_engine: Access is denied. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.
See 'C:\Program Files\Docker\docker.exe run --help
it works with an elevated powershell.
Any ideas?
This is normal - by default, a local named pipe is used for the Docker CLI to communicate with the service (aka daemon).
For development use you can configure the host machine's Docker service ("daemon") for TCP access but this is the least secure option. Just put this text in file daemon.json:
{
"hosts": ["tcp://0.0.0.0:2375"]
}
Once this is done you can connect with e.g.
docker --host tcp://1.2.3.4:2375 version
If this is for production use, you probably need to look at a container orchestration system.
A middle ground would be to useAttach-PSSession to attach to an admin PowerShell session on the remote machine. This still requires a privileged user but does work remotely.
I use Jupyter Notebook to run bioinformatic analyses, and I love it. However, it only really plays nice when I run it on my personal computer. However, I regularly do analysis using a remote computer with multiple cores to reduce processing time. I'd like to be able to use the Jupyter Notebook interface on my personal computer while everything is actually running on the remote computer. I generally do this via ssh access to the remote computer within the shell and execute all commands at the command line. I'd love to do this from the Jupyter notebook on my personal computer, rather than from the shell on my personal computer. It is relevant that I don't have sudo access on the remote computer.
So far, I've installed miniconda and jupyter notebook on the remote computer like this:
wget https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
Once conda is installed properly, I install jupyter notebook via miniconda with this line:
conda install jupyter
This installs successfully. I can then start a jupyter notebook session on the remote machine with the line:
jupyter notebook --no-browser
So far, so good. My next question: How do I have my local jupyter notebook connect to the remote machine, so that I can execute commands on the remote machine using my local jupyter notebook? There is some documentation here, however i have been trying different things for hours, but have failed to succeed.
Can anyone give a straight forward method to connect to my remote server, given that I am this far along? I feel like it should just be a matter of entering url addresses and passwords into my local Jupyter notebook (all of this is so easy via ssh in the shell).
Follow the steps below:
Enable port forwarding on remote machine
ssh -N -f -L 127.0.0.1:8898:127.0.0.1:8898 user#remote-machine.com
Do ssh to your remote machine and then run following command on remote machine
jupyter-notebook --no-browser --port=8898
you will see some thing as shown below
Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://localhost:8898/token=eaf2f51f9c053f43d8bd093e76f0cc6301b545549c998fa2&token=eaf2f51f9c053f43d8bd093e76f0cc6301b545549c998fa2
Copy and paste the URL in your local machine browser.
If you want to access Jupyter/Ipython notebook running on a VPS remotely, I wrote a tutorial on the digital ocean community site.
As shown in the guide, after installing and running Ipython Notebook using command line on the server, you can connect to the notebook using SSH tunnelling with Putty (on windows) or the ssh -L command on Unix-like systems (ie Mac and Linux)
I have setup and launched an instance of Amazon EC2 server with Ubuntu in it. Now I have integrated cygwin with command prompt also so all linux commands are working in command prompt.
I tried to access the server using ssh -i munish.pem ubuntu#52.11.190.155 (munish.pem contains my secret key).
After running this command I get an error: 'ssh' is not recognized as an internal or external command, operable program or batch file. I searched net and could find solution for github not for Amazon EC2 service...
You can use putty software in window for connecting to the AWS EC2 instance.
Follow the below steps:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html
Hope it helps..
You can open PowerShell and check ssh command is available or not.
If not, you can install OpenSSH in Windows 10. See following guide on how to install it.
https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse