I have read and implemented instructions from earlier posts like:
How to start an ipython shell(not notebook) within a conda or virtualenv
My goal is to use a kernel in ipython which has all conda packages from my virtual environment.
I have a google ubuntu 16.04 machine where I have installed anaconda and a virtual environment in which i installed all my packages..
when i run
python -m ipykernel.kernelspec
i get the following error:
/home/admin/anaconda3/envs/py36ve/lib/python3.6/site-packages/IPython/paths.py:61: UserWarning: IPython dir
'/home/admin/.ipython' is not a writable location, using a temp
directory.
" using a temp directory.".format(ipdir))
[Errno 13] Permission denied: '/usr/local/share/jupyter/kernels/python3'
I tried running with sudo too.. i created a kernel but when i use it then it has none of the packages i installed in the virtual environment..
I do have a similar issue with this when I try to submit my program to a cluster where it doesn't have access to my local directory and it shows the same message. But I don't get Permission denied message and everything is fine by me. But I wanted to address this issue and looked into it and I found that paths.py at line 62 in python package in the case of not writable, it creates a temp directory like the following:
ipdir = tempfile.mkdtemp()
As in tempfile documentation says:
Creates a temporary directory in the most secure manner possible. There are no race conditions in the directory’s creation. The directory is readable, writable, and searchable only by the creating user ID.
It is strange that you do get this but if you want to make it work, find the paths.py and change that to your liking and makes sure it works and replace it with the original.
Related
In a Jupyter notebook connected to a GCP Spark cluster, the cell !pip3 install pyLDAvis==3.2.1 works, but gives a warning:
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager.
It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
The warning is not unique to pyLDAvis, other packages — even numpy — give the same warning.
Running the notebook as root shouldn't be the default. How can the default user in the notebook be set to singhj rather than root? I have searched through IPython Configuration and customization for any hints.
Configuration: Fresh cluster in GCP Dataproc, default Jupyter notebook, nothing customized.
The Jupyter server in a Dataproc cluster is run by the systemd service defined in the file /usr/lib/systemd/system/jupyter.service.
If you want to change the user it runs as, then you can modify that file and replace the line saying User=root with one saying the name of the user you want (e.g. User=singhj in your example).
Then, once the file has been updated, restart the systemd service by running the following commands as root:
systemctl daemon-reload
systemctl restart jupyter
If you want to automate that, you can write an initialization action to make the change at cluster creation time.
I have a few quick navigation plugins such as "block travel" I use all the time. Is there a way to use these in cloud shell?
I imagine there are some restrictions, but even some simple editor plugins can be huge timesavers.
While I'm at it - alt-D to duplicate a line, or transpose lines - some of those seem to be missing and hard to use key remapping to get working, at least within the shell. In general maybe keyboard shortcuts seem to get trapped by the browser or PWA wrapper. I'm using cloudshell as a webapp on a chromebook FWIW, for various secure projects.
I have come up with a solution that covers both aspects of your question
To get Unlimited Persistent Disk:
You can use Google Cloud Storage FUSE
Google Cloud Storage FUSE lets you mount a GCS bucket as a folder to your linux instance. By doing that you get an “unlimited '' persistent disk and it is super simple to set up since gcsfuse is already installed in cloud shell.
1. Create a GCS bucket (you only need to run this once) -- replace BUCKET_NAME with any name:
gsutil mb "gs://BUCKET_NAME/"
2. Create a local directory for mounting -- replace FOLDER_NAME with the chosen directory name:
mkdir /home/[USER]/[FOLDER_NAME]
chmod 777 /home/[USER]/[FOLDER_NAME]
3. Mount the bucket onto the local filesystem (note: you need to re-run this every time Cloud Shell starts)
gcsfuse -o nonempty -file-mode=777 -dir-mode=777 --uid=1000 --debug_gcs [BUCKET_NAME] /home/[USER]/[FOLDER_NAME]
To use third party plugins in cloud shell:
You can use an environment customization script (.customization_environment) as mentioned in the public documentation. It allows you to install additional packages into your Cloud Shell environment when it starts.
For reference, below are the steps to install VS Code plug-in.
Step 1:
To install the VSCode server, run the script named visual_studio_code.sh as below, in the root directory workspace of Cloud Shell Editor.
visual_studio_code.sh file:
export VERSION=`curl -s https://api.github.com/repos/cdr/code-server/releases/latest | grep -oP '"tag_name": "\K(.*)(?=")'`
wget https://github.com/cdr/code-server/releases/download/$VERSION/code-server-3.10.2-linux-amd64.tar.gz
tar -xvzf code-server-3.10.2-linux-amd64.tar.gz
cd code-server-3.10.2-linux-amd64
Run the script using the below command in shell,
./visual_studio_code.sh
if getting permission denied error then run this following command in shell,
chmod +x visual_studio_code.sh
./visual_studio_code.sh
Step 2:
Make a customization script in the root directory workspace in Cloud Shell Editor to start VS Code Server on boot with the below commands :
.customization_environment file :
#!/bin/sh
#.customize_environmnet run in background as root, wait for your user to initialize
sleep 20
sudo -u [USER] /home/[USER]/code-server-3.10.2-linux-amd64/code-server --auth none --port 9090
Step 3:
To view Visual Studio Code server on port 9090 :
Click on Web Preview > Change Port > 9090
If getting a 404 error, remove ‘?authuser=0’ from the url.
Visual Studio Code Server will now be running on the browser!!!
Block travel navigation plugin:
To have the block travel navigation plugin in cloud shell,follow the below commands and run them in shell in root directory:
wget https://github.com/efatsi/block-travel/archive/refs/tags/v1.0.0.tar.gz
tar xzvf v1.0.0.tar.gz
ls
#You will see block-travel-1.0.0
block-travel-1.0.0/keymaps/block-travel.cson --auth none --port 9090
#You might get Permission denied if yes, then follow the next two commands else go to webport view in 9090
chmod +x block-travel-1.0.0/keymaps/block-travel.cson
block-travel-1.0.0/keymaps/block-travel.cson --auth none --port 9090
Open the webport view in 9090, you will be able to navigate through the vs code files using :
Alt+up for block-travel.jumpUp
Alt+shift+up for block-travel.selectUp
Alt+down for block-travel.jumpDown
Alt+shift+down for block-travel.selectDown
WARNING: This should not be considered a long term solution, just a stop gap until this is supported in an easier fashion.
This might not be the greatest idea but it does seem to work for the vim extension I tried in my environment. Probably best to make a request through the in product feedback to get it officially added but until then you can follow these steps.
Upload the .vsix package to your $HOME directory.
Unzip the package into the /google/devshell/editor/theia/plugins directory. This action will not persist so you'll want to add the command to the .customize_environment script actions.
e.g.
sudo unzip vscodevim.vsix -d /google/devshell/editor/theia/plugins/vscode-vim
Now for the questionable part. You'll want to install the pslist package to make life easy so you have access to the rkill command. You probably also want to add this to the .customize_environment file as well since it also will not persist.
sudo apt install pslist
Now we need to get the process id for the editor. Currently this seems to be spawned by a supervisord command, which also spawns the tmux section so we're going to grab the process id that is from the runuser command it spawns (and filter for the theia one just in case).
ps ax | grep runuser | grep "theia start"
Then we can use rkill to kill the process and all of the its children, which will cause supervisord to restart it for us and the plugin should be available.
sudo rkill PID_OF_GREP_OUTPUT
I'm not sure the best way to script the rkill command yet, since I don't know the timing of when it's up vs the .customize_environment execution, so right now I run this each time I start up a new VM.
If anything goes horribly wrong, you should be able to request a restart of the VM from the menu options and get a fresh one.
Cloud Shell offers VS Code editor experience through Theia. Did you try cloud Code editor in the cloud shell? that is exposed through "Open Editor" button on the top right, this will open cloud code editor that gives you VSCode experience. You have all the navigation keys that are available in the editor.
When I run vagrant up I get the following error:
Vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/plugins/hosts/suse/host.rb:20:in `initialize': Cannot translate name. # rb_sysopen - /etc/os-release (Errno::ELOOP)
I have installed Vagrant for Windows and I'm trying to launch Laravel's Homestead that I cloned in WSL2 by cd'ing into the Z: directory that WSL2 provides via PowerShell (so that I have access to Vagrant that's installed on Windows).
cd Z:\home\coder\projects\homestead
It seems that Vagrant is trying to recognize the OS from the filesystem if I'm understanding correctly. So if you're trying to run Vagrant on Windows across a network share that is Unix/WSL/Linux it seems that it will try to run as if it is Unix and fail.
Solution
I was able to copy the homestead directory from the network share into my Windows environment and then navigate to that directory and run vagrant up successfully using powershell.
Another Option
It sounds like you should also be able to install Vagrant within WSL2 and use it from within WSL2 instead of PowerShell.
Another possibility to note is that you can invoke exes from within WSL2, but it sounds like it will not work properly if you were trying to run Window's Vagrant from within WSL2.
Research
https://github.com/roots/trellis/issues/1083
https://www.vagrantup.com/docs/other/wsl.html
https://discourse.roots.io/t/command-vagrant-up-in-wsl-is-failed/16528
I cannot start up a new Meteor application on a Vagrant linux box (running on a Mac). It fails every time with a 'unspecified uncaught exception' in Mongo. I have tried a bunch of things to get this going, but even with the simplest set-up, I cannot get the project running. I would be grateful for any suggestions.
My steps are:
create a completely clean Vagrant box ("ubuntu/trusty64");
install Meteor on the new box (curl https://install.meteor.com/ | sh);
choose a location to create the project;
create a new Meteor project (meteor create app);
start up the project (cd app; meteor)
I know that the permissions on the vagrant shared folder are quirky, so for step #3 above I have tried putting the project:
in the shared guest/host folder, /vagrant,
in a subdirectory of the Vagrant home folder (/home/vagrant),
in a subdirectory of / (with permissions set to vagrant:vagrant), and
in a subdirectory of / with permissions set to root:root, the project created with sudo meteor create app and run with sudo meteor
In all cases, I see this error:
=> Started proxy.
Unexpected mongo exit code 100. Restarting.
Unexpected mongo exit code 100. Restarting.
Unexpected mongo exit code 100. Restarting.
Can't start Mongo server.
MongoDB had an unspecified uncaught exception.
This can be caused by MongoDB being unable to write to a local database.
Check that you have permissions to write to .meteor/local. MongoDB does
not support filesystems like NFS that do not allow file locking.
I cannot tell if this is a Vagrant issue (though I think not, given what I've tried) or a Meteor issue, but I suspect it is Meteor (or one of its many dependencies). I doubt it is a permissions issue, since it failed when running as root. I've tried building meteor from scratch and the build fails and I've tried creating the project with --release 0.9.0 and --release 0.9.2-rc1 and the download is simply killed without explanation.
(1) After step 2 'install Meteor on the new box (curl https://install.meteor.com/ | sh)'
user$ cd /vagrant
user:/vagrant$ meteor create myApp
You should see the myApp folder on your Mac host (the same folder for the vagrantfile)
(2) Insides the myApp folder, you will see the default .meteor folder, make a folder called local if it is no there
user:/vagrant$ cd myApp/.meteor
user:/vagrant/myApp/.meteor$ mkdir local
(3) Create the same folder structure in the /home/vagrant
user:/vagrant/myApp/.meteor$ cd ~
~$mkdir -p myApp/.meteor/local
(4) Link or mount the /vagrant/myApp/.meteor/local to /home/vagrant/myApp/.meteor/local
sudo mount --bind /home/vagrant/myApp/.meteor/local/ /vagrant/myApp/.meteor/local/
or make it permanently
echo “sudo mount --bind /home/vagrant/myApp/.meteor/local/ /vagrant/myApp/.meteor/local/” >> ~/.bashrc && source ~/.bashrc
(5) Now you can start the meteor
~$cd /vagrant/myApp
user:/vagrant/myApp$meteor
The reason why I mount the local folder rather than the <.meteor> folder is that you can still edit the files insides the <.meteor> folder on your Mac host. You can replace myApp with whatever name you want
Hope this help
I'm working with a Windows host, but maybe this will apply to your situation as well.
The only folder which causes the issue is ./meteor/local. If you relocate this with a symlink to be outside of the shared /vagrant folder you should be able to run the meteor app okay.
But, to put a symlink in the shared folder you need to enable symlinks in the VM... which requires starting Vagrant as an admin.
I put together an Vagrantfile with some scripts and instructions here:
https://github.com/ElectronVector/vagrant-meteor
I ran into similar issues trying to run meteor on windows. It seems that mongodb is not able to write in the /vagrant folder. I solved this by doing
sudo mount --bind /home/vagrant/meteorapp/.meteor/ /vagrant/meteorapp/.meteor/
(got that from https://gist.github.com/gabrielhpugliese/5855677)
Here is an answer that solved my problem. Launching meteor project from a shared folder on Debian VMware virtual machine(running on a Windows).
The issue is that mongodb can't create data files inside a shared folder, so in this case just use an existing mongodb for meteor project:
export MONGO_URL=mongodb://localhost:27017/your_db
Doing
vagrant reload --provision
solved my problem.
I think the reason might be some files got corrupted or deleted.
I am taking my first steps with ipython notebook and I installed it successfully on a remote server of mine (over SSH) and I started it using the following command:
ipython notebook --ip='*' ---pylab=inline --port=7777
I then checked on http://myserver.sth:7777/ and the notebook was running just fine. I then wanted to close the SSH connection with the server and keep ipython running in the background. When I did this, I couldn't connect to myserver.sth:7777 anymore. Once I connected again to the remote server by SSH, I could connect again to the notebook. I then tried to use screen to start ipython: I created a new screen by screen -S ipy, I started ipython notebook as above and I used Ctrl+A,D to detach the screen and exit to the TTY. I could still connect remotely to the notebook. I then closed the SSH connection and I got a 404 NOT FOUND error when I tried to access my previously stored notebook and I couldn't see it on the list of notebook at http://myserver.sth:7777/. I tried to create a new notebook, but I got a 500 Internal Server Error.
I also tried running ipython notebook with and without using sudo.
Any ideas?
Rather than use screen, perhaps you could switch to an init script or supervisord to keep IPython notebook up and running.
Let's assume you go the supervisord route:
Install supervisord
Install supervisord using your package manager. For ubuntu it's named supervisor.
apt-get install supervisor
If you decide to install supervisor through pip, you'll have to set up its init.d script yourself.
Write a supervisor configuration file for IPython
The configuration file tells supervisor what to run and how.
After you install supervisor, it should have created /etc/supervisor/supervisord.conf. These lines should exist in the file:
[include]
files = /etc/supervisor/conf.d/*.conf
If they contain these lines, you're in good shape. I only show them to demonstrate where it expects new configuration files. Your configuration file can go there, named something like /etc/supervisor/conf.d/ipynb.conf.
Here's a sample configuration that was generated by Chef by an ipython-notebook-cookbook that runs the notebook in a virtualenv:
[program:ipynb]
command=/home/ipynb/.ipyvirt/bin/ipython notebook --profile=cooked
process_name=%(program_name)s
numprocs=1
numprocs_start=0
autostart=true
autorestart=true
startsecs=1
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=ipynb
redirect_stderr=false
stdout_logfile=AUTO
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
stdout_capture_maxbytes=0
stdout_events_enabled=false
stderr_logfile=AUTO
stderr_logfile_maxbytes=50MB
stderr_logfile_backups=10
stderr_capture_maxbytes=0
stderr_events_enabled=false
environment=HOME="/home/ipynb",SHELL="/bin/bash",USER="ipynb",PATH="/home/ipynb/.ipyvirt/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games",VIRTUAL_ENV="/home/ipynb/.ipyvirt"
directory=/home/ipynb
serverurl=AUTO
The above supervisor config also relies on an IPython notebook configuration (located at /home/ipynb/.ipython/profile_cooked/ipython_notebook_config.py). This makes configuration much easier (as you can also set up your password hash and many other configurables).:
c = get_config()
# Kernel config
# Make matplotlib plots inline
c.IPKernelApp.pylab = 'inline'
# The IP address the notebook server will listen on.
# If set to '*', will listen on all interfaces.
# c.NotebookApp.ip= '127.0.0.1'
c.NotebookApp.ip='*'
# Port to host on (e.g. 8888, the default)
c.NotebookApp.port = 8888 # If you want it on 80, I recommend iptables rules
# Open browser (probably want False)
c.NotebookApp.open_browser = False
Re-read and update, now that you have the configuration file
supervisorctl reread
supervisorctl update
Reality
In reality, I used to use a Chef cookbook to do the entire installation and configuration. However, using configuration management with tiny stuff like this is a bit of overkill (unless you're orchestrating these in automation).
Nowadays I use Docker images for IPython notebook, orchestrating via JupyterHub or tmpnb.