I'm using this amazing IPython notebook. I'm very interested into parallel computing right now and would like to use MPI with IPython (and MPI4py). But I can't start a cluster with
ipcluster start -n 4
on Windows7. I just get back "failed to create process". If I use the notebook and start a cluster in the "Clusters" register it's all working fine. But with cmd (even with admin rights) I just get this message. Same with all attempts of using MPI (MPICH2). All path vars are set. Maybe this problem has no connection to Python at all...
I can't say anything about IPython's parallel features, but if you're having problems with MPI in Windows in general, I would offer these suggestions. I've had quite a few issues in the past in trying to get MPI working in Windows. The most convenient method for me in the past has been to use an OpenMPI Windows binary http://www.open-mpi.org/software/ompi/v1.6/. These are now only available in previous releases. And even then, you might have to try more than one before you find one that works. I don't know why, but the latest didn't work on my machine. The release before that one did, however. After this, you have to call mpicc and mpiexec from the Microsoft Visual Studio Command Prompt or it won't work (without a lot of other stuff).
After you have verified that MPI is working, you can try installing mpi4py separately and see if that works. In my experience, sometimes this has worked fine and sometimes I've had to wrestle with configurations. You might just try your luck with an unofficial, prepackaged binary (for example, http://www.lfd.uci.edu/~gohlke/pythonlibs/).
Hope this helps!
Related
I started getting the following warning repeatedly when running my PyTorch Lightning deep learning scripts, at execution start and then all through the training:
"OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead."
I get them when executing the main.py script; my scripts are publicly available here.
Symptoms:
I don't think it has anything to do with PyTorch Lightning, maybe even PyTorch.
It appeared overnight, so I don't know what could cause it.
It runs fine without those warnings on my PC.
I get the warnings when I run from my M1 Mac.
I use VSCode for both, each up to date.
I use separate miniconda environments.
Thanks for taking the time to reply!
I managed to sort myself out in the end.
I spotted the numba package in my miniconda env, which is a Python compiler and that seemed to be the root of the problem.
It was version 0.55.2 but the last version to date is 0.56.0. Trying to upgrade it via conda or pip didn't work for some reason (the 0.55.2 version couldn't be replaced).
I recreated my env step by step, and noticed this package comes with torch-audiomentations, a package for audio data augmentation for deep learning, under torch, that I use.
Re-installing it had numba version 0.56.0 installed properly, and the warnings disappeared.
That routine is deprecated, meaning that newer standards want you to use another routine. If you can not edit the code, just ignore. For now it's only an info message. You got that message when you switched to a newer OS / compiler version / OpenMP version.
I'm developing a Jupyter Notebook for my team to use to catalogue and analyse some proprietary data. I'm ready to share it with the team for on-going execution and development. The team generally have Windows 10 workstations and are skilled engineers, though not data scientists. No one currently uses Jupyter.
I now realise I might have thoroughly misjudged Jupyter's ability to support this sort of working environment.
Option 1: Individual installations
This is the worst case scenario. Anyone that wants to run or modify the notebook needs to install Jupyter. Anaconda is probably the best way to go, but its a big, ugly, scary install. Worse, every user will have to install and manage additional libraries. Any notebook change that requires a kernel change will have to be manually applied to each installation.
Surely, being client-server, this is not the intention of Jupyter.
Option 2: One server, many clients
The obvious alternative is to host the Jupyter server on a network accessible computer and have all users connect to it with a browser. That way there's only one shared installation to manage and each user just needs a URL to access it.
But there's a gotcha - the server expects the notebook to be on its own file system! So every user will access the same notebook file. This makes version control very problematic - no one can check out their own copy of the notebook for independent edit and commit sessions. Instead, changes will overwrite the only copy, and commits/reverts/diffs will have to be done on the server (or by mounting the server's file system).
Option 3: Server in Docker image, each user runs a container
Docker to the rescue? That way we can build/maintain one server image (and even version control it) and each user only needs to have a Docker engine installed to instantiate the image (which is a friendly 8GB download!!). They connect to their own container which, with a bit of scripting trickery, will be pointing at their own copy of the notebook.
This option only took 20 hours to investigate before discovering that it fundamentally sucks. Working with the kernel is tricky with lots of new skills necessary. But more showstopping - nothing that shows a Qt window will work. The qtconsole we can do without, but part of our notebook shows a File Open dialog and the best way to do that is with a Qt Widget. With the server in a Docker Container expecting an X Windows environment, and the client in a Windows browser, the Widget cannot be shown.
The Qt issue was the last of many, many issues trying to get the Docker option running. Everything from matplotlib to path mapping, from os library calls to ipywidgets needed to be investigated, tweaked, Googled, chopped and changed to work. I'm fairly convinced that these dramas would be on-going.
Conclusion
There are lots of discussions around Jupyter version control. There's lots of options for read-only sharing. And there's even a project for runtime-building a Docker container to provide executable access to a notebook. But there is scant advice on using Jupyter in a team environment.
Given the endless complications when the server is not natively running on the same machine as the client, I'm starting to believe Option 1 is the only sane way to go. Before I go to my colleagues with the crappy news, are there any other suggestions?
Ended up having a fruitful discussion on the Jupyter Google Group and have confirmed that out-of-the-box, Jupyter does not support this sort of working environment. Indeed, crucially, Jupyter expects the server to have a single user.
The most promising DIY solution was firstly to deploy JupyterHub, for two reasons:
It launches a new server instance for each user, preventing any multi-user per server issues.
It prompts for users to identify themselves, allowing different actions to be taken depending on the user.
And secondly, have the server mount each user's file system (or equivalent network architecture), so it can point the user back to their own local files.
I have not implemented this strategy (making do with Option 1 for now!) but it certainly makes sense.
I have problem with running EplSite program. This program is written in Perl. I'm new in Perl. I tried to run it, but Perl command line displays errors. Can you help me? https://sourceforge.net/projects/eplsiteetl/
CODE one of many files: http://pastebin.com/yM9srKGn
Don't waste your time with EplSite ETL. There's no useful documentation. I've wasted two days trying to install it with no success. Maybe something would work if there was some info about using hypertextperl.pl (especially with Apache server on Windows) but link to developer's site is dead.
Just find some other software for ETL, because if you somehow manage to install Eplsite ETL you would have to guess how to use it properly. It's not worth the time and effort.
I saw that there's a project ijython (https://github.com/graphaelli/ijython.git), but it doesn't say what it does and refers to the ipython project page where is no information about the project. The development stopped 4 years ago according to commit messages, so I won't bother with that, exept someone recommends and explains it. I was wondering whether there might a way to acchieve jython to behave like an ipython shell setting up a configuration file of either jython or ipython.
IPython almost support jython kernel cf https://github.com/ipython/ipython/issues/6213 for example (among others). Once the next jython release is out of beta, we'll be happy to include the patches to make it work.
Does anyone know of a good way to do remote administration of a Windows XP machine using just the command line?
At the moment the only things it needs to do is to be able to install applications/patches, and transfer files to and from the machine, and installing registry patches would be nice as well.
Currently we use a horrible hacked together solution that uses NetMeeting, in the past I've thrown together a proof of concept using SSH for windows (at the time windows 2000) but it didn't work to my satisfaction and was pretty buggy. Which was probably the result of the SSH Daemon I was running more then anything.
I'm pretty much open to anything, however a solution using SSH would be ideal since it's already approved for installation in my organization, and it's free. I work in the Canadian Government so anything free is best, and anything that we've already got approved for installation is even better.
psexec will allow you to run commands remotely. Some of the other PsTools can help you kill applications, get a list of processes, etc.
Why must it be
remote administration of a Windows XP machine using just the command line?
I think your very limiting yourself to what is possible by sticking to the command line. In windows environments you can easily use Group Policy to distribute most software and/or patches, and for the ones that you can't you can usually script these changes through any of the popular scripting languages such as JScript, VBScript, Kixtart, AutoIt, Powershell, etc. With these scripting languages you can easily leverage WMI to exceute and mointor processes on remotes systems, copy files, updates registry...basically everything that you're trying to accomplish....and it won't cost you anything but the cost of learning these technologies, and there many online resources and which document how to do them. Here is a link to the Microsoft Script Center, its a great start: http://www.microsoft.com/technet/scriptcenter/default.mspx
I wrote this a while back, and used it to maintain my home windows XP desktop for a while:
ssh and telnet on windows
I used the SSH option (not telnet). It worked for my purposes (killing remote tasks, copying files etc.) It uses Cygwin, but you're able to run regular windows commands as well as the bash commands that come with cygwin.
The Software Testing Automation Framework (STAF) is designed for remote access, installing software, transferring files. etc. It's open source and you can write your own service if there isn't one that does what you need. It also has a GUI component for writing, scheduling, queueing and monitoring jobs across a pool of machines.
At the moment the only things it needs to do is to be able to install
applications/patches, and transfer files to and from the machine, and
installing registry patches would be nice as well.
try to download and install eurysco to use in order:
transfer file, applications and patches with multiple-upload feature from eurysco file browser
install applications and patches in silent mode from eurysco command line feature
edit registry from eurysco system-registry feature
http://www.eurysco.com/features