At the beginning of my IJulia notebook, I have the following preamble that imports several libraries and puts them on all processors:
addprocs(4)
import PyCall
#everywhere using PyCall
#everywhere #pyimport numpy as np
#everywhere #pyimport scipy as sc
.
.
.
The format is from the excepted answer to one of my previous questions. What I want to do now is add the processors and import all my packages in one line. However, I am not sure what would be the most efficient way to do this. I am collaborating with several people on different scientific projects, and I would prefer this importing process to be as simple as possible. Should I make a separate file calling these libraries and then run that file, or should I make my own package that does the above--i.e., define a package that automatically adds the processors, imports the packages, and puts them on all processors?
Related
It seems basic but from what I see on databricks website, nothing works on my side
I have installed koalas package on my cluster
But when I try to import the package in my Scala notebook, I have issue.
command-3313152839336470:1: error: not found: value databricks
import databricks.koalas
If I do it in Python, everything works fine
Details cluster & notebook
Thanks for your help
Matt
Koalas is a Python package, which mimics the Pandas (another Python package) interfaces. Currently no Scala version is published, even though the project may contain some Scala code. The goal of Koalas is to provide a drop-in replacement for Pandas, to make use of the distributed nature of Apache Spark. Since Pandas is only available on Python I don't expect a direct of port on this in Scala.
https://github.com/databricks/koalas
For Scala your best bet is to use the DataSet and DataFrame APIs of Spark:
https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/Dataset.html
https://databricks.com/blog/2016/01/04/introducing-apache-spark-datasets.html
When using the scala kernel with Vegas we see the nice charts
But when switching to the scala-spark kernel the imports no longer work:
What is the way to fix the imports for the spark kernel?
As described here you'll probably need to tweak your notebook config to pre-load those libraries, so they are available at runtime.
Then you can do a normal import (without the funny $ivy syntax, which actually comes from Ammonite REPL).
I am searching to do a little programm with 3D animations for one homework for my school.
I work in Spyder space from Anaconda(Python) .
I want to import for that the module vpython installed by pip.
The first line of the function " from visual import * " doesn't work whereas in all examples found on the web, all the programms begins by this line for calling the whole function of this module. Also, i have seen different path and files "vpython" installes on my PC in the path Anaconda3.
So, for me, python don't arrive to find the good path with "import".
Have I to uninstall and install again Python and Anaconda or this is just a problem of grammary from me on my programm and i must help the function import to find the good path, writing some new instructions?
Thanks to have read and i wish you have one idea of what is wrong in this because for the moment I'm blocked.
In Jupyter/iPython/Anaconda, the import should be
from vpython import *
In classic VPython, it's
from visual import *
I am using IPython and Cython.
I am editing my Cython functions in modules in an external text editor.
I would like to import these modules and use them in IPython, but with IPython compiling them when importing.
What is the syntax for doing this? I do not want my code inside my IPython notebooks.
This is an unusual workflow but it should be possible to get something working. Firstly, for things to be importable in your IPython session, they must appear in sys.path. You can insert new folders in that list like this:
The path you would be adding would be the folder(s) in which your compiled Cython modules would be placed. Whichever of the next strategies you use, at a minimum the path to your compiled (.pyd) Cython modules needs to be on sys.path to be able to be imported.
Secondly, you need a way to compile your Cython modules. If you're using the recommended approach, you would have a setup.py file with your cython sources and python setup.py build_ext to produce the compiled .pyd files. You would have to restart your IPython Kernel every time you recompiled your Cython modules.
There are alternatives to setup.py. One of them is my easycython which is a command-line tool that compiles .pyx source into .pyd without needing a setup.py. This tool is AFAIK only used/tested by me, so it may not work, ymmv etc. You would still have to restart your IPython Kernel every time you recompiled your Cython modules. (I mention it here only because it is one of my babies.)
A better approach would be to use pyximport, because it will compile on demand and has reload support:
# This is inside your IPython Notebook
import pyximport
pyximport.install(reload_support=True)
import my_cython_module
You can reload the module with
reload(my_cython_module)
You could try to be clever with some kind of logic to make it so that simply rerunning the notebook will either reload or import:
# Speculative code: I have not tried this!
# This is inside your IPython Notebook
import pyximport
pyximport.install(reload_support=True)
if 'my_cython_module' in dir(): # Maybe sys.modules is better?
reload(my_cython_module)
else:
import my_cython_module
You may have to play around a bit to find something that works, but something should be workable for your use-case.
I'm trying to use a python package from IronPython.
Everything works fine if I import regular python modules.
But when I try to do the following:
import win32ui
I get:
No module named win32ui
I've hunted through the code in IronPython.Runtime.Importer and there's no mention of .pyd
Anyone know a way around this?
You can check out IronClad which is working to provide this support. It may or may not work w/ your PYD of choice.
A .pyd file is a DLL. So unless IronPython (which is written in .net) can correctly load C DLLs written for CPython, you might be out of luck.
Update
In fact, according to the IronPython FAQ, you are unfortunately unable to import .pyd files:
Q: How do I build and call into PYD libraries?
A: IronPython does not support using PYDs built for CPython since they
leverage implementation details of CPython. You can get a similar
effect for new "PYD"s you would like to implement by writing them in C#
or VB and building a DLL for .NET.