I'm developing a distributed application using IPython parallel. There are several tasks which are carried out one after another on the IPython cluster engines.
One of these tasks inevitably makes use of closures. Hence, I have to tell IPython to use Dill instead of Pickle by calling dv.use_dill(). Though this should be temporarily.
Is there any way to activate Pickle again once Dill is enabled? I couldn't find any function (something of the form dv.use_pickle()) which would make such an option explicit.
I'm the dill author. I don't know if IPython does anything unusual, but you can revert to pickle if you like through dill directly with dill.extend(False)… although this is a relatively new feature (not yet in a stable release).
If IPython doesn't have a dv.use_pickle() (it doesn't at the moment), it should… and could just use the above to do it.
Related
With the rise of Spark, Scala has gained a tremendous momentum as programming language of choice for data science applications.
To increase the efficiency when working on data science applications, specialized IDEs have been released for
R (e.g. RStudio) and
Python (e.g. Spyder or Rodeo, see Is there something like RStudio for Python?).
Is there something similar for Scala?
Unfortunately there doesn't seem to be any dedicated Data Science IDEs for Scala at this time. I think these would be your best options:
IntelliJ Worksheets:
This is basically a text editor with an output window which gets updated as often as you want. Eclipse has something similar, I just prefer IntelliJ.
Pros:
Backed by IntelliJ's fantastic code completion, error checking, and sbt/maven integration.
You can prototype within the same project setup as your actual development system (if you have one).
Cons:
I am not aware of any caching/selective evaluation so the entire worksheet is evaluated each time you want an answer, something you may not want if you have some operations which take a long time to complete.
No workspace variables window or plot integration.
Jupyter Notebooks
The Jupyter Notebook is a generalization of the iPython notebook which now supports dozens of interpreted languages (new kernels are being added all of the time).
Pros:
Scala and Spark Scala Kernels are fairly easy to install, both have the ability to add maven/sbt dependencies and JARs.
The cells in the notebook can be run individually (allowing you to train a model once and use it many times, for example).
The cells support markdown (with LaTeX!) which can be rendered on its own (a github example), allowing you to use your notebooks as a report/demonstration.
Notebooks are backed by a Notebook Server so you could easily use a more powerful computer as your notebook server and then interact with the notebook from another location.
Some kernels have autocompletion.
Looks like there is some plot integration (example) but it is not totally polished.
Cons:
Not all kernels are perfect, some have bugs or limited functionality.
No workspace variables window.
You really need to be careful about the ordering of your cells, failure to do so can cause a lot of confusion.
For most of the data-sciency stuff I do I use Jupyter but it is far from perfect. In order for Scala to really take over as a Data Science language it really needs more data science libraries (scikit-learn is sooo far ahead here) and it needs a solid plotting library (there are a few options but none I have seen both use idiomatic Scala and are able to run without a server). I think as soon as it has those two elements it will become more popular and hopefully someone will make a nice RStudio-esque IDE.
Your best shot (nothing like rstudio but this would be your best shot for scala) is apache zeppelin
I would recommend you to look at Scala IDE for Eclipse. But i think, it really depends on your personal choice in which you are comfortable writing the code. For testing code by code, i would still use jupyter notebook
Is QtConsole developed by the ipython team?
I always assumed it was a nice little widget provided from Qt, implemented by iPy. If Qt merely provides a base widget, and all the terminal magic is part of the iPy effort, that changes things. Like seriously.
Im stuck in Windows for the foreseeable months and would like to help what I can. If QtConsole can polish a few minor issues and provide another moderate feature or two, I think it would be something to get pretty excited about if you are command-line junky with Windows as a roomate.
QtConsole is part of IPython. It is built on the QtRichTextWidget, but all of the console / terminally stuff is implemented in IPython.
I am planning to port an existing application (or at least part of it where we process data to create graphs interactively) into an ipython based UI. I am wondering if it is possible to create a menu based app using ipython notebook as an engine. Any functionality to create menu-based applications in Ipython? From my experience with Ipython so far, I guess this is not available.
I am thinking of mimicking it by creating html code in markdown cells that will produce menus as select lists, choosing and submitting from there would call some cgi on a server that would update lower parts of the notebook using AJAX. Anyone did similar stuff?
Nothing prevent you from reusing the component.
We try to make them as reusable as possible and is should be easy to use our javascript to create your own js frontend. cf #minrk example here.
If some modification make component more standalone and reusable Patches are welcomed. at some point we might even have each component (codecell, tooltip, completer) installable with bower/component.io/whatever
I would recommend not to add menu through javascript in markdown cell as it will be disable soon.
You might want to have a look at Exhibitionist that uses ipython notebook for some noce stuff.
When developing a Node.js application in Eclipse, you usually import your own modules with functionality tied to exports (append functions) or module.exports (allows popular object literal notation).
But how do you set this up for code completion in files where you import your module?
Module
As you can see in the outliner, Eclipse is "aware" of the function:
Yet importing the module as tools doesn't make it's functions available:
I also tried the oldskool //#import tools.js but it doesn't seem to work like this.
How do I get code completion for my own functions in my project?
#Jey Keu: this question has some suggestions towards our mutual dream of code completion, namely:
Amateras plugin
JSDT editor
Aptana Plugin
SPKET plugin
VJET plugin
But it was closed by a bunch of people who know nothing about Eclipse. Profiling is the basis of code completion. But they think profiling is unicorns and rainbows and therefore not constructive.
I think, depending on your specific development needs, that VJET is your best bet. If you can get it to work without destroying every other function within Eclipse, please share your wizardry in this question.
http://eclipse.org/vjet/
Can go run dynamically in order to be used for a plugin based application ?
In eclipse, we can create some plugins that Eclipse can run dynamically.
Would the same thing be possible in Go ?
I'll argue that those are two separate problems :
having dynamic load
having plugins
The first one is simply no : A Go program is statically linked, which means you can't add code to a running program. And which also means you must compile the program to let it integrate plugins.
Fortunately, you can define a program accepting plugins in Go as in most languages, and Go, with interfaces and fast compilation doesn't make that task hard.
Here are two possible approaches :
Solution 1 : Plugin integrated in the main program
Similarly to Eclipse plugins, we can integrate the "plugins" in the main program memory, by simply recompiling the program. In this sense we can for example say that database drivers are plugins.
This may not feel as simple as in Java, as you must have a recompilation and you must in some point of your code import the "plugin" (see how it's done for database drivers) but, given the standardization of Go regarding directories and imports, it seems easy to handle that with a simple makefile importing the plugin and recompiling the application.
Given the ease and speed of compilation in Go, and the standardization of package structure, this seems to me to be a very viable solution.
Solution 2 : Separate process
It's especially easy in Go to communicate and to handle asynchronous calls. Which means you could define a solution based on many process communicating by named pipes (or any networking solution). Note that there is a rpc package in Go. This would probably be efficient enough for most programs and the main program would be able to start and stop the plugin processes. This could very well feel similar to what you have in Eclipse with the added benefits of memory space protection.
A last note from somebody who wrote several Eclipse plugins : you don't want that mess; keep it simple.
Go 1.8 supports plugins (to be released soon Feb 2017.)
https://tip.golang.org/pkg/plugin/
As dystroy already said, it's not possible to load packages at runtime.
In the future (or today with limitations) it may be possible to have this feature with projects like go-eval, which is "the beginning of an interpreter for Go".
A few packages I found to do this:
https://golang.org/pkg/net/rpc/
https://github.com/hashicorp/go-plugin
https://github.com/natefinch/pie