I have some very large IPython (1.0) notebooks, which I find very unhandy to work with. I want to split the large notebook into several smaller ones, each covering a specific part of my analysis. However, the notebooks need to share data and (unpickleable) objects.
Now, I want these notebooks to connect to the same kernel. How do I do this? How can I change the kernel to which a notebook is connected? (And any ideas how to automate this step?)
I don't want to use the parallel computing mechanism (which would be a trivial solution), because it would add much code overhead in my case.
When I have a long noetbook, I create functions from my code, and hide it into python modules, which I then import in the notebook.
So that I can have huge chunk of code hidden on the background, and my notebook smaller for handier manipulation.
Related
I get that Canopy "is its own ecosystem" as my friend told me (when he explained to me that I didn't have Python on my computer and that's why my command line instructions weren't doing anything). Could this be the reason that it takes longer to load than Eclipse, which (I guess) I downloaded separately from Java? Or is it some other reason?
Good question! Fast loading was not a design goal of Canopy. Most Canopy users leave it open for many hours at a time, so load time (while always nice, of course!) was not as high a priority as having it include a robust scientific Python distribution. The technical reason why it loads slowly (especially when the OS's cache is "cold") is that it opens and interprets many tens of Python modules, some quite complex. Unlike C, Python is usually not compiled, so the startup time of Python programs is typically slower, though when important, there are ways to counteract that.
What options are there for interfacing with / controlling ImageJ/Fiji from another program?
I need to use some image filters that are available in Fiji. However, I cannot have Fiji be the centre of my workflow. I am using another system (scripting language) for that. I must manipulate some data in that other system, apply a filter to an image, get the result, then continue manipulating that image.
What are my options? What features does Fiji provide that make this possible?
In particular, is it possible to do this if the system I am using does not have a Java interface? It does have a C interface, and it can invoke command line programs.
For context: I want to interface Fiji with Mathematica. Mathematica does have a Java interface but it appears to lack support for some Java 8 features that ImageJ requires. I also don't know much Java and I am looking for a simple solution that I can set up quickly, rather than a long-term robust solution that requires a high initial investment.
If you can't run the ImageJ filters through Java integration, you may be able to write the necessary operations into an .igm macro and then get ImageJ to run it in headless mode.
Otherwise, it may be simpler to port the Java code for the particular filters that you need into a more convenient environment. ImageJ is primarily set up for GUI use, and Java/CLI/other integration can get messy very quickly.
Are there any tools to monitor network's training in PyTorch? Like tensorboard in tensorflow.
PyTorch 1.1.0 supports TensorBoard natively with torch.utils.tensorboard. The API is very similar to tensorboardX. See the documentation for more details.
I am using tensorboardX. It supports most (if not all) of the features of TensorBoard. I am using the Scalar, Images, Distributions, Histograms and Text. I haven't tried the rest, like audio and graph, but the repo also contains examples for those use cases. The installation can be done easily with pip. It's all explained in the README file of the repo.
There are also other github repos which implement a wrapper for PyTorch (and other languages/frameworks) to tensorboard. As far as I know they support fewer functionalities. But have a look at:
Crayon
Tensorboard-Logger
I have asked this question before in the forums. Tensorboard seems very convenient for Tensorflow and it is also made part of the library/framework itself. However, PyTorch wouldn't take the same approach. But there is a library called visdom here that is released by Facebook, that helps you log the training information. This gives you the flexibility of logging information the way you want. While this means a lot of flexibility, it also means you need to write some extra code to make things work.
Following up on blckbird's answer, I'm also a big fan of Tensorboard-PyTorch. However I also found that its API is relatively low level and I was writing a lot of similar code over and over to do the logging. So (shameless plug) I've written a small package on top of it to automate monitoring network training experiments with minimal code. Hopefully someone else finds it helpful. pytorch-monitor
Minetorch helps me a lot at the past 2 Kaggle competitions. I think it's ready for others to use. It has built-in tensorboard or matplotlib supported. And many other features which make the work easy, includes:
Logger
Tensorboard supported
Matplotlib (to generate png to file)
Auto resume training
Auto best model saving
Hook points for customize
...
It's still in developing so any issues or PRs are very welcomed : )
I have recently found a Matlab program (TraitLab) that I would like to try and use, but I do not have a Matlab license. Given that I thought GNU Octave and Matlab are mostly compatible, I tried to run it in octave, but the GUI way to run the program depends on loading a .fig file (now I know that I can use load to get the struct content of that file, but that does not make it a figure I can pass around to various GUI functions), and the headless way to run it reports many parse errors about
nested functions not implemented in this context
Is there an obvious way to get a function written for Matlab that shows these two symptoms to run under Octave?
I have downloaded TraitLab and it is obvious to me that it is a program that will need heavy modification for you to run on octave, which you probably don't have time (or possibly expert knowledge) to implement.
In this case you really only have two other options.
1) Write to the developers and ask them if they could re-implement their code in such a way to make it Octave compatible. I wouldn't hold my breath since it's an academic library, but it's always worth pointing out that there is demand for Octave compatible code. Also given this is old code, it's still worth contacting the stats lab in oxford, since they may have moved to a different codebase that might also suit your needs (e.g. I know that a lot of stats people in oxford have made the switch to julia).
2) In the meantime, if you really want to run this but you don't have matlab, find a matlab installation (e.g. from a friend, or at your university) and create a standalone-version using Matlab's "Application Compiler".
OR
Consider buying matlab. It has a reputation for being prohibitively expensive, partly because University-wide licences do cost an arm and a leg. But for personal use (especially if you are a student?), with the right licence and only the necessary toolkits, it's actually not as prohibitively expensive as you might think.
i am new in linux and using ubuntu'10.10'...i basically want to know as what is the significance of learning shell....????...i googled but max time i find tutorial but didnot get any satisfactory answers...
Whenever you want to automate small tasks where using real programming languages is overkill (e.g. batch convert files, rip DVD chapters, synchronize directories, ...) shell scripts often are invaluable.
Moreover, a lot of "plumbing" in the system is done with shell scripts (e.g. daemons start/stop scripts, event scripts), so knowing at least how to read them often can give you a grasp on how some parts of the system work under the hood, and give you a clue about where the problem can lie in case of malfunctions; knowing how to write them can also be useful if you need to customize the system in some way.
That is a nice question to ask. The fact is that studying shell script help you to do a lot of tasks, (well it helped me ). I used shell script whenever i had to perform more than one Linux commands one by one and sometimes when I had to run jobs over and over again with same pattern. Shell really helps in doing such tasks.
And off the syllabus, studying shell for tasks made me more familiar with the OS and commands.