Merging two noteboooks into one in Jupyter (ipython) - jupyter

I have two different Jupyter notebooks for two different projects. I would like to merge them to get them both on the same notebook.
One way is to copy cells from one to the other one but it seems to be quite a bit of work for projects containing many cells. Is there a feature supporting importing Notebook documents or copying multiple cells at a time?

You can merge notebooks with nbmerge
Install it with pip
pip install nbmerge
and merge your files with
nbmerge file_1.ipynb file_2.ipynb file_3.ipynb > merged.ipynb

pip install nbmerge
nbmerge file_1.ipynb file_2.ipynb file_3.ipynb > merged.ipynb
full documentation - https://github.com/jbn/nbmerge

You can indeed copy multiple cells at a time - use Shift + J or Shift + Down to select the next cell in a downwards direction, and in an upwards direction with Shift + K or Shift + Up. You could then copy and paste them en masse to another notebook, and preserve the cell divisions by making sure the destination notebook is in Command mode (cells have blue border).
See http://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Notebook%20Basics.html# and https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/.

install nbmerge:
pip install nbmerge
run the following:
!nbmerge file_1.ipynb file_2.ipynb file_3.ipynb > merged.ipynb
Answer for those who missed to exclaim

I've done this sort of thing a few times in the past using a text editor on the .ipynb file. It's a json format that's fairly easy to follow. My use case was to move a series of cells into another notebook, more or less as an archive activity.
Make sure you have a backup strategy in place in case you break the json format by leaving out a comma or some such!
You end up with an invalid checksum, which you are warned about when you first open the changed notebook(s). After saving them the first time, a new checksum is created and the warning goes away.

Make sure all the notebooks you want to merge are in the same folder/directory and put these code in new jupyter cell (pls don't forget the exclamation mark)
!nbmerge file_1.ipynb file_2.ipynb file_3.ipynb > merged_ouput.ipynb

You should have a look at the following issue on GitHub which tracks this feature. It will likely be made possible via a command line tool. The issue also links to scripts to do that.
Copy-Pasting cells between browser windows is unfortunately a tricky problem, as browser security often prevents getting enough data to do so. It might be possible in some cases using some third party extensions.

Copy/pasting parts of a notebook into another one is a use case for Jupytext. Jupytext can export (the inputs) of your notebooks as .py or .md files. Execute either
jupytext --set-formats ipynb,py --sync *.ipynb
or
jupytext --set-formats ipynb,md --sync *.ipynb
Then edit the text files, move the desired inputs from one file to the other, and update the corresponding .ipynb files with
jupytext --sync *.ipynb
Please note that only the outputs for which the inputs were unchanged are preserved in this operation.
Jupytext is also available as a plugin for Jupyter. In that context, you don't need to use the command line. You can pair your notebook to the desired text file using the File/Jupytext menu (Jupyter Notebook) or the Jupytext commands (Jupyter Lab). And you just have to refresh your paired notebook in the browser to get the latest edits on the paired text file.

First, create a new Jupyter Notebook in the same directory as the previous notebooks you wish to merge.
In the new Jupyter Notebook, run this: pip install nbmerge .
Then, restart your Kernel.
Finally, run this:
!nbmerge your_first_file.ipynb your_second_file.ipynb your_third_file.ipynb -o merged.ipynb
Check your directory for the newly created merged file (merged.ipynb)

You would first have to create a new Notebook in the same directory as the notebooks you wish to merge.
then type the following:
pip install nbmerge
import nbmerge
!nbmerge name1.ipynb name2.ipynb name3.ipynb -o new_name.ipynb
This should work

Related

Where are %stored Jupyter files located?

The %store magic feature can save large python objects for you to use in between different sessions, but I'd like to find out where the files are actually located so I can also transfer them between different computers. I am using Windows 7.
IPython does that, see github for storemagic.py
Stored value will be kept under %IPYTHONDIR/<profile_name>/db/autorestore/<variable_name>
Or by default ~/.ipython/profile_default/db/autorestore/<variable_name>

Where should I place my settings and profiles for use with IPython/Jupyter 4.0?

I've recently updated IPython (to 4.0) and have started using Notebooks after a period of some time not using them (since before The Big Split, I think), and have discovered that some of my settings need to be modified and moved from ~/.ipython/ to ~/.jupyter/.
For example, it appears that the NotebookManager.notebook_dir in
~/.ipython/profile_default/ipython_notebook_config.py
is ignored and has had its functionality replaced by FileContentsManager.root_dir in
~/.jupyter/jupyter_notebook_config.py
Beyond isolated incidents like this, however, it's not clear to me how the configuration settings and profiles now work. In particular what files should be located where, and what they should contain. If, for example, I generate a new profile with
ipython profile create foo
I now get different files than I used to (and than those present in my existing profiles): where I used to get
ipython_config
ipython_notebook_config.py
ipython_nbconvert_config.py
I now get
ipython_config.py
ipython_kernel_config.py
and as near as I can tell, none of my ipython_notebook_config.py are used.
This (and the limited documentation) all leaves me a bit confused about where should I place my settings and profiles for use with IPython/Jupyter 4.0, specifically:
Where should I place general notebook-related settings for use by all kernels/profiles? (I see that some go in ~/.jupyter/jupyter_notebook_config.py, but it's not clear to me whether these would override settings loaded from elsewhere.)
Where should I place profile-specific settings for use with notebooks? (Can I continue to use ipython_notebook_config.py files in ~/.ipython/profile_someprofile/ directories?)
Where should I place kernel-specific settings for use by all profiles and applications. (For example, where should settings used by IPython for all profiles in both the shell and notebooks go?)
How should I modify or update existing IPython profiles for use with 4.0.0? (For example, should all have a ipython_kernel_config.py file; should I delete the ipython_notebook_config.py and ipython_nbconvert_config.py files?)
OS X 10.10.4; Xcode 6.4; CLT: 6.4.0.0.1; Clang: 6.1; Python Python 2.7.10 (Homebrew); Jupyter 4.0.4; IPython 4.0.0.
First of all, check what's in your ~/.jupyter/ folder. Some of the comments under the question mention you have a file named "MIGRATED" that contains only a timestamp.
If you are in this state, run the command: jupyter notebook --generate-config which will create a new file ~/.jupyter/jupyter_notebook_config.py. If you already have this file, you don't need to run this command.
This file is where you put the code that will modify your IPython Notebook behavior. It's a fully commented out (and well documented) file.
I have a related question open (How do I get IPython profile behavior from Jupyter 4.x?) that deals with attempting to set up multiple profiles, which is currently unanswered.

PyCharm - automatically set environment variables

I'm using virtualenv, virtualenvwrapper and PyCharm.
I have a postactivate script that runs an "export" command to apply the environment variables needed for each project, so when I run "workon X", the variables are ready for me.
However, when working with PyCharm I can't seem to get it to use those variables by running the postactivate file (in the "before launch" setting). I have to manually enter each environment variable in the Run/Debug configuration window.
Is there any way to automatically set environment variables within PyCharm? Or do I have to do this manually for every new project and variable change?
I was looking for a way to do this today and stumbled across another variation of the same question (linked below) and left my solution there although it seems to be useful for this question as well. They're handling loading the environment variables in the code itself.
Given that this is mainly a problem while in development, I prefer this approach:
Open a terminal
Assuming virtualenvwrapper is being used, activate the virtualenv of the project which will cause the hooks to run and set the environment variables (assuming you're setting them in, say, the postactivate hook)
Launch PyCharm from this command line.
Pycharm will then have access to the environment variables. Likely because of something having to do with the PyCharm process being a child of the shell.
https://stackoverflow.com/a/30374246/4924748
I have same problem.
Trying to maintain environment variables through UI is a tedious job.
It seems pycharm only load env variables through bash_profile once when it startup.
After that, any export or trying to run a before job to change bash_profile is useless
wondering when will pycharm team improve this
In my case, my workaround for remote interpreter works better than local,
since I can modify /etc/environment and reboot the vm
for local interpreter, the best solution I can do are these:
1. Create a template Run/Debug config template and clone it
If your env variables are stable, this is a simple solution for creating diff config with same env variables without re-typing them.
create the template config, enter the env variables you need.
clone them
see picture
2. Change your script
Maybe add some code by using os.environ[] = value at your main script
but I don't want to do this, it change my product code and might be accidentally committed
Hope someone could give better answer, I've been spent too much time on this issue...
Another hack solution, but a straightforward one that, for my purposes, suffices. Note that while this is particular to Ubuntu (and presumably Mint) linux, there might be something of use for Mac as well.
What I do is add a line to the launch script (pycharm.sh) that sources the needed environment variables (in my case I was running into problems w/ cx_Oracle in Pycharm that weren't otherwise affecting scripts run at command line). If you keep environment variables in a file called, for example, .env_local that's in your home directory, you can add the following line to pycharm.sh:
. $HOME/.env_local
Two important things to note here with respect to why I specifically use '.' (rather than 'source') and why I use '$HOME' rather than '~', which in bash are effectively interchangeable. 1) I noticed that pycharm.sh uses the #!/bin/sh, and I realized that in Ubuntu, sh now points to dash (rather than bash). 2) dash, as it turns out, doesn't have the source "builtin", nor will ~ resolve to your home dir.
I also realize that every time I upgrade PyCharm, I'll have to modify the pycharm.sh file, so this isn't ideal. Still beats having to manage the run configurations! Hope it helps.
OK, I found better workaround!
1.install fabric in your virtualenv
go to terminal and
1. workon your virtualenv name
2. pip install fabric
2. add fabric.py
add a python file and named it "fabric.py" under your project root, past the code below,and change the path variables to your own
from fabric.api import *
import os
path_to_your_export_script = '/Users/freddyTan/workspace/test.sh'
# here is where you put your virtualenvwrapper environment export script
# could be .bash_profile or .bashrc depend on how you setup your vertualenvwrapper
path_to_your_bash_file = '/Users/freddyTan/.bash_profile'
def run_python(py_path, virtualenv_path):
# get virtualenv folder, parent of bin
virtualenv_path = os.path.dirname(virtualenv_path)
# get virtualenv name
virtualenv_name = os.path.basename(virtualenv_path)
with hide('running'), settings(warn_only=True):
with prefix('source %s' % path_to_your_export_script):
with prefix('source %s' % path_to_your_bash_file):
with prefix('workon %s' % virtualenv_name):
local('python %s' % py_path)
3. add a external tool
go to
preference-> External tools -> click add button
and fill in following info
Name: whatever
Group: whatever
Program: "path to your virtualenv, should be under '$HOME/.virtualenvs' by default"/bin/fab
Parameter: run_python:py_path=$FilePath$,virtualenv_path=$PyInterpreterDirectory$
Working directory: $ProjectFileDir$
screenshot
wolla, run it
go to your main.py, right click, find the external name (ex. "whatever"), and click it
you could also add shortcut for this external tool
screenshot
drawbacks
this only work on python 2.x, because fabric don't support python 3

Running IPython Notebook viewer locally

Im trying to introduce IPython notebook in my work. One of the ways I want to do that is by sharing my own work as notebooks with my colleagues so they would be able to see how easy it is to create sophisticated reports and share them.
I obviously can't use Notebook viewer since most of our work is confidential. I'm trying to set up notebook viewer locally. I read this question and followed the instructions there, but now that nbconvert is part of IPython the instructions are no longer valid.
Can anybody help with that?
You have a couple of options:
As described above convert to HTML and then serve them using a Simple server e.g python -m "SimpleHTTPServer" You can even set up a little python script that would "listen" in one directory. If changes or new notebooks is added to the directory the script will run nbconvert and move the HTML file to the folder you are serving from. To navigate to the server you are running go to yourip:port e.g. 10.0.0.2:8888 (see the IPython output when you run the IPython notebook command) (If you can serve over the network you might just as wel look into point 2 below)
If your computers are networked you can serve your work over the lan by sharing your IP address and port with your colleagues. This will however give them editing access but should not be a problem? This means that they will navigate to your ipython server and see the ipython notebook and be able to run your files.
Host your notebooks on an online server like Linode etc... entry level servers cheap. Some work is needed to add a password though.
Convert to PDF and mail it to them.
Convert to a slideshow (now possible in Version 1.00) and serve via option 1,2 or just share the HTML file with them.
Let them all run ipython notebook and check your files into a private repo at bitbucket (its free private git repo). They can then get your files there and run it themselves on their own machines.Or just mail it to them. Better yet if they wont make changes share a dropbox folder with everyone. If they run ipython notebook in that folder they will see your files (DANGEROUS though)
Get them in a boardroom and show them. :)

How to combine all the packages of the eclipse?

As I need to develop java, php, c++, I use several eclipse packages.
so I try to simple to copy several packages into one, and then I use these option to start the eclipse:
1. -clean
2. osgi.checkConfiguration=true
but it doesn't work, I only find the last copied packages after the IDE start.
Get one of the packages from their site and use the Install new software... feature under the help menu.
I think the most straightforward way to get the modules for all of the languages to play nicely together, though I admire your DIY spirit.
Try to use Yoxos for building your custom Eclipse distribution.
From luxsie:
How to merge several setup packages of Eclipse?
Whom this is for?
The ones that can not hold back their temper any longer when trying to install another Eclipse function package. Right the installation speed from some countries is too slow to bear, also the Equinox P2 always try to connect the Download.Eclipse.org. Much to damn.. when your Internet connection closed or reset all of a sudden and got all downloaded files broken. Also OSGi based Eclipse plugins is chaos -- always have some conflict errors.
Oh that is another problem..
Do the following steps:
1 Download the Install Packages that you need from www.eclipse.org. Also please remember verify to see if that is correct package. I choosed J2EE with C/C++.
2 Unpack one package with uncompress softwares, such as 7-zip and WinRAR. unzip all the files to a directory you wanna install to. for example "D:\Eclipse\".
3 open Configurations. Files "\configuration\org.eclipse.equinox.source\source.info" and "\configuration\org.eclipse.equinox.simpleconfigurator\bundles.info" in the installation directory is the ones that need to be merged. also you need "\configuration\org.eclipse.update\platform.xml".
4 open the other packages and fetch their configuration files, and merge files.
to Bundles.info:
Open the bundle.info with any Functional Text Editor, copy all text begin from the line after "#version={Number}" and paste it to another. [{Number} means any Integer number.]
to Source.info:
Similiar with what you did with the bundle.info. If not modified, that line should be the 3rd line in the text file.
to Platform.xml:
Open the file, then find where "feature id=" starts just after the "site" node. Find the "/site" tag and copy all the text between them, paste to another file just before the "/site" tag.
[You can do the similiar also marked feature tags with artifacts.xml]
5 when merging i suggest you to make a new directory and when finished please remember copy the merged file to the one where should it be and overwrite. [I mean where it comes from]
Although not merge "platform.xml" will not cause any functional errors, it will make the About dialog with no button -- to ugly yeah?
[= =|||]Ugly is caused by the artifacts.xml in the installation directory...
6 Open all the package, find "plugins" and "features" directories.now extract!
first extract the one you want most. I extracted JEE package.
then extract the other ones. I don't see any problem whether Overwrite the ones or not.
7 do open a console window, and locale in the installation directory, and then execute "eclipse.exe".
Well, somebody ask me why i paid no attention to the Artifacts.xml in the installation directory. That's because Eclipse will never check the file. It seems to be when you want to update using zip files -- however this way is blocked for lacking site.xml file now.
Then guys, wait a several minutes for eclipse to do some sorting and cleaning job for the merged configurations.. and install your plugins!
INFO: if you face some problems after install a new plugin and restart.. that is because you haven't finish a complete artifacts.xml merging.
You may delete all the artifacts.xml 's header [document start to the "artifacts size = '{Number}'"] and its bottom [from "/artifacts" to document end]. just merge the left content. and make one file just contains the header and bottom, paste the merged one in.
Eh..maybe you can calculate the {Number}s' sum and correct the one in the final document.