synchronizing history when ipython notebook and console are connected to the same kernel - ipython

I have ipython notebook running on a remote server, i.e.
ipython notebook --profile=nbserver
which I access from my local machine. Further, I ssh to the remote server from my machine, and start ipython console (terminal) on that server. I have found following command to work well:
ipython console --existing \
~/.config/ipython/profile_nbserver/security/kernel-*.json
Now I am connected to the same remote kernel from two different clients (lets call them browser and terminal). Everything works well, except one annoying detail:
1) in browser, I type a=1
2) in terminal, I type b=2
3) in both clients I can see both commands using %history. But when I want to cycle through the history (in terminal) using Up, it only shows the commands which have been typed in the terminal, (i.e b=2). Similarly, I am unable to use a + PageDown in the terminal, to go back in history and find the command starting with a.
From what I understand, my two clients are using two separate history files history.sqlite. But why does %history show all commands ?
Question:
Is there any way to configure using one history.sqlite for both clients ?
I find, having easy access to history is absolutely crucial. Moreover, I see using both terminal and browser as complementary, they both have tradeoffs and are best used combined.

You can set where the history gets loaded either by setting it at the terminal:
ipython --HistoryManager.hist_file=$HOME/ipython_hist.sqlite
or within the ipython config files:
import os
c.HistoryManager.hist_file=os.path.expanduser("~/ipython_hist.sqlite")

Related

MongoDB server not starting from vs code terminal

I am a beginner of MERN and while learning Mongodb I used to open my server using my PowerShell by the command mongod but I got to know that I can do the same from my VS code terminal. But unfortunately, it is not starting the server for some reason. There is no error that it prints on the screen. It simply prints some data and doesn't start the server. It is running mongo command perfectly on my terminal. Can anyone tell me why it is starting?
The error got fixed. Actually my project was in D drive and my data folder was into my C drive so whenever I was opening the terminal using mongod command. It used to find the data folder and as it didn't used to find it on D drive. It therefore used to close it up at the same time. Though if it would have shown the error correctly then it would have been better to fix it up.
I used cd C: command to change into the C: drive and then used the command mongod

PSQL_HISTORY ignored by PyCharm

I have a Django project connecting to a PostgreSQL database which I develop in PyCharm, and I want to enable PostgreSQL history logging.
There is PSQL_HISTORY env variable set to /home/user/apps/postgres/logs/.pycharm_log, but when I start the project in PyCharm and update some data via the Django Admin (which certainly hits the database) -- nothing gets logged and the file is not created at all.
Is there a way to make PyCharm and PSQL_HISTORY work together as I expected?
'psql' is the name of a specific client tool. Why would a completely different tool use psql's configuration options? If you want to log every statement sent to the server, you could configure that in the server side with log_statement=all.

breakpoints in eclipse using postgresql

I am using helios Eclipse for debugging my code in postgresql.
My aim is to know how postgresql uses join algorithms during the join query, so I started to debug nodenestloop.c which is in the Executor folder.
I gave break points in that file, But whenever I try to debug that file, the control goes to main.c and never comes back,How do I constraint the control only to that particular file(nodenestloop.c)
Below are the following fields which I gave in Debug configurations of Helios Eclipse.
C/C++ Application - src/backend/postgres and
project - pgsql
I followed the steps given in the following link for running the program.
https://wiki.postgresql.org/wiki/Working_with_Eclipse#
I even uncheked the field "Start on Start up=main" , but When I do that, The step in and Step over buttons are not activated and the following problem has popped up.
Could not save master table to file '/home/ravi/workspace/.metadata/.plugins/org.eclipse.core.resources/.safetable/org.eclipse.core.resources'.
/home/ravi/workspace/.metadata/.plugins/org.eclipse.core.resources/.safetable/org.eclipse.core.resources (Permission denied)
So I started eclipse using sudo, but this time the following error has come in the console of eclipse.
"root" execution of the PostgreSQL server is not permitted.
The server must be started under an unprivileged user ID to prevent
possible system security compromise. See the documentation for
more information on how to properly start the server.
Could any one help me with this.
Thank you
Problem 1: User ID mismatch
Reading between the lines, it sounds like you're trying to debug a PostgreSQL instance that's running as the postgres user, or a different user ID to your own anyway. Hence your attempt to use sudo.
That's painful, especially when using an IDE like Eclipse. With plain gdb you can just sudo the gdb command to the desired uid, e.g. sudo -u postgres -p 12345 to attach to pid 12345 running as user postgres. This will not work with Eclipse. In fact, running it with sudo has probably left your workspace with some messed up file permissions; run:
sudo chown -R ravi /home/ravi/workspace/
to fix file ownership.
If you want to debug processes under other user IDs with Eclipse, you'll need to figure out how to make Eclipse run gdb with sudo. Do not just run all of Eclipse with sudo.
Problem 2: Trying to run PostgreSQL under the control of Eclipse
This:
"root" execution of the PostgreSQL server is not permitted. The server must be started under an unprivileged user ID to prevent possible system security compromise. See the documentation for more information on how to properly start the server.
suggests that you're also attempting to let Eclipse start postgres directly. That's very useful if you're trying to debug the postmaster, but since you're talking about the query planner it's clear you want to debug a particular backend. Launching the postmaster under Eclipse is useless for that, you'll be attached to the wrong process.
I think you probably need to read the documentation on PostgreSQL's internals:
Tour of PostgreSQL Internals
PostgreSQL internals through pictures
Documentation chapter - internals
Doing it right
Here's what you need to do - rough outline, since I've only used Eclipse for Java development and do my C development with vim and gdb:
Compile a debug build of PostgreSQL (compiled with ./configure --enable-debug and preferably also CFLAGS="-ggdb -Og -fno-omit-frame-pointer"). Specify a --prefix within your homedir, like --prefix=$HOME/postgres-debug
Put your debug build's bin directory first on your PATH, e.g. export PATH=$HOME/postgres-debug/bin:$PATH
initdb -U postgres -D $HOME/postgres-debug-data a new instance of PostgreSQL from your debug build
Start the new instance with PGPORT=5599 pg_ctl -D $HOME/postgres-debug-data -l $HOME/postgres-debug-data.log -w start
Connect with PGPORT=5599 psql postgres
Do whatever setup you need to do
Get the backend process ID with SELECT pg_backend_pid() in a psql session. Leave that session open; it's the one you'll be debugging.
Attach Eclipse's debugger to that process ID, using the Eclipse project that contains the PostgreSQL extension source code you're debugging. Make sure Eclipse is configured so it can find the PostgreSQL source code you compiled with too (no idea how to do that, see the manual).
Set any desired breakpoints and resume execution
In the psql session, do whatever you need to do to make your extension run and hit the breakpoint
When execution pauses at the breakpoint in Eclipse, debug as desired.
Basic misunderstandings?
Also, in case you're really confused about how all this works: PostgreSQL is a client/server application. If you are attempting to debug a client program that uses libpq or odbc, and expecting a breakpoint to trigger in some PostgreSQL backend extension code, that is not going to happen. The client application communicates with PostgreSQL over a TCP/IP socket. It's a separate program. gdb cannot set breakpoints in the PostgreSQL server when it's connected to the client, because they are separate programs. If you want to debug the server, you have to attach gdb to the server. PostgreSQL uses one process per connection, so you have to attach gdb to the correct server process. Which is why I said to use SELECT pg_backend_pid() above, and attach to the process ID.
See the internals documentation linked above, and:
PostgreSQL site - coding
PostgreSQL wiki - developer resources
Developer FAQ
Attaching gdb to a backend on linux/bsd/unix
I also faced similar issue and resolved it after some struggle
I misunderstood the following point under Debugging with child processes in the wiki (https://wiki.postgresql.org/wiki/Working_with_Eclipse).
5."Start postmaster & one instant of postgresql client (for creating one new postgres)"
The above step should be performed from terminal by starting postgres server and one client.
Hope this helps
Once this is done then debugger in eclipse needs to be started for C/C++ Attach to Application

ipython: %paste over ssh connection

In ipython >=0.11, the %paste command is required to paste indented commands. However, if I run an ipython shell in a remote terminal, the buffer %paste refers to is on the remote machine rather than the local machine. Is there any way around this?
I think this is exactly what %cpaste is for (I am always forgetting about all the things IPython does). %cpaste enters a state allowing you to paste already formatted or indented code, and it will strip leading indentation and prompts, so you can copy/paste indented code from files, or even from an interactive Python session including leading >>> or In [1] which will be stripped.
Not a brilliant solution, but I think this will work:
Use %autoindent to turn off autoindenting, type if True: and press enter. Then paste your indented code.
I think it should be possible to write an IPython extension to handle this better. I'll try to get round to looking into it. I've made an issue.
If you use IPython a lot, you may want to get the new kernel/client architecture working - it should be possible to tunnel the connections over SSH, so you can use the Qt console on your local machine, talking to a kernel on a server. But that might take a bit of fiddling to get in place.

How to do remote development with Emacs?

I started using emacs as my main editor a few days ago, and I'm in the process of gathering all customizations I need.
The main usage I'm giving it is for development of a C project on a remote Linux machine (RHEL 5). So far I was using plain vim (only as a text editor) and a bunch of shell scripts, and one day I decided I would try emacs. So far I like it.
But as a drawback I see that setting up the environment to run emacs is a long way to go, and instead of using it directly on the development machine (and set up the environment on every machine I use) I would try to just use it from my laptop and do the remote development.
I know about TRAMP but that only edits remote files as far as I know. The big obstacle is running shell commands on the remote machine with a single keystroke (I bound F5 to run a custom compile command on the remote machine's emacs).
How do you run shell commands on a remote machine with a single emacs keystroke?
Do you know any other tips & tricks to do full remote development?
TRAMP supports compilation and debugging on the remote machines (on which the files reside). This should be automatic and relatively seamless. Check out the documentation for remote processes.
Don't worry. You can run anything in Emacs with a single keystroke. Others have noted ways you can do this elegantly with TRAMP, but if that doesn't suit (for whatever reason) you could also just cook up your own function using:
(defun arbitrary-remote-command ()
(interactive)
(shell-command
"ssh user#remotehost arbitrary-command"))
(global-set-key (kbd "<f5>") 'arbitrary-remote-command)
When doing remote development, I use screen which provides multiple virtual remote terminals available at the press of a couple of keys. This is indispensable for me. Not only does it provide virtual terminals, but your session is preserved on the server if your connection drops for any reason.
See the GNU Screen Survival Guide question for more information about getting started with screen.
I usually have one terminal open for source control operations, one for editing, one for compiling (I just have to switch to it and hit UpArrow+Enter), and one for debugging. Depending on the environment and application I'm working on, I may have more or fewer than this, but that's the general layout.
Nice answer from Trey Jackson, but the answer from Matthew Flaschen actually helped me more.
I couldn't find a way to get TRAMP do the compiling, but accidentally I found the command remote-compile.
Now I just add these lines to my .emacs:
(setq remote-compile-user "root")
(setq remote-compile-host "localhost -p 3000") ;This is just when tunneling
(setq compile-command " cd /usr/src/nexus/traffic && ./builder.sh compile ; cat builder.log")
(global-set-key (kbd "<f5>") 'remote-compile)
And before doing anything, set up the ssh connection using ControlMaster and I don't have to type passwords.
Summarizing:
Editing remote files: TRAMP
Multiple terminals saving states:
SCREEN
Define emacs keybindings to execute
remote commands: See Matthew Flaschen
response
Remote compiling: Command
remote-compile with ControlMaster
I think that covers some important actions to do remote-development.
I've tried TRAMP before for a situation like yours, but I too found it failing in strange ways, and slow in general.
I am now using Emacs to edit files mounted over an sshfs mount. This works very well for me.
To mount the remote file system (assuming you have the host set up correctly in .ssh/config and set up public keys to connect without a password)
$ sshfs remote:dir localdir
and edit as you like from then on!
As for compilation, see the other answers, they cover this in full.
I would just ssh into the remote machine and run emacs in terminal mode instead of using TRAMP. TRAMP is really slow when you don't expect slowness, at least in my experience. I have also seen it fail in weird ways, and it is not always obvious how to "fix" it. If you ssh in, you won't be able to use emacs UI stuff, like the menus etc, but since you are used to vi that should not be an issue for you. In fact, you can just turn them off so that your ssh emacs experience is very similar to your local experience (if local != terminal mode) (1).
Using screen may be useful if you care about preserving your session in case the connection drops - apart from that you get similar functionality as virtual terminals with emacs buffers. For example, you can open many shell buffers and run various shell commands in emacs buffers. I use this to run many instances of sqlplus (use rename-buffer to give them all nice, friendly names).
Since you are ssh'd into the remote machine, you don't need to worry about running "remote" shell commands.
On the other hand, I am not sure what you mean by "I see that setting up the environment to run emacs is a long way to go". If you have source control, this should be trivial...(I bet you have a good reason).
(1) turn off menu, tool bar and scroll bars:
(if (fboundp 'menu-bar-mode) (menu-bar-mode -1))
(if (fboundp 'tool-bar-mode) (tool-bar-mode -1))
(if (fboundp 'scroll-bar-mode) (scroll-bar-mode -1))
For me, there is a problem with TRAMP, that is the remote toolchain is too old, and often it's not under my control.
So I wrote a package (ppcompile) to help me develop locally and compiling remotely, and then fixes compilation errors locally with the help of next-error .
Assuming you don't want to run emacs in terminal mode on your remote
machine then it's fairly easy. As you know you can use TRAMP to edit
remote files. I just set compile-command to 'ssh me#remote "cd
/path/to/root && make"' and be done with it.
About tramp it may depends on tramp version you are using.
Personnally I was encoutering lot of problem with default tramp package and plink as backend to connect to remote hosts, so I upgraded tramp to a more recent version and solve numerous issues.
Actually I'm on a windows machine accessing to a linux (RH5) for remote compiling and browsing directories, reading/writing some files, compiling on remote machine works fine.
One thing no one has mentioned is running emacs off the remote machine, with a valid $DISPLAY variable corresponding to your current display.
Obviously you must run an X Server for your local display, but that's not a big deal even on windows [use the X Server built into Cygwin].
This generally requires network latency < 50ms. If you ping that remote box and get something > 50ms, you will be frustrated with this approach.
Below that, works great.
You will need to have SSH map the X port, there are plenty of docs out there on that.
What I've used since about 1994:
cat <<-EOF >> ~/.emacs
(define-key global-map "\e`" 'compile)
(define-key global-map "\e~" 'remote-compile)
(require 'compile)
(setq remote-shell-program "ssh")
EOF
Now ESC-` runs compile, and ESC-~ runs /remote/ compile. It'll ask you for the hostname, and then pre-populate the command. Both compilations share the same history buffer.
It's soooo easy. It leverages a feature that's worked the same for decades, and it doesn't require any other lifting to work -- and it's just text over an ssh, so it's going to be very responsive and efficient over long or slow or congested links. I've found this prevents Fists of Network Rage a bit more.
I have the same needs: virtual machine over Windows running Debian 10 connecting to RHEL7 development machines. and here is my solution.
First of all, I use sshfs to mount my remote home in a way that preserves paths in my local machine : shfs -o allow_other <remote-user-name>#<remote-host>:/home/<remote-host-home> /home/<remote-host-home>
Second, I configure ssh so that I don't need to type password again and again: I can follow the instructions in this link.
Finally, I can launch compilations through M-x compile, I edit the command and enter something like : ssh -C <remote-user-name>#<remote-host> "make -C <build-folder>"
The great thing with it is that you can click on the links when it founds compilation errors and the source files are opened automatically. It is very convenient.
Two cents: if bandwidth is not a big concern, I would give a crack at VNC.