breakpoints in eclipse using postgresql - eclipse

I am using helios Eclipse for debugging my code in postgresql.
My aim is to know how postgresql uses join algorithms during the join query, so I started to debug nodenestloop.c which is in the Executor folder.
I gave break points in that file, But whenever I try to debug that file, the control goes to main.c and never comes back,How do I constraint the control only to that particular file(nodenestloop.c)
Below are the following fields which I gave in Debug configurations of Helios Eclipse.
C/C++ Application - src/backend/postgres and
project - pgsql
I followed the steps given in the following link for running the program.
https://wiki.postgresql.org/wiki/Working_with_Eclipse#
I even uncheked the field "Start on Start up=main" , but When I do that, The step in and Step over buttons are not activated and the following problem has popped up.
Could not save master table to file '/home/ravi/workspace/.metadata/.plugins/org.eclipse.core.resources/.safetable/org.eclipse.core.resources'.
/home/ravi/workspace/.metadata/.plugins/org.eclipse.core.resources/.safetable/org.eclipse.core.resources (Permission denied)
So I started eclipse using sudo, but this time the following error has come in the console of eclipse.
"root" execution of the PostgreSQL server is not permitted.
The server must be started under an unprivileged user ID to prevent
possible system security compromise. See the documentation for
more information on how to properly start the server.
Could any one help me with this.
Thank you

Problem 1: User ID mismatch
Reading between the lines, it sounds like you're trying to debug a PostgreSQL instance that's running as the postgres user, or a different user ID to your own anyway. Hence your attempt to use sudo.
That's painful, especially when using an IDE like Eclipse. With plain gdb you can just sudo the gdb command to the desired uid, e.g. sudo -u postgres -p 12345 to attach to pid 12345 running as user postgres. This will not work with Eclipse. In fact, running it with sudo has probably left your workspace with some messed up file permissions; run:
sudo chown -R ravi /home/ravi/workspace/
to fix file ownership.
If you want to debug processes under other user IDs with Eclipse, you'll need to figure out how to make Eclipse run gdb with sudo. Do not just run all of Eclipse with sudo.
Problem 2: Trying to run PostgreSQL under the control of Eclipse
This:
"root" execution of the PostgreSQL server is not permitted. The server must be started under an unprivileged user ID to prevent possible system security compromise. See the documentation for more information on how to properly start the server.
suggests that you're also attempting to let Eclipse start postgres directly. That's very useful if you're trying to debug the postmaster, but since you're talking about the query planner it's clear you want to debug a particular backend. Launching the postmaster under Eclipse is useless for that, you'll be attached to the wrong process.
I think you probably need to read the documentation on PostgreSQL's internals:
Tour of PostgreSQL Internals
PostgreSQL internals through pictures
Documentation chapter - internals
Doing it right
Here's what you need to do - rough outline, since I've only used Eclipse for Java development and do my C development with vim and gdb:
Compile a debug build of PostgreSQL (compiled with ./configure --enable-debug and preferably also CFLAGS="-ggdb -Og -fno-omit-frame-pointer"). Specify a --prefix within your homedir, like --prefix=$HOME/postgres-debug
Put your debug build's bin directory first on your PATH, e.g. export PATH=$HOME/postgres-debug/bin:$PATH
initdb -U postgres -D $HOME/postgres-debug-data a new instance of PostgreSQL from your debug build
Start the new instance with PGPORT=5599 pg_ctl -D $HOME/postgres-debug-data -l $HOME/postgres-debug-data.log -w start
Connect with PGPORT=5599 psql postgres
Do whatever setup you need to do
Get the backend process ID with SELECT pg_backend_pid() in a psql session. Leave that session open; it's the one you'll be debugging.
Attach Eclipse's debugger to that process ID, using the Eclipse project that contains the PostgreSQL extension source code you're debugging. Make sure Eclipse is configured so it can find the PostgreSQL source code you compiled with too (no idea how to do that, see the manual).
Set any desired breakpoints and resume execution
In the psql session, do whatever you need to do to make your extension run and hit the breakpoint
When execution pauses at the breakpoint in Eclipse, debug as desired.
Basic misunderstandings?
Also, in case you're really confused about how all this works: PostgreSQL is a client/server application. If you are attempting to debug a client program that uses libpq or odbc, and expecting a breakpoint to trigger in some PostgreSQL backend extension code, that is not going to happen. The client application communicates with PostgreSQL over a TCP/IP socket. It's a separate program. gdb cannot set breakpoints in the PostgreSQL server when it's connected to the client, because they are separate programs. If you want to debug the server, you have to attach gdb to the server. PostgreSQL uses one process per connection, so you have to attach gdb to the correct server process. Which is why I said to use SELECT pg_backend_pid() above, and attach to the process ID.
See the internals documentation linked above, and:
PostgreSQL site - coding
PostgreSQL wiki - developer resources
Developer FAQ
Attaching gdb to a backend on linux/bsd/unix

I also faced similar issue and resolved it after some struggle
I misunderstood the following point under Debugging with child processes in the wiki (https://wiki.postgresql.org/wiki/Working_with_Eclipse).
5."Start postmaster & one instant of postgresql client (for creating one new postgres)"
The above step should be performed from terminal by starting postgres server and one client.
Hope this helps
Once this is done then debugger in eclipse needs to be started for C/C++ Attach to Application

Related

Paagent job doesn't run [duplicate]

Created a job using pgagent.
Successfully it got created but could not check its status if its running or not.
Neither it has performed its specified code which I have given in steps.
Verion Used:
PostgreSQL 9.6.2 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-17), 64-bit
With the help of pgAgent I have created a JOB
Job successfully got created but I cant check if its working or not as there is no notification for the same, neither it is performing the SQL code which i have passed in Job -> Step.
If I got it right, you installed pgAagent but it is not starting when you execute the jobs. Have you tried starting it using the terminal? The service might be down (I do start it manually every time the database system is restarted).
Using the postgres user ...
sudo su - postgres
... execute the following command (assuming you're on the db server):
pgagent hostaddr=127.0.0.1 dbname=postgres user=postgres -s pgagent_log.log
I hope it helps.
Cheers!
Maybe it'll be helpful to some windows users, because I faced the same issue recently.
to check run's statistic in pgadmin click at the your job and then select "statisctic" tab.
Second unclear thing to me is step's settings.
I have the one server, so i thought I have to choose "local" connection type, but in didn't work. Even on the same machine try to select "remote", and then put connection string like:
host=localhost port=5432 dbname=YOUR_DB user=postgres password='asd#3wf!'
Please pay attention that for passwords which contain special symbols you have to set it in the ''.
In my case the problem was that agent couldn't authorize to database and service hasn't been started, so after I've made all connections trusted from localhost the service started successfully and job works fine
for more information about error you should see into windows event viewer or eq in unix based systemC:\Program Files\PostgreSQL\10\data\pg_hba.conf

PSQL_HISTORY ignored by PyCharm

I have a Django project connecting to a PostgreSQL database which I develop in PyCharm, and I want to enable PostgreSQL history logging.
There is PSQL_HISTORY env variable set to /home/user/apps/postgres/logs/.pycharm_log, but when I start the project in PyCharm and update some data via the Django Admin (which certainly hits the database) -- nothing gets logged and the file is not created at all.
Is there a way to make PyCharm and PSQL_HISTORY work together as I expected?
'psql' is the name of a specific client tool. Why would a completely different tool use psql's configuration options? If you want to log every statement sent to the server, you could configure that in the server side with log_statement=all.

pgagent - not running jobs - pgpass file is correct - postgresql

I have Pgagent installed on my Debian OS. Along with Postgresql 9.4.
I have checked .pgpass file as this seems to be the most common cause for a job to not run.
host port 5432 database = * username = postgres password = xxxx.
for both local and the remote host. The database I'm trying to set a job for is on a remote host.
I made sure it was enabled. It's just a simple INSERT script that should repeat every 5 minutes.
No errors are being triggered that I can find. Any ideas of what would cause the job not to run at all - even when selecting 'run now'?
Check postgre db, pgAgent Catalog, pga_jobsteplog
IDK about Linux but I had similar problem in windows where the thing won't run and it doesn't raise any notice on the error even after doing RUN NOW. The only error i could find out was that if i click on the job and click on statistics, i could see like shit ton of times it ran and everytime it ran, its status was F.
The reason for this failure is becuase the pgagent couldn't connect to the main database of postgresql.
The services of pgagent isn't running at all (as we can see this information under services in task manager in windows).
Forcing the service to run would create a failure which can be viewed in the event manager in windows.
To solve this issue, first try putting that pgpass.txt file in the environment variable (if not automatically put), if this didn't work, then what I did was to uninstall and delete all possible folders of Postgres, pgagent, and pgadmin, clearing out all temp files, clearing out registry details which have been put by Postgres, pgagent, and pgadmin and also from environment variable. Then reinstall it and it would normally work :)

Failed to load sql modules into the database cluster during PostgreSQL Installation

I have attempted to install PostgreSQL 9.4 and 8.4 multiple times and it is failing no matter what I have tried. I am attempting to install on Windows 7 SP1 x64. After each failed install I have uninstalled and deleted the installation folder to start fresh.
Each time I attempt the install I get an error pop up near the end of installation that says:
"failed to load sql modules into the database cluster".
Then another error pop up displays immediately after that says:
"Error running post install step. Installation may not complete correctly. Error reading C:/Program Files/PostgreSQL/9.4/postgresql.conf"
I have attempted installation with the following actions:
Always installed as administrator
Turned off all virus protection and windows firewall
Changed the installation directory to something other than the Program Files directory.
Changed the data directory to something other than the installation directory of postgres
None of the actions above have helped and I always receive the error. Any help that someone can provide would be greatly appreciated!
Encountered a very similar problem that OP is reporting today while installing Postgres 9.4.
It turns out that the password generator I was using has made a password that contains non-alphanumeric characters. ("^") I believe was the culprit in this case. Removing that allowed the installation of Postgres 9.4 (Windows 64 bit) to complete.
Very easy fix once you've found it, but the error message that comes up is not descriptive, so in this case I didn't notice at first what I was doing wrong.
I was getting this same error when trying to install PostgreSQL v9.4.4 on Windows 10 Pro. Starting with a solution hosted on Stack Exchange, I came up with the following steps that allowed the installer to run successfully:
1) Create a new user account, called postgres
2) Add the new account to the Administrators and Power Users groups
3) Restart the computer
NOTE: I added step #3, since step #4 didn't work without it
4) Run a command prompt as the postgres user, using the command:
runas /user:postgres cmd.exe
5) Run the installer from the postgres command window
6) Delete the postgres user account, as well as the user directory
NOTE: I added step #6, since the postgres account is not required after installation
What worked for me is, during the install, specifying a Postgre SQL data folder that's outside of any Windows user profile directory (C:\Users), such as C:\postgres-data.
My setup:
Win 10 Pro
PostgreSQL 9.5 RC1
I ran into this issue when I tried setting PostgreSQL's data directory somewhere under my user profile's directory, such us somewhere under "My Documents".
I tried Jeff G's solution and it didn't work at first. It worked only when I kept the data directory as the default (C:\Program Files\PostgreSQL\9.5\data). So then I tested further and tried setting the data directory to somewhere outside of any user profile directories (i.e. somewhere that isn't under C:\Users). For example, C:\postgres-data and this worked.
I then just tried using that directory with my default user, and not bothering with the postgres temp user as outlined in Jeff G's solution and that also worked. So in the end, it had to do with my data directory being somewhere under C:\Users. As long as it was outside of that, it worked.
None of these answers here helped me, finally I solved this problem by creating the folder before the installation (C:\PostgreSQL\data) and giving it full access for the group "Users".
Windows 7 x64, postgresql-9.5.2-1-windows-x64
Hope It's Work.
1.)Uninstall PostgreSQL
2.)Delete the postgres user if it still exists :
net user postgres /delete
3.) Create the postgres user with a password you can remember:
net user /add postgres
4.) Add the postgres user to the Administrators group:
net localgroup administrators postgres /add
5.) Add the postgres user to the Power Users group
net localgroup "power users" postgres /add
6.) Run a command window as the postgres user:
runas /user:postgres cmd.exe
7.) Change user postgres and install postgresql
8.)Back your and remove the postgres user from the Administrators group.
net localgroup administrators postgres /delete
I had the same error ("Failed to load SQL modules into the database cluster.") when installing on W2K12R2 using the EnterpriseDB installer linked to from the PostgreSQL Windows download page. I tried running the installer with admin privs, and using the postgres-user solution offered by Jeff G, but neither worked. Finally I tried the second installer, BigSQL, and that installed without issue.
When installing PostgreSQL, do not use the following symbols in your database's admin password: %, <, and >.
Bug report and problem solution here
Changed the data directory to something other than the installation directory of postgres
Make sure NETWORK SERVICE has read/write permissions on that folder, the installation was failing for me until I did this.
I had this issue too with 9.5 and got around it by:
Installing with the default data folder.
Making sure my new data folder had full control access for the "NETWORK SERVICE" account.
Then changing the default PGDATA folder as per this instruction: https://wiki.postgresql.org/wiki/Change_the_default_PGDATA_directory_on_Windows
I had the same problem, and noticed that some Postgres .bat files were opening in a text editor. Text editors opening files during installation isn't too uncommon, but I noticed these batch files were setting directory permissions. I then found that .bat files were set to open in a text editor (Notepad++) by default. I changed the default program for .bat files, uninstalled Postgres, re-ran the installer, and it worked perfectly. Hope this helps.
I had the same problem before. I solved this by installing Visual Studio C++ Express first.
I had the same experience as #gomisha. #Jeff G's solution did not work for me no matter which user ran the installer.
As long as the target data directory was anywhere under c:\users, the install or database cluster init would fail.
I was successful after creating c:\postgresql-data and ensuring that the postgres user had full access in the 'effective access' tool.
Note: I did not use control userpasswords2 to set up the postgres user; I did it through the traditional crappy Windows 10 user wizard that tries to make the user sign up for hotmail. I did run the install as postgres.
I was getting this issue with the BigSQL windows installer PostgreSQL-9.6.5-1-win64-bigsql.exe
The problem turned out to be that my system path was too long. After removing some uncessary junk from my System Path, and then reinstalling, the problem was gone.
One of the answers here is the only thing that worked on my machine:
I had the same error message when trying to install 9.2.4. My issue
was because even though the Win2k8 server had %SYSTEMROOT%\system32 as
part of the path, no programs could "see" anything in
C:\Windows\system32. The installer heavily uses icacls.exe during the
init routine. Since my path was screwed up, the installer bombed out.
Once I explicitly added C:\Windows\system32 to the SYSTEM's Path
environment variable and re-ran the installer as an admin, everything
worked fine.
Credits to the user.
Go to services.msc
Search for postgres service
Go to Log On tab
Select local account
Profit
The alternative solution for problems with PostgreSQL installer is setting this up with a Docker.
Install Docker Desktop
https://www.docker.com/products/docker-desktop/
Start installed Docker Desktop
In PowerShell, run:
docker pull postgres
In PowerShell, run:
docker run -d -p 5432:5432 --name postgres -e POSTGRES_PASSWORD=mypassword postgres
Now you can connect to the Postgres server for example with pgAdmin on the same machine, using:
host: localhost
port: 5432
maintenance database: postgres
username: postgres
password: mypassword
I think this would be better as individual comments above, but I don't have the required rep. I just spent days on this, with Postgres 10 on Win 10 Creators. My "answer" was to let Postgres put the data in its default install folder instead of my User folder (where it would be automatically backed-up).
1. This problem can happen with no reported install errors at all. On my very first try, I saw:
Failed to load SQL modules into the database cluster.
But that seemed to be due to:
Executing C:\Users\loren\AppData\Local\Temp\postgresql_installer_aee8e5a76f\vcredist_x64.exe /passive /norestart
Script exit code: 3010
MSI (s) (AC:FC) [14:21:35:341]: Product: Microsoft Visual C++ 2013 x86 Minimum Runtime - 12.0.40660.
Restart required. The installation or update for the product required a restart for all changes to take effect.
The restart was deferred to a later time.
Restarting Windows after the failure of the rest of the Postgres installation did not let Postgres run.
Once VC was in place, uninstalling and re-installing Postgres (still with its data in my User folder) completed with no reported errors. But every attempt to access it, by any means, still resulted in something like this:
C:\Program Files\PostgreSQL\10\bin>psql
psql: could not connect to server: Connection refused (0x0000274D/10061)
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused (0x0000274D/10061)
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
The RawCap sniffer showed TCP SYN followed immediately by RST,ACK on localhost for both v4 and v6 for each try. Netstat did not see port 5432 in use at all. All Postgres runtime logs were totally empty. In Windows Services, I could manually start the postgresql-x64-10 service, but it immediately stopped itself, with the note that some services do that normally.
2. I tried moving the data directory via the Postgres wiki procedure. Apparently for Postgres 10 it is incomplete. It does not deal with:
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\PostgreSQL\Installations\postgresql-x64-10\Data Directory
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\PostgreSQL\Services\postgresql-x64-10\Data Directory
C:\Program Files\PostgreSQL\10\pg_env.bat
C:\Program Files\PostgreSQL\10\scripts\serverctl.vbs
But fixing every data file instance I could find did not let it run. And even after fixing all those, and uninstalling and re-installing again with the default location, the new pgAdmin4 somehow still remembered my original User data directory location!
3. I did not try giving NETWORK SERVICE read/write permissions on my User folder. Once the default install worked I wanted to get on with my original project... Maybe that would have worked.

Portable PostgreSQL for development off a usb drive

In order to take some development work home I have to be able to run a PostgreSQL database.
I don't want to install anything on the machine at home. Everything should run off the usb drive.
What development tools do you carry on your USB drive?
That question covers pretty much everything else, but I have yet to find a guide to getting postgresql portable. It doesn't seem easy if it's even possible.
So how do I get PostgreSQL portable? Is it even possible?
EDIT:
PostgreSQL Portable works. It's very slow on the usb-drive I have, but it works. I can't recommend doing constant development with it but for what I need it's great.
Perhaps if I pick up a full speed external drive I'll try out virtualization. Given the poor performance of just running the database off this drive, a full virtual OS running off of it would be unusable.
Here's how you can do this on your own:
http://www.postgresonline.com/journal/archives/172-Starting-PostgreSQL-in-windows-without-install.html
An alternate route would be to use something like VirtualBox and just install your development environment (database, whatever) on there.
There are 2 projects to try in 2014: http://sourceforge.net/projects/pgsqlportable/ and http://sourceforge.net/projects/postgresqlportable/?source=recommended.
I can't vouch for the second, but I'm using the first and it works right out of the box.
After unzipping using 7-zip (http://www.7-zip.org/download.html):
1) Run "start service without usuario.bat" ( english translation )
2) Then run "pgadmin3.bat"
The only minimal problem for me was that its in spanish. I've been able to change the language to english by following Change language of system and error messages in PostgreSQL. Using google translate the instructions are:
Description
This is a zip to automatically run postgresql 9.1.0.1 for windows. This version already has pgagent and pldebugger. To run must: 1) unzip
the zip 2) run the "start service without usuario.bat" found in the
pgsql directory within the folder you just unzipped. 3) Optional. If
you want to run the agent works postgresql (pgagent) should only run
the "start pgagent.bat" found in the pgsql directory inside the folder
you just unzipped. 4) Optional. To manage and / or develop the bd you
can run the pgadmin3.bat 5 files) Optional. To stop and / or restart
the server correctly use file "service without stopping usuario.bat"
usuario.bat or restart service without depending on the case.
Now option for Linux (file. Tar.gz). Postgresql portable Linux 9.2
Please use the tickets for your answer bugs.
Username: postgres Password: 123
Just a Note : on a new computer , to get pgadminIII working you may need to add a db. The settings are in attached screenshot.
Hope it helps.
I agree with virtualization solution, but maybe you can find useful this link from portable freeware collection, I have used this locally, not from usb though
1.download and extract : zip version
2.inside pgsql folder create data folder(put any name,I used 'data')
3.initalize data folder: c:\pgsql\bin\initdb.exe -D c:\pgsql\data -U postgres -W -E UTF8 -A scram-sha-256
4.to start/stop see next cmd code that I use (press any key inside it to stop)
c:\pgsql\bin\pg_ctl.exe -D c:\pgsql\data -l logfile start
pause
c:\pgsql\bin\pg_ctl.exe -D c:\pgsql\data stop
more info