I am not too familiar with Bucardo and Postgres , so I was hoping to get some feedback / how to from this question.
We have 3 computers running in parallel in various parts of the building. When one updates something the other shows the data etc thanks to bucardo syncs running between the computers.
However here is the requirement. At anytime one computer can be brought offline and reimaged. Now when this computer comes back on line, the operator should be able to hit replicate and get the data from the master computer.
What is the best way to accomplish this ?
My thought was to run a pg_dump on the master computer and run pg_restore on the reimaged computer.
or do you think setting bucardo onetimecopy=2 is the best course of action.
Related
I have a few PCs that have local PostgreSQL databases running, just logging data. Data is only ever inserted, never removed or updated. The remote PCs are connected to the internet by cellular modem and depending on their location, often do not have internet access. When they do have an internet connection I would like them to push a copy of their databases to a central location and keep the remote database up to date with any new data. Essentially, I need an 'rsync' for databases.
At first it seemed like what I need is to set up PostgreSQL Hot-Standby but I'm unsure if this is actually what I need because my situation seems to differ from the examples I've seen.
Each remote PC has a Postgres server with a single database that has a unique name, the tables within the DBs have generic names. I would like to synchronize these databases to a single remote Postgres server. I think this should be okay due to the unique DB names.
My connectivity is very intermittent, days to weeks without a connection. I've seen PgAdmin be very reliable despite a terrible (cellular) internet connection, if Postges Hot-Standby is the same I may be alright.
As far as I can see my options are either to set up PostgreSQL Hot-Standby, or roll my own solution. I don't want to roll my own solution. However it is simple enough if I can't find anything better; a Python daemon run by systemd to find the diff between the local and remote DB, then push the new rows from the local to the remote DB. But I'm sure someone has solved this problem, I just haven't found the solution yet.
You don't need hot standby (which is the PostgreSQL term for being able to query the replicated database), but streaming replication. You need a central standby server for each intermittently connected remote database server. If you use replication slots, you can be sure that replication will never fall behind.
I've been having a bit of trouble trying to install PostgreSQL 14 for the first time.
I would like to apologize in advance if this question has been asked in the manner that I am about to ask it, but I do not think it has. If it has been, please direct me to the appropriate location!
I've done a fair amount of Googling on the matter, and all the information that I find seems to be rather fragmented, or I end up following a spaghetti trail of hyperlinks (a la-do-this-and-follow-this-other-link-with-more-information-than-you-need-to-understand-this-other-required-portion).
Personally, I don't want to jump around to 50 different locations on the web to try and conjure up a piecemeal solution that I believe works, only to be proven wrong later. I want to know what to do and why it works. I've tried reading the documentation, and have given up on it, because to me, it seems to assume that the server has already been set up by a database administrator.
Instead of articulating my problem directly (as I seem to be having more trouble than I would like by trying to do so), I believe it would be easier to articulate my problem indirectly by stating what my expectations would be after installing PostgreSQL for the first time.
So to start, I will mention that I'm running Ubuntu 18.04.6 LTS, and am installing PostgreSQL 14.1 with the following command:
sudo apt install postgresql-14
Before continuing, I would like to add a side note in advance, that I do not want suggestions for an alternative OS or install method. I just want to be able to get "up and running" in a common-sense fashion from this exact point.
Moving on, I know that the aforementioned command creates a *nix user called postgres.
From here, I can now indirectly state my problem using an outline of what my goals and expectations are immediately after installing the software via that command.
After installing PostgreSQL via apt, these are my expected goals:
I want any client to be able to connect to the database server from any computer where a route exists from the client to the server.
For the sake of simplicity with these stated goals, when it is directly or implicitly stated that I am trying to connect a client to the database server, I am making the assumption that the client is able to, at a minimum, ping the machine that the server is running on, and vice versa.
For now, I'm not completely worried about the database being accessible from the public Internet.
I expect to be able to access the database from any computer on my LAN, whether it is an actual LAN, or some sort of logical LAN (like a WAN or a VPN).
If I change the PostgreSQL password of the postgres user, I expect that any client logging into the database server via the postgres user will require the password.
This means if I want to change the password to some_password via \password postgres or ALTER USER postgres WITH PASSWORD 'some_password'; (I am assuming this is how you change the login password of a PostgreSQL user), then...
I expect running psql [-h host] -U postgres -W from any host...
That when I am prompted to enter the password...
I can only log in by entering the exact password of some_password.
Entering any other arbitrary text for the password should not allow me to log in.
I am adding this as a requirement because previous install attempts have shown me that this is NOT the case.
I expect to be able to create a PostgreSQL user account other than postgres (e.g. db_user) with a password and have it be subject to the same requirements as the postgres user.
i.e. once the new account is given permission to log in, the same common-sense login requirements to log in must be imposed, i.e. you can't get in if you don't have the correct username/password combination.
If the process to achieve the aforementioned can be explained in such a way that it can be understood with minimal mental friction, I would be extremely grateful.
Feel free to assume that my knowledge is on par with that of a undergraduate CS student who just completed their first year of university, who also understands Linux filesystems and basic computer networking. I just want the answer to be as accessible to as many people as possible, as I am sure I'm not the only person who has struggled with installing PostgreSQL, in spite of having a power user's level of computer literacy.
sudo apt install postgresql
sudo -u postgres psql
Set a password for this user with \password or the other method you mention
sudo vi /etc/postgresql/10/main/pg_hba.conf
Make the only uncommented nonblank line in this file be host all all all md5
sudo vi /etc/postgresql/10/main/postgresql.conf
uncomment listen_addresses line and set it to '*'
sudo service postgresql restart
When you make a new user, you should also make a new database which has the same spelling as the user does. Otherwise you will need to specify the database name when you try to log in with psql -U, such as psql -U newname -d postgres -h[hhh]. Should you actually be running 14 not 10, then you will need to change the paths of the config files you need to edit accordingly.
Two days ago, i started learning postgresql. Most tutorials I followed online were either old or the codes wont just work on my mac. I followed a lot of tutorials that did a lot of totally different things.
When i switched on my system today. I noticed Postgresql created a user on my mac. I don't know what this is or maybe i used the wrong CLI code.
When I tried viewing the user, I saw this
should I delete this user or it has a function?
postgres user account
Creating a user account specifically for Postgres, commonly named postgres, is a normal part of a Postgres installation. Your installer app likely prompted you for a password to assign to this new user account.
One reason for this is security: The database’s data files and security configuration files are stored in folders owned by the postgres user. So if your main user account is hijacked, the intruder does not yet have access to the database (often the most valuable thing in storage). The intruder must jump through more hoops to compromise Postgres. Also, the separate ownership prevents other apps from inadvertently stomping on the Postgres files.
You will find Postgres is much more enterprise-oriented than other products such as MySQL. This means locking-down for security. Another example: Postgres by default is configured to not accept connections over the network. To enable connections from other computers, you must change the configuration. Inconvenient for the beginner, but more secure. Like a bar on your car steering wheel and deadbolts on your doors, more security always means more steps to take and more annoyance.
Use a virtual machine
Installing the postgres user account is one of the things that makes Postgres a rather heavyweight installation. I suggest to those learning Postgres to use a virtual machine for Postgres. Something like:
Parallels or Fusion or VirtualBox on your own computer
Cloud server such as FreeBSD on DigitalOcean.com.
To remove Postgres, simply discard the vm.
Postgres.app for macOS
Another option for a Mac user is Postgres.app, created by the person who built one of the first Postgres-as-a-Service implementations (on Heroku). I have not used Postgres.app, but I understand it wraps Postgres, so it does not install the postgres user account. Also, Postgres starts and stops when launch and quit the app, rather than running in the background all the time.
Be aware: you may have conflicts with Postgres.app on a Mac where you already have a conventional installation. I suggest you first carefully remove the conventional Postgres from your Mac before installing Postgres.app. Uninstalling involves finding and deleting various files and folders in various places.
Database-as-a-Service (DBaaS)
Another option to avoid local installation is the increasing choices for running Postgres as a service. This is sometimes referred to as “managed Postgres” because the vendor maintains the installation of Postgres on your behalf. You simply use Postgres to create your database, but you do not fully control Postgres in such a service.
Some examples:
Heroku
Digital Ocean
Azure
Amazon Web Services (AWS)
ElephantSQL
My experience
Personally, I often install Postgres on a Mac using the installer by EnterpriseDB.com. That company sells added-value versions of Postgres, but kindly provides an installer for plain-vanilla Postgres, as a service to the community.
I have also used that same installer from EnterpriseDB.com to install onto a Parallels VM running macOS as the guest OS within the VM on a MacBook Pro running macOS as the host OS. You can easily configure the VM to share the host Mac’s IP address on the network, or you can give the VM its own network address which might be handy for demo/dev/test work.
Thirdly, I have installed Postgres on FreeBSD on DigitalOcean.com.
All three of these options have worked quite well for me. Which is preferable depends on the scenario. For example, the DigitalOcean.com approach is good if I want colleagues to be able to reach the database 24x7 without my own MacBook being available.
This discussion is for development work. For mission-critical deployment, I strongly recommend using heavy-duty server equipment with error-correcting memory and redundant storage such as RAID or ZFS pool. Postgres is extremely reliable but depends, of course, on reliable hardware.
Your tag says Postgres 9.1. That version is quite old now. I suggest using the latest version. By the way, the version numbering system has changed for postgres. The first number is now the roughly-annual release number likely requiring you to dump and reload data to upgrade, and the second number is compatible updates.
As pointed out by #basil-bourque, the account is required for several reasons.
That said, if it annoys you to have the PostgreSQL showing up in the login screen -as it did me-, you can remove it as long as you have admin user rights in MacOS.
Apple Support gives the following command to hide a user from the login screen:
$ sudo dscl . create /Users/hiddenuser IsHidden 1
However, since the PostgreSQL user was not included listed in the login window at installation, that command will yield no result -at least in Catalina, which is my OS.
You should use the following two commands instead, as suggested by josemarluedke:
## add postgres to the list of hidden users on login screen
$ sudo defaults write /Library/Preferences/com.apple.loginwindow HiddenUsersList -array-add 'postgres'
## instruct not to show any hidden accounts at login
$ sudo defaults write /Library/Preferences/com.apple.loginwindow SHOWOTHERUSERS_MANAGED -bool FALSE
Worked for me!
I've been a MySQL guy, and now I'm working with Postgres so I am learning. Wondering if someone can tell me why my postgres process on my macbook is sending and receiving data over my network. I am just noticing this is happening for the first time - so maybe it's been going on before this and I just never noticed postgres does this.
What has me a bit nervous, is that I pulled down a production datadump from our server which is set up with replication and I imported it to my local postgres db. The settings in my postgresql.conf don't indicate replication is turned on. So it shouldn't be streaming out to anything, right?
If someone has some insight into what may be happening, or why postgres is sending/receiving packets, I'd love to hear the easy answer (and the complex one if there's more to what's happening).
This is a postgres install via Homebrew on MacOSX.
Thanks in advance!
Some final thoughts: It's entirely possible, I guess, that Mac's activity monitor also shows local 'network' traffic stats. Maybe this isn't going out to the internets.....
In short, I would not expect replication to be enabled for a DB that was dumped from a server that had it if the server to which it was restored had no replication configured at all.
More detail:
Normally, to get a local copy of a database in Postgres, one would do a pg_dump of the remote database (this could be done from your laptop, pointing at your server), followed by a createdb on your laptop to create the database stub and then a pg_restore pointed at the dump to populate its contents. [Edit: Re-reading your post, it seems like you may perhaps have done this, but meant that the dump you used had replication enabled.)]
That would be entirely local (assuming no connections into the DB from off-box), so long as you didn't explicitly setup any replication or anything else that would go off-box. Can you elaborate on what exactly you mean by importing with replication?
Also, if you're concerned about remote traffic coming from Postgres, try running this command a few times over the period of a minute or two (when you are seeing the traffic):
netstat | grep postgres
In general, replication in Postgres in configured at a server level, and has to do with things such as the master server shipping WAL files to the standby server (for streaming replication). You would have almost certainly have had to setup entries in postgresql.conf and pg_hba.conf to ensure that the standby server had access (such as a replication entry in the latter conf file). Assuming you didn't do steps such as this, I think it can pretty safely be concluded that there's no replication going on (especially in conjunction with double-checking via netstat).
You might also double-check the Postgres log to see if it's doing anything replication related. In a default install, that'd probably be in /var/log/postgresql (although I'm not 100% sure if Homebrew installs put it somewhere else).
If it's UDP traffic, to and from a high port, it's likely to be PostgreSQL's internal statistics collector.
These are pre-bound to prevent interference and should not be accessible outside of PostgreSQL.
I know the way to set up a Master/Slave DB in Postgres is having 2 DB servers, but unfortunately i have only one server for now.
How can i mirror my production db into another "backup db" in "real_time"? I want to give access to another user to the mirrored db, so even if he does something there it will not affect production.
Nothing stops you setting up hot standby streaming replication, or another replication option like Londiste, between two PostgreSQL instances on the same computer.
The two copies of PostgreSQL must use different ports, but that's the only real restriction.
How to set up the second PostgreSQL instance depends on your operating system and how you installed PostgreSQL, which you have not mentioned.
You'll want to use streaming replication with hot standby if you want a read-only replica. If you want it to be read/write, then you can do a one-off copy of the database with pg_basebackup and not keep them in sync after that. Or you can use a tool like Londiste to replicate changes selectively.
You can run multiple instances of PostgreSQL on the same computer, by using different ports.