We have a policy requirement to use items using wtmp, such as the 'last' command or GDM-Last-Login details. We've discovered that these items will have gaps depending on when wtmp was last rotated, and need to try to work around this.
Because these gaps have been determined to be unacceptable, and keeping wtmp data in a single active logfile forever without splitting off the old data into archives is not really viable, I'm looking for a way to rollover / age-out old wtmp entries while still keeping more recent ones.
From some initial research I've seen this problem addressed in the Unix (AIX, SunOS) world with the use of 'fwtmp' and some pre/post logrotate scripts. Has this been addressed in the Linux world and I've just missed it?
So far as I can tell 'fwtmp' is a Unix built-in that's not made it into RHEL 5 & 6, per searching the RHEL customer portal and some 'yum whatprovides' searches on my test boxes.
Many thanks in advance!
Related
In TYPO3 10 logs are beeing stored, amoung others, in var/log. Files that are stored there are growing over time. Is there a way to keep it clean and keep automatically entries from x last days?
Would be eventually quite a task of an operation. Normaly, log rotating is a job for the OS, so you may use something like logrotate or similar to rotate the logs.
Whats recommendation can you give me on setting up a database failover postgresql cluster? I have only 2 virtual machine.
Right now i read this https://wiki.clusterlabs.org/wiki/PgSQL_Replicated_Cluster
I have some questions about it:
Where is it written in the configuration files when the second machine should turn on as an active one?
How does the first car understand that the second car is active?
Why does not the virtual IP address conflict?
When the main machine turns on, how will the system understand what needs to be done replication from the second server?
Sorry for my bad English
Its almost 2 months you asked it but it seems you are in same boat as I was in few weeks back. I have gone through your link and it explains that you need to use corosync + pacemaker + pcs. Frankly, I have no experience on any of them but I used pgpool2 4.0.4 (latest at the time of writing) with PostgreSQL 9.5.14 and 10.7, successfully able to brought up 2 clusters in last 2 months.
With pgpool you do not need to use any other tool/library and all configuration goes to one file pgpool.conf along with few password (1 liners) in pool_password and pcp.conf.
All the needed configuration of watchdog(component of pgpool cluster) to find out the live/dead status of cluster comes with pgpool and merely need configuration to handle it.
You may find more information on pgpool2 at here and about latest version at here.
Also you may refer (just read first to get a gist of whole process) at link which is super useful and quite detailed on how the whole process goes.
Also let us know if you were able to setup cluster with mentioned technologies at your link.
Edit: you may find extracted configurations of pgpool.conf at my gist page
I have kept only the settings which I changed. Rest have been left as default , or may be i forgot to add 1-2 to this.
Most of the comment on the file come right from standard documentation and self-explanatory but few places I have added my own comment and they are
vip configuration.
At one place I am using a different postgres password.
note about recovery_1st_stage
note about key file referred by logdir
Also most important things is , sit back and read through original links referring to std. documentation to just a gist of what the whole thing/process is. It will be easier for you to modify it as per your needs later.
I read it , 3-4 times ( slow learner ) both the documentation and then used a mix of both approaches.
Also there are 4 files, i created
recovery_1st_stage
pgpool_remote_start.sh
failover.sh
promote_standby.sh
You will find guidance on these at both the places : std. documentation and other tutorial. they are plain sh file with a bunch of ssh and psql commands.
I have a huge daily textual logs (2-3 GB), which I want to investigate for specific event (which I know when it occurred), I'm using less (since it all on ssh to remote server).
I'm looking for an option to jump as fast as I can to the exact time, and I think if there is an option to a binary search to find it, it should be the fastest (right now jump to end of the day takes tens of seconds)
Thanks!
Based on this other question's answer:
sgrep might work for you:
sudo apt-get install sgrep
sgrep -l '"needle"' haystack.txt
The project page http://sgrep.sourceforge.net/ says:
Sgrep uses a binary search algorithm, which is very fast, but requires sorted input.
So, I am (well... I was) running PostgreSQL within a container (Ubuntu 14.04LTS with all the recent updates, back-end storage is "dir" because of convince).
To cut the long story short, the container folder got deleted. Following the use of extundelete and ext4magic, I have managed to extract some of the database physical files (it appears as if most of the files are there... but not 100% sure if and what is missing).
I have two copies of the database files. One from 9.5.3 (which appears to be more complete) and one from 9.6 (I upgraded the container very recently to 9.6, however it appears to be missing datafiles).
All I am after is to attempt and extract the SQL code the relates to the user defined functions. Is anyone aware of an approach that I could try?
P.S.: Last backup is a bit dated (due to bad practices really) so it would be last resort if the task of extracting the needed information is "reasonable" and "successful".
Regards,
G
Update - 20/4/2017
I was hoping for a "quick fix" by somehow extracting the function body text off the recovered data files... however, nothing's free in this life :)
Starting from the old-ish backup along with the recovered logs, we managed to cover a lot of ground into bringing the DB back to life.
Lessons learned:
1. Do implement a good backup/restore strategy
2. Do not store backups on the same physical machine
3. Hardware failure can be disruptive... Human error can be disastrous!
If you can reconstruct enough of a data directory to start postgres in single user mode you might be able to dump pg_proc. But this seems unlikely.
Otherwise, if you're really lucky you'll be able to find the relation for pg_proc and its corresponding pg_toast relation. The latter will often contain compressed text, so searches for parts of variables you know appear in function bodies may not help you out.
Anything stored inline in pg_proc will be short functions, significantly less than 8k long. Everything else will be in the toast relation.
To decode that you have to unpack the pages to get the toast hunks, then reassemble them and uncompress them (if compressed).
If I had to do this, I would probably create a table with the exact same schema as pg_proc in a new postgres instance of the same version. I would then find the relfilenode(s) for pg_catalog.pg_proc and its toast table using the relfilenode map file (if it survived) or by pattern matching and guesswork. I would replace the empty relation files for the new table I created with the recovered ones, restart postgres, and if I was right, I'd be able to select from the tables.
Not easy.
I suggest reading up on postgres's storage format as you'll need to understand it.
You may consider https://www.postgresql.org/support/professional_support/ . (Disclaimer, I work for one of the listed companies).
P.S.: Last backup is a bit dated (due to bad practices really) so it would be last resort if the task of extracting the needed information is "reasonable" and "successful".
Backups are your first resort here.
If the 9.5 files are complete and undamaged (or enough so to dump the schema) then simply copying them in place, checking permissions and starting the server will get you going. Don't trust the data though, you'll need to check it all.
Although it is possible to partially recover given damaged files, it's a long complicated process and the fact that you are asking on Stack Overflow probably means it's not for you.
Is there any way to recover or re-create pg_filenode.map file that was accidentally deleted? Or is there any solution on how to fix this issue without affecting the database? Any suggestions to fix this issue is highly appreciated! The postgres version that we have is 9.0 running in Redhat Linux 5. Thanks!
STOP TRYING TO FIX ANYTHING RIGHT NOW. Everything you do risks making it worse.
Treat this as critical database corruption. Read and act on this wiki article.
Only once you have followed its advice should you even consider attempting repair or recovery.
Since you may have some hope of recovering the deleted file if it hasn't been overwritten yet, you should also STOP THE ENTIRE SERVER MACHINE or unmount the file system PostgreSQL is on and disk image it.
If this data is important to you I advise you to contact professional support. This will cost you, but is probably your best chance of getting your data back after a severe administrator mistake like this. See PostgreSQL professional support. (Disclaimer: I work for one of the listed companies as shown in my SO profile).
It's possible you could reconstruct pg_filenode.map by hand using information about the table structure and contents extracted from the on-disk tables. Probably a big job, though.
First, if this is urgent and valuable, I strongly recommend contacting professional support initially. However, if you can work on a disk image, if it is not time critical, etc. here are important points to note and how to proceed (we recently had to recover a bad pg_filenode.map. Moreover you are better off working on a disk image of a disk image.
What follows is what I learned from having to recover a damaged file due to an incomplete write on the containing directory. It is current to PostgreSQL 10, but that could change at any time
Before you begin
Data recovery is risky business. Always note what recovery means to your organization, what data loss is tolerable, what downtime is tolerable etc before you begin. Work on a copy of a copy if you can. If anything doesn't seem right, circle back, evaluate what went wrong and make sure you understand why before proceeding.
What this file is and what it does
The standard file node map for PostgreSQL is stored in the pg_class relation which is referenced by object id inside the Pg catalogs. Unfortunately you need a way to bootstrap the mappings of the system tables so you can look up this sort of informatuion.
In most deployments this file will never be written. It can be copied from a new initdb on the same version of Postgres with the same options passed to initdb aside from data directory. However this is not guaranteed.
Several things can change this mapping. If you do a vacuum full or similar on system catalogs, this can change the mapping from the default and then copying in a fresh file from an initdb will not help.
Some Things to Try
The first thing to try (on a copy of a copy!) is to replace the file with one from a fresh initdb onto another filesystem from the same server (this could be a thumb drive or whatever). This may work. It may not work.
If that fails, then it would be possible perhaps to use pg_filedump and custom scripting/C programming to create a new file based on efforts to look through the data of each relation file in the data directory. This would be significant work as Craig notes above.
If you get it to work
Take a fresh pg_dump of your database and restore it into a fresh initdb. This way you know everything is consistent and complete.