How does nxlog track the line number? - elastic-stack

In nxlog config I have these params set:
SavePos True
ReadFromLast True
When removing lines from a log file (this should never happen) nxlog ships the entire log file. Is this related to how nxlog tracks the line number?
To recreate:
I stop the nxlog service
Delete the nxlog cache (just to make sure im starting fresh)
Right now the log folder ive configured nxlog to watch is empty
I add a new log file to the folder
nxlog ships the log file
I open the log file and add a few lines
nxlog ships those lines
I delete those new lines I just added
nxlog ships the entire log file

NXLog and generally other log shippers are designed to deal with append-only log files.
When you delete lines from the log file it sees that the file size is less. Under the append-only assumption this can only mean that the file was replaced/rotated and the current file is a new one that needs to be fully read.
Also note that when you edit a log file in a text editor the editor will usually replace the file with a new one even if you only append data to the end. This is not equivalent to echo test >> test.log.
If you want to transfer all kinds of changes in files you should use rsync or other tools.

Related

How to create a log file in pg_log directory for each execution of my function

I have a log file as " /opt/postgres/9.2/data/pg_log/postgresql-2018-08-19.csv". Due to "log_rotation_age=1d", one log file will be created for me in this pg_log directory on everyday.
While I am debugging a particular user defined function which contains the lot of raise notice messages , I would like to create a new log file instead of appending the logs to existing one. How to achieve this?
Like this each and every execution of my function, I wold like to get a new log file. How to do this.

How to overwrite existing files with ncftpget?

When getting a remote ftp file that exists in the local destination
ncftpget says that
local file appears to be the same as the remote file, download is not necessary.
What does appears mean? How does ncftpget check if this is the same file?
It seems that it compares time and size of the file. But does it compare content or at least checksum?
Is there a way to force to overwrite the existing file. Of course other than remove it first.

How to skip "Access Denied" Folder when zip folder with command-line?

I have a batch file to copy data between 2 Disk below:
"C:\Program Files (x86)\WinRAR\WinRAR.exe" a -ag E:\Backup C:\NeedBackup -ms
Maybe use Winrar or 7-zip but they cannot copy folder with Deny for all permission. I want to skip that folder and continue to copy other files.
Anyone help me???
Start WinRAR and click in menu Help on Help topics. On tab Contents open list item Command line mode. Read first the help page Command line syntax.
Next open sublist item Switches and click on item Alphabetic switches list. While reading the list of available switches for GUI version of WinRAR build the command line.
For example:
"%ProgramFiles(x86)%\WinRAR\WinRAR.exe" a -ac -agYYYY-MM-DD -cfg- -ep1 -ibck -inul E:\Backup C:\NeedBackup\
Note 1: Switch -inul implicitly enables -y which is not documented but I know from author of WinRAR and my own tests.
You might use also the switch -dh although I recommend not using it for this backup operation.
By using additionally switch -ao the created backup archive would contain only files with having currently archive attribute set. This means only files added/modified since last backup are added to new archive because of usage of switch -ac in previous backup operation, i.e. creating incremental backup archives instead of always complete backups.
Well, the switch -df could be also used instead of -ac and -ao to create incremental backups. WinRAR deletes only files which could be 100% successfully compressed into the archive.
For details on those switches read their help pages.
Note 2: The command line creates a RAR archive file. For a ZIP file you would need additionally the switch -afzip.
Note 3: 7-Zip has also a help file explaining in detail also the usage from command line with all available commands and switches.

how to use backup files to create regular files in emacs

I am trying to create a file named caseexp.sml . Emacs created a backup file of this file when I was working on it at some earlier point, and now when I try to open it as caseexp.sml, emacs opens a #caseexp.sml# file and everytime I try to save it using C-x C-w, emacs saves it as another backup file with another tilde added to its name. Several attempts later, I have only managed to save it as #caseexp.sml"~~~.
How can I avoid creating these "tilde" backup files and save my file simply as caseexp.sml ?
There are a few unexpected behaviors here, so I can't be sure that this is what's going on, but usually what happens is that if files with hashes are left around, it's possible that Emacs crashed while you had unsaved changes. However, usually Emacs should prompt you to run "M-x recover-this-file" to restore changes from the unsaved-changes file (the filename with the hashes) to the actual file, so it's not clear what's going on there. Try fixing this from the command line.
You probably want to cp all the files to another location first, in order to have a backup (I'm assuming a Unix-like OS):
$ cp *caseexp* /tmp
Then delete the extra files while preserving the one with the most recent changes:
$ cp <most recent file with latest changes> caseexp.sml
$ rm \#caseexp*

Logrotate not generating all files after run

Hello people
It's my first time using logrotate and I don't know if I'm configuring it in the right way. I'm using it with loggerhead log file in Ubuntu 11.04
Log is under
/log/loggerhead/loggerheadd.log
My configuration file looks like this
/log/loggerhead/loggerheadd.log {
daily
rotate 7
compress
delaycompress
missingok
}
Then I run a force rotation
logrotate -f /etc/logrotate.d/loggerhead
and that change the name of the log file to
/log/loggerhead/loggerheadd.log.1
And didn't create the original file (loggerheadd.log) again, so I couldn't run a new force rotation, because "the file doesn't exist".
So, it's supposed that the application write entries in "loggerheadd.log" but when logrotate run the file will be renamed, so where will be written the log entries? Am I missing something?
Hope you can help me
By default logrotate will just rename your files, so your old file will be gone.
You can either use the create option to create a new file after the old one is used, or copytruncate to copy the original file to one with a new name, then truncate the original. Either option will do what you're asking for (more details on the man page here)