I'm trying to write event based ftp server that notifies me when some user uploads any file(users will never do any delete or other stuff, they just upload). Apache ftpServer with ftplets is what I need but it is java based and does not satisfy my environment. So what I'm planning to do is to run vsftpd and another process besides it which listens to IN_CLOSE_WRITE event. When vsftpd close the file descriptor after finishing file upload, my process will be notified and I will do some appropriate action. My questions are:
Is it possible that I miss any file upload?
Is it possible that IN_CLOSE_WRITE fired in other situations that I get inconsistent files instead of ready to use files?
if the incrond service crashes or is stopped you will loose notification during that time
IN_CLOSE_WRITE should work.
Related
I accidentally screwed up my darktable configuration, so I reloaded it from scratch. To avoid losing all my recorded changes I have done to my pictures, I wrote a powershell backup script for the darktable database. I want to launch this script from the windows task scheduler when ever I launch darktable. I have found the event id which indicates in the security log of a new process has occurred which I should be able to use to automatically launch my backup script from task scheduler. I want to add code to the script to check the services to see if darktable is actually running and only perform the backup if it is. Anyone know how I can identify this?
I am running an ELK-Stack as a central syslogserver and I set up rsyslog to send logfiles, which are not logging into /var/lib/messages by default, to it.
The setup is working very well but since I made the configuration the external logfiles actually show up in the messages file, which is blowing it out of proportion and makes debugging normal systemlogs difficult.
I want the logs to be send to the syslogserver but not into the messages file.
This is my current configuration:
111-elk-syslog.conf:
*.* ##IP_OF_THE_SYSLOGSERVER:514
101-external-log.conf
$ModLoad imfile
$InputFileName PATH_TO_LOGFILE
$InputFileTag FILE_TAG
$InputFileStateFile FILE_TAG
$InputFileFacility local3
$InputRunFileMonitor
I know, using filebeat, I could circumvent this but rsyslog is working very well in my enviroment and this application is the only one logging so much, that this is an actual problem.
I don't understand in detail your setup, but it may help to know that in general in rsyslogd if you don't want further handling of a message that you have matched, you simply repeat the filter on the next line by using &, and the action stop. For example, you might try
*.* ##IP_OF_THE_SYSLOGSERVER:514
& stop
A cronjob runs every 3 hours to download a file using SFTP. The scheduled program is written in Perl and the module used is Net::SFTP::Foreign.
Can the Net::SFTP::Foreign download files that are only partially uploaded using SFTP?
If so, do we need to check the SFTP file modified date to check copy process completion?
Suppose a new file is uploading by someone in SFTP and he file upload/copy is in progress. If a download is attempted at the same time, do I need to code for the possibility of fetching only part of a file?
It's not a question of the SFTP client you use, that's irrelevant. It's how the SFTP server handles the situation.
Some SFTP servers may lock the file being uploaded, preventing you from accessing it, while it is still being uploaded. But most SFTP servers, particularly the common OpenSSH SFTP server, won't lock the file.
There's no generic solution to this problem. Checking for timestamp or size changes may work for you, but it's hardly reliable.
There are some common workarounds to the problem:
Have the uploader upload "done" file once upload finishes. Make your program wait for the "done" file to appear.
You can have dedicated "upload" folder and have the uploader (atomically) move the uploaded file to "done" folder. Make your program look to the "done" folder only.
Have a file naming convention for files being uploaded (".filepart") and have the uploader (atomically) rename the file after upload to its final name. Make your program ignore the ".filepart" files.
See (my) article Locking files while uploading / Upload to temporary file name for example of implementing this approach.
Also, some FTP servers have this functionality built-in. For example ProFTPD with its HiddenStores directive.
A gross hack is to periodically check for file attributes (size and time) and consider the upload finished, if the attributes have not changed for some time interval.
You can also make use of the fact that some file formats have clear end-of-the-file marker (like XML or ZIP). So you know, when you download an incomplete file.
For details, see my answer to SFTP file lock mechanism.
The easiest way to do that when the upload process is also under your control, is to upload files using temporal names (for instance, foo-20170809.tgz.temp) and once the upload finishes, rename then (Net::SFTP::Foreign::put method supports the atomic option which does just that). Then on the download side, filter out the files with names corresponding to temporal files.
Anyway, Net::SFTP::Foreign get and rget methods can be instructed to resume a transfer passing the option resume => 1.
Also, if you have full SSH access to the SFTP server, you could check if some other process is still writing to the file to be downloaded using fuser or some similar tool (though, note that even then, the file may be incomplete if for instance there is some network issue and the uploader needs to reconnect before resuming the transfer).
You can check the size of the file.
Connect to SFTP.
Check file size.
Sleep for 5/10 seconds.
Check file size again.
If size did not change, download the file, if the size changed do step 3.
I'm trying to achieve following scenario
When a new message arrives to queue manager it should automatically trigger my console application. Then it will fetch that message and write the message in to a file. I already completed this part. But I'm having problem with triggering message queue
Here is how I configured my message queue
I created two queues one is called LocalQueue and other one is InitQueue which is SYSTEM.DEFAULT.INITIATION.QUEUE.
and set "Trigger" properties in my LocalQueue
Then Created trigger monitor
In process definition object I set my console application path
So my problem is, It doesn't work as it expect and I checked my client machine
C:\Program Files (x86)\IBM\WebSphere MQ\errors\AMQERR01.LOG and server machine C:\Program Files (x86)\IBM\WebSphere MQ\errors log files. I couldn't find anything on them
Update
Based on Morag answer I changed my process name
You appear to have named your TriggerMonitor Service object in the queue attribute where you should put the process object name FetchMessages.
The Start Args for your trigger monitor Service object appears to be telling the trigger monitor to monitor the LocalQueue and not the InitQueue.
You need to make sure that the trigger monitor is running and has opened the initiation queue in INPUT(i.e. for GET) mode. Only then queue manager will put a trigger message into initiation queue. Here is what I did to get it working:
I did the setup just like you except for service. I manually started runmqtmc as
SET MQSERVER=<channel name>/TCP/<connname>
runmqtmc -m <qmname> -q <initq>
Put a test message on local queue. That triggered an event and my application started.
I'm very new to MSMQ.
We have a critical system using MSMQ and it is not able to start due to insufficient resources. It appears that MSMQ is at capacity.
I am trying to purge messages (or even delete unneccessary queues), but I receive the following error when purging:
Cannot delete all messages from queue.
Error: Access to Message Queuing system is denied.
What are my options? Is there a way to delete queues when the services is off?
There's an easier way:
Open Computer Management on the machine,
expand the Services and Applications node (Features on 2008),
expand the Message Queuing service,
expand the private queues folder,
expand the private queue you're working with,
right click onto the queue messages folder and
click the purge option.
I think there is rights issue.
You are not able to give rights then do following step for forcefully delete queue.
Stop following services
Message Queuing Triggers,
Net.Msmq Listener Adapter
Message Queuing.
Go to C:\Windows\System32\msmq\storage\lqs
Now open file in notepad or notepad++ and
see the name of queue at QueueName=\private$\YourQueueName
Before delete file backup the file. Now delete that file.
Don't delete other file which does not have your queue name.
Do these things as your own risk.
Now start following services
Message Queuing Triggers,
Net.Msmq Listener Adapter
Message Queuing.
This trick work for me...
If you open Computer Management on the machine, expand the Services and Applications node (Features on 2008) and right-click on the Message Queuing service.
Right click on the Properties option and open it up to the General tab.
You can specify storage limits for messages -- you may have ran into the upper limit for messages storage. If you temporarily increase this value, it may allow you back into the messaging system so you can purge out those queues and restore operation.
Failing that, if you can deal with the loss of the messages (which if you are trying to purge I presume is okay), maybe delete the queue and recreate it.
Get hold of a copy of Queue Explorer - it's a commercial app but the trial is fully functional and it's worth it's weight in gold when debugging MSMQs http://www.cogin.com/mq/
If you don't have permissions though, then you don't have permissions! Are you a box admin? If you go to computermanagement and right click on one of your privete queues and select properties can you access the security tab and edit/see the permissions there?