How to continuously load a saved flow file from mitmproxy? (i.e. "follow"/-f the file) - mitmproxy

I've saved a file with the --save-stream-file (save_stream_file) option.
Now I want to replay that as asked in my previous question, but I want it to continuously follow the saved "flow".
How can I do that?
This is a follow-up question of How to replay/show a saved flow in the console without triggering requests?.

After some asking I got around and this should do the trick:
tail -f ./mitmproxy-flow.cap | mitmproxy --rfile - -n
Thanks to #mhils!
Though, at least for me, v5.2 currently still has a bug where it crashes.

Related

Openpanel and symbol communication not working

I am trying to make a patch that plays audio when a bang is pressed. I have put a symbol so that I don't need to keep reimporting the file. However it works sometimes but not all the time.
A warning in the Pd console reads: Start requested with no prior open
However I have imported an audio file
Is there something that I have done wrong?
Use [trigger] to get the order-of-execution correct.
One problem is, that whenever you send a [1( to [readsf~] you must have sent an [open ...( message directly beforehand.
Even if you have just successfully opened a file, but then stopped it (with [0() or played it through (so it has been closed automatically), you have to send the filename again.
The real problem is, that your messages are out of order: you should never have a fan-out (that is: connecting a message outlet to multiple inlets), as this will create undefined behavior.
Use [trigger] to get the order-of-execution correct.
(Mastering [trigger] is probably the single most important step in learning to program Pd)

NSStream Event Timer - iPhone

All,
Is there a way to have a minimum time to keep a stream open before it closes? For some reason, my stream is closing prematurely which is causing errors elsewhere. I need to keep it open to make sure ALL of the data is gathered, and then it can run the other code.
Thanks,
James
In the case that someone falls upon this question later, I ended up creating nested if statements to pull it off.
Basically, there is one statement that checks if the end tag is not found (for my code, the END of the ENTIRE data that I should be receiving is </SessionData> - So, I did if([outputString rangeOfString:#"</SessionData>"].location == NSNotFound). I created a string called totalOutput that would have the outputString added onto the end of totalOutput until the </SessionData> is found.
If anyone ever needs help, just go ahead and comment on here and I can give more information.

the svnpoller is not triggered (warning in the twistd.log)

I am not sure what is going on, but i get this weird issue with buildbot.
The SVNPoller is configured as it should (checked various config example files), when i run the buildbot checkconfig it says that everything is fine....but it won't work at all.
If i trigger a build via the scheduler class it works fine, i can retrieve the source updates and build without problems (tried with a 1h timeframe).
The problem thou is that the poller is not working, so even if i build each hour, the changes column stays empty (i get the changes for the various versions thou, so if i click on the build detail i can see the sourcestamp carrying the right and most recent revision everytime that i modify the codebase); so I have no way to know if the build fails who did the last change.
Another peculiar thing is that in the twistd.log i see this line:
Warning: no ChangeSources specified in c['change_source']
And i am not sure why it wouldn't work since the checkconfig does not raise any error.
The result of this is of course that the only thing built is the hourly one, leaving me without the poller, and without knowing who is putting code in each build.
This is the code for the poller:
c['change source']=SVNPoller
(svnurl="svn+ssh://user#svnserver.domain.com/svn/project/trunk,
pollinterval=60*5,
histmax=10,
project=myproj,
svnbin = '/usr/bin/svn')
So far it looks good, so I am not really sure what is wrong here...why the SVNPoller is not triggering any build.
Anyone that has some suggestions about why is this happening ? Is there any other way to get changes from an SVN server? I am a total newbie at BuildBot and I am not really getting too much out of the manual; that looks much more like a scholastic book instead of being a manual that shows you how you do stuff :)
Thanks!!!!!
Ok, silly me :) the problem is the missing underscore on change_source...once added it the problem is solved
c['change_source'] = SVNPoller (svnurl=source_svn_url,
pollinterval=60,
histmax=10,
project='The_project',
svnbin= '/usr/bin/svn'
)
this will poll the svn codebase at source_svn_url (just put your svn:// path); and will check every minute to see if anyone has done changes; and will keep 10 changes in the record list (any change after the 10th will not show up so use it carefully if you do a lot of commits).
Hope that this helps who uses buildbot!

How can I get file size in Perl before processing an upload request?

I want to get file size I'm doing this:
my $filename=$query->param("upload_file");
my $filesize = (-s $filename);
print "Size: $filesize ";`
Yet it is not working. Note that I did not upload the file. I want to check its size before uploading it. So to limit it to max of 1 MB.
You can't know the size of something before uploading. But you can check the Content-Length request header sent by the browser, if there is one. Then, you can decide whether or not you want to believe it. Note that the Content-Length will be the length of the entire request stream, including other form fields, and not just the file upload itself. But it's sufficient to get you a ballpark figure for conformant clients.
Since you seem to be running under plain CGI, you should be able to get the request body length in $ENV{CONTENT_LENGTH}.
Also want to sanity check against possibly already having post max set (from perldoc CGI):
$CGI::POST_MAX
If set to a non-negative integer, this variable puts a ceiling on the size of
POSTings, in bytes. If CGI.pm detects a POST that is greater than the ceiling,
it will immediately exit with an error message. This value will affect both
ordinary POSTs and multipart POSTs, meaning that it limits the maximum size of
file uploads as well. You should set this to a reasonably high value, such as
1 megabyte.
The uploaded file is stashed in a tmp location on the server when the form is submitted, check the file size there.
Supply the value for $field.
my $upload_filehandle = $query->upload($field);
my $tmpfilename = $query->tmpFileName($upload_filehandle);
my $file_size = (-s $tmpfilename);
This has nothing to do with Perl.
You are trying to read the filesize of a file on the user's computer using commands that read files on your server, what you want can't be done using Perl.
This is something that has to be done in the browser, and looking briefly at these questions it's either very hard or impossible.
Your best bet is to allow the user to start the upload and abort if the file is too big.
If you want to check before you process the request, you might be better off checking on the web page that triggers the request. I don't think the web browser can do it on it's own, but if you don't mind Flash, there are many Flash upload tools that can check things like size (as well as file types) and prevent uploading.
A good one to start with is the YUI Uploader. Lots more here: What is the best multiple file JavaScript / Flash file uploader?
Obviously you would want to check on the server side too, but by the time the user has started sending the request to the server, you are already using up your CPU cycles and bandwidth.
Thanks everyone for your replies; I just found out why $filesize = (-s $filename); was not working before, it is due that I was checking file size while sending Ajax request and not while re submitting the page.That's why I was having size to be zero. I fixed that to submit the page and it worked. Thanks.
Just read this post but while checking the content-length is a good approximate pre-check you could also save the file to temporary folder and then perform any kind of check on it. If it doesn't meet your criteria just delete and don't send it to it's final destination.
Look at the perl documentation for file stats -X - perldoc.perl.org and stat-perldoc.perl.org. Also, you can look at this upload script which is doing the similar thing what you are trying to do.

How can I validate an image file in Perl?

How would I validate that a jpg file is a valid image file. We are having files written to a directory using FTP, but we seem to be picking up the file before it has finished writing it, creating invalid images. I need to be able to identify when it is no longer being written to. Any ideas?
Easiest way might just be to write the file to a temporary directory and then move it to the real directory after the write is finished.
Or you could check here.
JPEG::Error
[arguments: none] If the file reference remains undefined after a call to new, the file is to be considered not parseable by this module, and one should issue some error message and go to another file. An error message explaining the reason of the failure can be retrieved with the Error method:
EDIT:
Image::TestJPG might be even better.
You're solving the wrong problem, I think.
What you should be doing is figuring out how to tell when whatever FTPd you're using is done writing the file - that way when you come to have the same problem for (say) GIFs, DOCs or MPEGs, you don't have to fix it again.
Precisely how you do that depends rather crucially on what FTPd on what OS you're running. Some do, I believe, have hooks you can set to trigger when an upload's done.
If you can run your own FTPd, Net::FTPServer or POE::Component::Server::FTP are customizable to do the right thing.
In the absence of that:
1) try tailing the logs with a Perl script that looks for 'upload complete' messages
2) use something like lsof or fuser to check whether anything is locking a file before you try and copy it.
Again looking at the FTP issue rather than the JPG issue.
I check the timestamp on the file to make sure it hasn't been modified in the last X (5) mins - that way I can be reasonably sure they've finished uploading
# time in seconds that the file was last modified
my $last_modified = (stat("$path/$file"))[9];
# get the time in secs since epoch (ie 1970)
my $epoch_time = time();
# ensure file's not been modified during the last 5 mins, ie still uploading
unless ( $last_modified >= ($epoch_time - 300)) {
# move / edit or what ever
}
I had something similar come up once, more or less what I did was:
var oldImageSize = 0;
var currentImageSize;
while((currentImageSize = checkImageSize(imageFile)) != oldImageSize){
oldImageSize = currentImageSize;
sleep 10;
}
processImage(imageFile);
Have the FTP process set the readonly flag, then only work with files that have the readonly flag set.