I need to do some data analysis on xml data coming in http responses. Is there a way to set fiddler dump these responses automatically on disk?
(I have filtered the responses, dumping the sessions works too)
In a situation like this one I make a simple edit to the OnBeforeResponse function in the Fiddler rules. Choose Rules | Customize Rules and then add something similar to this::
if (oSession.url.Contains(".xml")) {
var directory: String = "C:\\Temp\\XML";
var path: String = System.IO.Path.Combine(directory, Guid.NewGuid() + ".xml");
oSession.SaveResponseBody(path);
}
The line that I am using to 'filter' the requests may not be appropriate for your situation -- you should attempt to repeat the filter condition that you used in the Fiddler UI.\
Hope that helps.
Related
Hope someone can help me explain some of my questions in order:
1. When i set application/config/config.php:
Determines whether the XSS filter is always active when GET, POST or
COOKIE data is encountered.
$config['global_xss_filtering'] = TRUE;
So if I set the default value is FALSE. What benefits will I get? For example, the performance or processing speed of the server?
2. Session
function save(){
$data = $this->input->post('number',TRUE);
$this->session->set_userdata('TEST',$data);
}
//Suppose Client request GET to action
function insert(){
$num = $this->session->userdata('TEST');
//Do I need to filter data in session?
$num_clean = $this->security->xss_clean($num );
$this->model->run_insert($num_clean);
}
I do not trust the user. And I still do not understand much about: session activity
The server just sends the ID Session to the client. Does the server send the data, which I set up to the session, to the client?
Best way xss_clean for session Which i am using is: Filter the client data by xss_clean input class. Is that enough? And need to re-filter session again?
Hope someone helped me because I just using only Codeigniter's XSS filter. Thanks
part 1:
From CodeIgniter User Guide Version 2.2.6
XSS Filtering
CodeIgniter comes with a Cross Site Scripting Hack prevention filter which can either run automatically to filter all POST and COOKIE data that is encountered, or you can run it on a per item basis. By default it does not run globally since it requires a bit of processing overhead, and since you may not need it in all cases.
It's not something that should be used for general runtime processing since it requires a fair amount of processing overhead.
So answerto your 1st part of question : yes ,
setting $config['global_xss_filtering'] = false; has performance benefits. also in codeigniter 3 its This feature is DEPRECATED. So i prefer to set it false.
part 2 :
Session is different from cookie
Unlike a cookie, the information is not stored on the users computer. So when you store a session ,its safe to trust the session data.
session data are stored in server. Most sessions set a user-key on the user's computer that looks something like this: 765487cf34ert8dede5a562e4f3a7e12. Then, when a session is opened on another page, it scans the computer for a user-key. If there is a match, it accesses that session, if not, it starts a new session.
here is a simple guide to session to read https://www.w3schools.com/php/php_sessions.asp
deftailed one : http://php.net/manual/en/intro.session.php
in short $num_clean = $this->security->xss_clean($num ); this is unnecessary.
I'm currently working on ng-admin.
I'm having a problem retrieving user data from my REST API (connected to a MongoDB) and displaying it.
I have identified the problem as the following:
When I enter http://localhost:3000/users into my browser, I get a list of all users in my database.
When I enter http://localhost:3000/users?_page=1&_perPage=30&_sortDir=DESC&_sortField=id,
I get [] as a result.
I am quite new to this, I used both my browser and the POSTMAN Chrome extension to test this and get the same result.
http://localhost:3000/users_end=30&_order=DESC&_sort=id&_start=0
This (/users_end) is a different request than /users.
It should be:
http://localhost:3000/users?end=30&_order=DESC&_sort=id&_start=0
Or, by looking at the other parameters:
http://localhost:3000/users?_end=30&_order=DESC&_sort=id&_start=0
with end or _end being the first parameter (mark the ?).
Update (it is ? and before the _, I have edited.):
If adding parameters to the request returns an empty list, try adding only one at a time to narrow down the problem (there's probably an error in the usage of those parameters - are you sure you need those underscores?).
Your REST API must have a way to handle pagination, sorting, and filtering. But ng-admin cannot determine exactly how, because REST is a style and not a standard. So ng-admin makes assumptions about how your API does that by default, that's why it adds these _end and _sort query parameters.
In order to transform these parameters into those that your API understands, you'll have to add an interceptor. This is all thoroughly explained in the ng-admin documentation: http://ng-admin-book.marmelab.com/doc/API-mapping.html
I am new to sails.js and I have a simple blueprint model set up. Right now, my controller and model are pretty much empty except for attribute definitions on the model.
After the model is created via POST, I would like the response to be a custom XML response (some plain text I generate essentially), not the standard JSON response. I figured that I could overwrite the entire create method on the controller, essentially copy-and-pasting the code from the default and just overwriting the response, but that seems really heavy too me.
There must be a better way to do this?
Note that I am only attempting to do this for this specific model, not generally.
Thanks!
The best way is to simply add the header as DigitalDesignDj mentioned.
/**
* TestController
*/
module.exports = {
create: function(res, req) {
// get your data
var xml = 'some xml string';
res.setHeader( "Content-type", "text/xml" );
res.send(xml);
}
}
To change headers for a specific response.
response.setHeader( "Content-type", "text/xml" );
When you already have some XML for the response.send()
If your result was to do this for all actions on that single model you could simply overwrite the toJSON method to generate XML instead of JSON in the model itself. Then if your running blueprints, it will spit out XML instead of json when you hit those endpoints.
However your question is specific to the create action. In this instance, I would ask if your running alternate view files for non ajax requests. And if not, just drop an view files into the create action that views/foo/create.[ejs,jade,ect...] with your xml layout. The response will see the view file and override the json output with that file. This means you have to change no code just add that single file.
Their are a dozen ways to accomplish this, and your question would need more detail (as mentioned on the comments) for a specific answer to your use case.
I'm experiencing unexpected behaviour while trying to access query string parameters in a mojolicious websocket request. Say my request looks like this:
ws://127.0.0.1:3000/websock_action?item_id=1234
Then in my mojo controller code I try and get the value of item_id in any of the following ways:
#in mojo controller
my $item_id = $self->param('item_id');
my $item_id = scalar $self->param('item_id');
my $item_id = scalar $self->tx->req->url->query->param('item_id');
The issue is that the item_id I get is often from a previous request, whichever of these techniques I use. My app is currently being served with hypnotoad.
Are query string parameters supported on websocket requests in mojolicious? Is there a more reliable way to access them? Essentially I'd like to know if I'm trying to something that isn't supported, so I can know whether the problem is something specific to my app.
Thanks in advance for any help
I suspect that what is happening, is that the parameters are passed in the html request, which is then upgraded to a websocket request at which point they are no longer available.
As Daren said, pass the data in the Web-Socket data. Something like...
var ws = $.websocket("ws://127.0.0.1:3000/websock_action", {
events: { message: function(e) {}
});
ws.send('message', 1234);
What I'm Trying To Do
I'm trying to create a solution of any kind that will run nightly on a Windows server, authenticate to a website, check a web page on the site for new links indicating a new version of a zip file, use new links (if present) to download a zip file, unzip the downloaded file to an existing folder on the server, use the unzipped contents (sql scripts, etc.) to build an instance of a database, and log everything that happens to a text file.
Forms App: The Part That Sorta Works
I created a Windows Forms app that uses a couple of WebBrowser controls, a couple of threads, and a few timers to do all that except the running nightly. It works great as a Form when I'm logged in and run it, but I need to get it (or something like it) to run on it's own like a Service or scheduled task.
My Service Attempt
So, I created a Windows Service that ticks every hour and, if the System.DateTime.Now.Hour >= 22, attempts to launch the Windows Forms app to do it's thing. When the Service attempts to launch the Form, this error occurs:
ActiveX control '8856f961-340a-11d0-a96b-00c04fd705a2' cannot be instantiated because the current thread is not in a single-threaded apartment.
which I researched and tried to resolve by either placing the [STAThread] attribute on the Main method of the Service's Program class or using some code like this in a few places including the Form constructor:
webBrowseThread = new Thread(new ThreadStart(InitializeComponent));
webBrowseThread.SetApartmentState(ApartmentState.STA);
webBrowseThread.Start();
I couldn't get either approach to work. In the latter approach, the controls on the Form (which would get initialized inside IntializeComponent) don't get initialized and I get null reference exceptions.
My Scheduled Task Attempt
So, I tried creating a nightly scheduled task using my own credentials to run the Form locally on my dev machine (just testing). It gets farther than the Service did, but gets hung up at the File Download Dialog.
Related Note: To send the key sequences to get through the File Download and File Save As dialogs, my Form actually runs a couple of vbscript files that use WScript.Shell.SendKeys. Ok, that's embarassing to admit, but I tried a few different things including SendMessage in Win32 API and referencing IWshRuntimeLibrary to use SendKeys inside my C# code. When I was researching how to get through the dialogs, the Win32 API seemed to be the recommended way to go, but I couldn't figure it out. The vbscript files was the only thing I could get to work, but I'm worried now that this may be the reason why a scheduled task won't work.
Regarding My Choice of WebBrowser Control
I have read about the System.WebClient class as an alternative to the WebBrowser control, but at a glance, it doesn't look like it has what I need to get this done. For example, I needed (or I think I needed) the WebBrowser's DocumentCompleted and FileDownload events to handle the delays in pages loading, files downloading, etc. Is there more to WebClient that I'm not seeing? Is there another class besides WebBrowser that is more Service-friendly and would do the trick?
In Summary
Geez, this is long. Sorry! It would help to even have a high level recommendation for a better way to do what I'm trying to do, because nothing I've tried has worked.
Update 10/22/09
Well, I think I'm closer, but I'm stuck again. I should end up with a decent-sized zip file with several files in it, but the zip file resulting from my code is empty. Here's my code:
// build post request
string targetHref = "http://wwwcf.nlm.nih.gov/umlslicense/kss/login.cfm";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(targetHref);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
// encoding to use
Encoding enc = Encoding.GetEncoding(1252);
// build post string containing authentication information and add to post request
string poststring = "returnUrl=" + fixCharacters(targetDownloadFileUrl);
poststring += getUsernameAndPasswordString();
poststring += "&login2.x=0&login2.y=0";
// convert to required byte array
byte[] postBytes = enc.GetBytes(poststring);
request.ContentLength = postBytes.Length;
// write post to request
Stream postStream = request.GetRequestStream();
postStream.Write(postBytes, 0, postBytes.Length);
postStream.Close();
// get response as stream
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
// writes stream to zip file
FileStream writeStream = new FileStream(fullZipFileName, FileMode.Create, FileAccess.Write);
ReadWriteStream(responseStream, writeStream);
response.Close();
responseStream.Close();
The code for ReadWriteStream looks like this.
private void ReadWriteStream(Stream readStream, Stream writeStream)
{
// taken verbatum from http://www.developerfusion.com/code/4669/save-a-stream-to-a-file/
int Length = 256;
Byte[] buffer = new Byte[Length];
int bytesRead = readStream.Read(buffer, 0, Length);
// write the required bytes
while (bytesRead > 0)
{
writeStream.Write(buffer, 0, bytesRead);
bytesRead = readStream.Read(buffer, 0, Length);
}
readStream.Close();
writeStream.Close();
}
The building of the post string is taken from my previous forms app that works. I compared the resulting values in poststring for both sets of code (my working forms app and this one) and they're identical.
I'm not even sure how to troubleshoot this further. Anyone see anything obvious as to why this isn't working?
Conclusion 10/23/09
I finally have this working. A couple of important hurdles I had to get over. I had some problems with the ReadWriteStream method code that I got online. I don't know why, but it wasn't working for me. A guy named JB in Claudio Lassala's Virtual Brown Bag meeting helped me to come up with this code which worked much better for my purposes:
private void WriteResponseStreamToFile(Stream responseStreamToRead, string zipFileFullName)
{
// responseStreamToRead will contain a zip file, write it to a file in
// the target location at zipFileFullName
FileStream fileStreamToWrite = new FileStream(zipFileFullName, FileMode.Create);
int readByte = responseStreamToRead.ReadByte();
while (readByte != -1)
{
fileStreamToWrite.WriteByte((byte)readByte);
readByte = responseStreamToRead.ReadByte();
}
fileStreamToWrite.Flush();
fileStreamToWrite.Close();
}
As Will suggested below, I did have trouble with the authentication. The following code is what worked to get around that issue. A few comments inserted addressing key issues I ran into.
string targetHref = "http://wwwcf.nlm.nih.gov/umlslicense/kss/login.cfm";
HttpWebRequest firstRequest = (HttpWebRequest)WebRequest.Create(targetHref);
firstRequest.AllowAutoRedirect = false; // this is critical, without this, NLM redirects and the whole thing breaks
// firstRequest.Proxy = new WebProxy("127.0.0.1", 8888); // not needed for production, but this helped in order to debug the http traffic using Fiddler
firstRequest.Method = "POST";
firstRequest.ContentType = "application/x-www-form-urlencoded";
// build post string containing authentication information and add to post request
StringBuilder poststring = new StringBuilder("returnUrl=" + fixCharacters(targetDownloadFileUrl));
poststring.Append(getUsernameAndPasswordString());
poststring.Append("&login2.x=0&login2.y=0");
// convert to required byte array
byte[] postBytes = Encoding.UTF8.GetBytes(poststring.ToString());
firstRequest.ContentLength = postBytes.Length;
// write post to request
Stream postStream = firstRequest.GetRequestStream();
postStream.Write(postBytes, 0, postBytes.Length); // Fiddler shows that post and response happen on this line
postStream.Close();
// get response as stream
HttpWebResponse firstResponse = (HttpWebResponse)firstRequest.GetResponse();
// create new request for new location and cookies
HttpWebRequest secondRequest = (HttpWebRequest)WebRequest.Create(firstResponse.GetResponseHeader("location"));
secondRequest.AllowAutoRedirect = false;
secondRequest.Headers.Add(HttpRequestHeader.Cookie, firstResponse.GetResponseHeader("Set-Cookie"));
// get response to second request
HttpWebResponse secondResponse = (HttpWebResponse)secondRequest.GetResponse();
// write stream to zip file
Stream responseStreamToRead = secondResponse.GetResponseStream();
WriteResponseStreamToFile(responseStreamToRead, fullZipFileName);
responseStreamToRead.Close();
sl.logScriptActivity("Downloading update.");
firstResponse.Close();
I want to underscore that setting AllowAutoRedirect to false on the first HttpWebRequest instance was critical to the whole thing working. Fiddler showed two additional requests that occurred when this was not set, and it broke the rest of the script.
You're trying to use UI controls to do something in a windows service. This will never work.
What you need to do is just use the WebRequest and WebResponse classes to download the contents of the webpage.
var request = WebRequest.Create("http://www.google.com");
var response = request.GetResponse();
var stream = response.GetResponseStream();
You can dump the contents of the stream, parse the text looking for updates, and then construct a new request for the URL of the file you want to download. That response stream will then have the file, which you can dump on the filesystem and etc etc.
Before you wonder, GetResponse will block until the response returns, and the stream will block as data is being received, so you don't need to worry about events firing when everything has been downloaded.
You definitely need to re-think your approach (as you've already begun to do) to eliminate the Forms-based application approach. The service you're describing needs to operate with no UI at all.
I'm not familiar with the details of System.WebClient, but since it
provides common methods for sending
data to and receiving data from a
resource identified by a URI,
it will probably be your answer.
At first glance, WebClient.DownloadFile(...) or WebClient.DownloadFileAsync(...) will do what you need.
The only thing I can add is that once you have scraped your screen and have the fully qualified name of the file you want to download, you could pass it along to the Windows/DOS command 'get' which will fetch files via HTTP. You can also script a command-line FTP client if desired. It's been a long time since I tried something like this in Windows, but I think you're almost there. Once you have fetched the correct file, building a batch file to do everything else should be pretty easy. If you are more comfortable with Unix, google "unix services for windows" just keep an eye on the services they start running (DHCP, etc). There are some nice utilities which will let your treat dos as a unix-like shell (ls -l, grep, etc) Finally, you could try another language like Perl or Python but I don't think that's the kind of advice you were looking for. :)