when ever i am consuming a web service xml is received and sent right...is there a way to view this xml...i need to see the exact structure to fine the errors...
It really depends on the library/language/framework you are using. Most of them can dump request/response messages to a log file.
Alternatively you can always use Ethereal/Wireshark to sniff the network and see what is actually going on (assuming your stream is not encrypted)
One "brute force" way would be to use Wireshark to capture the network traffic, right click one of the packets from the exchange and do "Follow TCP Stream", which will give you8 a nice view of both the sent and recieved XML in one spot.
Related
I'm trying to enhance a server-app-website architecture in reliability, another programmer has developed.
At the moment, android smartphones start a tcp connection to a server component to exchange data. The server takes the data, writes them into a DB and another user can have a look on the data through a website. The problem is that the smartphones very regularly are in locations where connectivity is really bad. The consequence is that the smartphones lose the tcp connection and it's hard to reconnect. Now my question is, if there are any protocols that are so lightweight or accomodating concerning bad connectivity that the data exchange could work better or more reliable.
For example, I was thinking about replacing the raw TCP interface with a RESTful API, but I don't really know how well REST works in this scenario, as I don't have any experience in this area.
Maybe useful to know for answering this question: The server component is programmed in c#. The connecting components are android smartphones.
Please understand that I dont add some code to this question, because in my opinion its just a theoretically question.
Thank you in advance !
REST runs over HTTP which runs over TCP so it would have the same issues with connectivity.
Moving up the stack to the application you could perhaps think in terms of 'interference'. I quite often have to use technical stuff in remote areas with limited reception and it reminds of trying to communicate in a storm. If you think about it, if you're trying to get someone to do something in a storm where they can hardly hear you and the words get blown away (dropped signal), you don't read them the manual on how to fix something, you shout key words such as 'handle', 'pull', 'pull', 'PULL', 'ok'. So the information reaches them in small bursts you can repeat (pull, what? pull, eh? PULL! oh righto!)
Can you redesign the communications between the android app and the server so the server can recognise key 'words' with corresponding data and build up the request over a period of time? If you consider idempotency, each burst of data would not alter the request if it has already been received (pull, PULL!) and over time the android app could send/receive smaller chunks of the request. If the signal stays up, just keep sending. If it goes down, note which parts of the request haven't been sent and retry them when the signal comes back.
So you're sending the request jigsaw-style but the server knows how to reassemble the pieces in the right order. A STOP word at the end tells the server ok this request is complete, go work on it. Until that word arrives the server can store the incomplete request or discard it if no more data comes in.
If the server respond to the first request chunk with an id, the app can use the id to get the response and keep trying until the full response comes back, at which point the server can remove the response from its jigsaw cache. A fair amount of work though.
I have a set up that involves exactly one client and one server. The client can generate 32K data really fast. As I generate that data I'd like to send it in parallel over TCP to the server and have it reassembled in the same order that I sent it out in.
So I have the thought where I add each 32K packet to a queue on the client, then something sends out those packets in parallel. On the server, those packets are received in some random order, but then put back in order into a queue and then I can simple dequeue the packets. I have a picture of this set up:
Does this set up have a name? Fanout/fanin? What should I be searching for? It seems middleware such as ZeroMQ might help, but I haven't been able to see any specific examples that display this type of architecture.
I assume this is a solved problem with some nice open source libraries out there, but my assumptions have been wrong in the past.
Thanks for any help.
I want to capture and save the data that's being sent to a certain process through internet .
Are there any tools for the job?
If not, does listening to the same port as the process that I'm trying to get data from, will get me the data?
Any help appreciated !
You can try Wireshark: http://www.wireshark.org/
Or RawCap: http://www.netresec.com/?page=RawCap
I don't know what is the data format you are trying to capture. I used these two tools to capture xml data from web service.
On Windows, use Winsock Packet Editor (WPE). You will be able to hook a process' all Winsock-related functions and capture (and even modify/block) any TCP/IP, UDP packets that the application receives or sends. For all other operating systems, you will have to either:
write your own tool that hooks various socket functions (e.g. send, recv, etc.)
or just use Wireshark which will capture all Layer-3 packets that goes through your network card. You will have to use your own knowledge of the application that you're trying to monitor in order to filter the packets that are specific to the application.
Are there any tools for the job?
Wireshark. But what have you tried?
If not, does listening to the same port as the process that i'm trying to get data from, will get me the data?
Not if you don't forward the traffic to the real destination, otherwise the other party will be waiting forever on a response, or simply timeout and close the connection. You should create something like a proxy.
Wireshark is easier.
I am trying to build an application that can inform a user about website specific information whenever they are visiting a website that is present in my database. This must be done in a browser independent way so the user will always see the information when visiting a website (no matter what browser or other tool he or she is using to visit the website).
My first (partially successful) approach was by looking at the data packets using the System.Net.Sockets.Socket class etc. Unfortunately I discoverd that this approach only works when the user has administrator rights. And of course, that is not what I want. My goal is that the user can install one relatively simple program that can be used right away.
After this I went looking for alternatives and found a lot about WinPcap and some of it's .NET wrappers (did I tell you I am programming c# .NET already?). But with WinPcap I found out that this must be installed on the user's pc and there is nog way to just reference some dll files and code away. I already looked at including WinPcap as a prerequisite in my installer but that is also to cumbersome.
Well, long story short. I want to know in my application what website my user is visiting at the moment it is happening. I think it must be done by looking at the data packets of the network but can't find a good solution for this. My application is build in C# .NET (4.0).
You could use Fiddler to monitor Internet traffic.
It is
a Web Debugging Proxy which logs all HTTP(S) traffic between your computer and the Internet. Fiddler allows you to inspect traffic, set breakpoints, and "fiddle" with incoming or outgoing data. Fiddler includes a powerful event-based scripting subsystem, and can be extended using any .NET language.
It's scriptable and can be readily used from .NET.
One simple idea: Instead of monitoring the traffic directly, what about installing a browser extension that sends you the current url of the page. Then you can check if that url is in your database and optionally show the user a message using the browser extension.
This is how extensions like Invisible Hand work... It scans the current page and sends relevant data back to the server for processing. If it finds anything, it uses the browser extension framework to communicate those results back to the user. (Using an alert, or a bar across the top of the window, etc.)
for a good start, wireshark will do what you want.
you can specify a filter to isolate and view http streams.
best part is wireshark is open source, and built opon another program api, winpcap which is open source.
I'm guessing this is what you want.
capture network data off the wire
view the tcp traffic of a computer, isolate and save(in part or in hole) http data.
store information about the http connections
number 1 there is easy, you can google for a winpcap tutorial, or just use some of their sample programs to capture the data.
I recomend you study up on the pcap file format, everything with winpcap uses this basic format and its structers.
now you have to learn how to take a tcp stream and turn it into a solid data stream without curoption, or disorginized parts. (sorry for the spelling)
again, a very good example can be found in the wireshark source code.
then with your data stream, you can simple read the http format, and html data, or what ever your dealing with.
Hope that helps
If the user is cooperating, you could have them set their browser(s) to use a proxy service you provide. This would intercept all web traffic, do whatever you want with it (look up in your database, notify the user, etc), and then pass it on to the original location. Run the proxy on the local system, or on a remote system if that fits your case better.
If the user is not cooperating, or you don't want to make them change their browser settings, you could use one of the packet sniffing solutions, such as fiddler.
A simple stright forward way is to change the comupter DNS to point to your application.
this will cause all DNS traffic to pass though your app which can be sniffed and then redirected to the real DNS server.
it will also save you the hussel of filtering out emule/torrent traffic as it normally work with pure IP address (which also might be a problem as it can be circumvented by using IP address to browse).
-How to change windows DNS Servers
-DNS resolver
Another simple way is to configure (programmaticly) the browsers proxy to pass through your server this will make your life easier but will be more obvious to users.
How to create a simple proxy in C#?
I'm working on an iPhone application which will use long-polling to send event notifications from the server to the client over HTTP. After opening a connection on the server I'm sending small bits of JSON that represent events, as they occur. I am finding that -[NSURLConnectionDelegate connection:didReceiveData] is not being called until after I close the connection, regardless of the cache settings I use when creating the NSURLRequest. I've verified that the server end is working as expected - the first JSON event will be sent immediately, and subsequent events will be sent over the wire as they occur. Is there a way to use NSURLConnection to receive these events as they occur, or will I need to instead drop down to the CFSocket API?
I'm starting to work on integrating CocoaAsyncSocket, but would prefer to continue using NSURLConnection if possible as it fits much better with the rest of my REST/JSON-based web service structure.
NSURLConnection will buffer the data while it is downloading and give it all back to you in one chunk with the didReceiveData method. The NSURLConnection class can't tell the difference between network lag and an intentional split in the data.
You would either need to use a lower-level network API like CFSocket as you mention (you would have access to each byte as it comes in from the network interface, and could distinguish the two parts of your payload), or you could take a look at a library like CURL and see what types of output buffering/non-buffering there is there.
I ran into this today. I wrote my own class to handle this, which mimics the basic functionality of NSURLConnection.
http://github.com/nall/SZUtilities/blob/master/SZURLConnection.h
It sounds as if you need to flush the socket on the server-side, although it's really difficult to say for sure. If you can't easily change the server to do that, then it may help to sniff the network connection to see when stuff is actually getting sent from the server.
You can use a tool like Wireshark to sniff your network.
Another option for seeing what's getting sent/received to/from the phone is described in the following article:
http://blog.jerodsanto.net/2009/06/sniff-your-iphones-network-traffic/
Good luck!
We're currently doing some R&D to port our StreamLink comet libraries to the iPhone.
I have found that in the emulator you will start to get didReceiveData callbacks once 1KB of data is received. So you can send a junk 1KB block to start getting callbacks. It seems that on the device, however, this doesn't happen. In safari (on device) you need to send 2KB, but using NSURLConnection I too am getting no callbacks. Looks like I may have to take the same approach.
I might also play with multipart-replace and some other more novel headers and mime types to see if it helps stimulate NSURLConnection.
There is another HTTP API Implementation named ASIHttpRequest. It doesn't have the problem stated above and provides a complete toolkit for almost every HTTP feature, including File Uploads, Cookies, Authentication, ...
http://allseeing-i.com/ASIHTTPRequest/