I have a budget minded client who want to see when there is a ddos attack on the network so they can blackhole route the attack with a homespun tool. Currently they have devices exporting only sflow traffic. The sysadmin says that they need different hardware and full netflow export for ddos detection, is that accurate?
In my experience sFlow is actually really good for fast DDoS detection, at least for volumetric DDoS attacks like reflection attacks or packet floods. The reason for this lies in the differences between sFlow and NetFlow.
NetFlow keeps state in the router, and if the flow is either inactive for awhile (usually 15 seconds or so), or goes on long (usually 60 seconds) then the summary state of that flow is sent to the collector. This means an accurate account of the traffic is made, but might not arrive at your detector until the attack is already a minute underway!
Unlike NetFlow, the sFlow strategy is to send packet samples every 1 in N (usually 1/512 or 1/1024 or so). This means your detection software can "see" the attack almost immediately!
So stick with the sFlow exports, no need to add hardware. Here is some additional detail on the differences between NetFlow and sFlow: http://www.flowtraq.com/corporate/resources/whitepapers/the-netflowsflowcflowjflow-flow-dilemma/
There is no need to replace the network hardware, sFlow works well for DDoS detection. Here are a couple of DDoS mitigation tools on GitHub:
FastNetMon
Blackhole DDoS
Actually you definitely could use sFLOW for DDoS detection. As FastNetMon's author I could recommend to use sflow instead netflow. But please keep in mind you should select correct sflow sampling_rate for it.
Please check this awesome reference about selecting correct sampling rate according on your traffic amount: http://blog.sflow.com/2009/06/sampling-rates.html
Related
Is anyone aware of any open source codes for Netflow Anomaly detection for DDOS and tunneling? I am a newbie in this area . I did find very few on github but anyone who has more experience with it, do advise.
Just want to try a few to understand how they work so language of python or r or c++ language are fine
There's some great resources around for ingesting the various flow formats. The harder part is in doing anomaly detection. You could consider 'R', see for instance: http://www.ojscurity.com/2014/10/r-netflow-analytics-i.html
When trying to detect tunneling you will need to establish one or more metrics that you can use to "profile" the traffic. Typically this would be on a per-endpoint, per-protocol basis. For instance, HTTPS traffic to Amazon looks different than watching NetFlix content. The metrics you establish should enable you detect a chance in the typical pattern for a given type of traffic.
So it might be hard to detect HTTP traffic tunneled over HTTPS by using just flow data. However, tunneling HTTP traffic over DNS should be fairly easy to detect due to the different volumetric and session timing characteristics of each protocol.
DDoS is more straightforward, and can be detected by a volumetric "baseline", since typical attacks are extremely loud in nature. Although, the more specific you get in terms of protocol, and type of packet, the faster and more accurate your DDoS detection will be.
Finally, the more you "know" about the network you are monitoring, the better you are able to pick up anomalies. There are some obvious first-principles here, as DDoS attacks are loud, and most protocols have fairly well-known volume/timing characteristics, but learning what is typical for your network is the best way to reduce false positives.
I want write a code to get the available bandwidth.
Using one of the algorithm.ex.spruce / pathload.
I wanted to a code in C++ in Windows.
I have got linux code .
But i wanted a Windows based code , which can get me up and down bandwidth.
Bandwidth for what resource? If this is a network resource there isn't anything in any language or the OS that will give you any real estimation of bandwidth. You would need to call out to something at the other end of the link you need to traverse and get an estimation of bandwidth at that point in time.
Or better said... You would need to call a file on a web server to test the download speed of someone's home Internet connection. Keep in mind that the numbers obtained are only accurate for that point in time though. As the bandwidth on any resource can be higher or lower when you actually use them since external factors always affect bandwidth (other prorcesses, users, etc.)
Why do you need the bandwidth and for what resource?
If you asking, you not up to it. Converting linux to windows requires knowledge of both platform, which you clearly doesnt have.
In my experience, almost all network friendly bandwidth estimation algorithm (pathload, pathchirp etc) are unsuitable for high speed bandwidth. Those old algorithm are suitable and practical if the bandwidth is around 1mb. Also, these algorithm assume the network is 'clean'(no other traffic). Nowadays, almost all of these 'network friendly' algorithm is not practical.
Other variant bandwidth estimation tools like netperf, netcps is based on brute force method. Brute force method are not network friendly. Most of this algorithm have problem with latency(if tcp based) and reached hdd read/write speed(if write to hdd instead of memory).
IMO, the best bandwidth estimation tools is UDP based(not influenced by latency unlike tcp) brute force(not influenced by other traffic) with custom made control flow tuned for high speed networks.
Other problem you will encounter is code optimization. You must ensure that your code is highly optimized. If you use c#, GC will pose a possible problem.
I need to know what internet connection is available when my application is running. I checked out the Reachability example from Apple, but this differs only between wifi and carrier network. What I need to know is what carrier network is selected, UMTS or EDGE or GPRS.
Currently, this information is not available. If you want this feature, file a new bug and mention that this is a duplicate of bug 6014806.
You could take a guess at what kind of network you are on by checking the latency of a round trip to your server. If you are getting figures of under 100ms, you are almost certainly on WiFi.
GPRS and EDGE run at around 600ms latency. UMTS/HSDPA is 100-200ms.
Source: my informal testing, and [AT&T][1] figures.
Rather than hardcoding different versions of your site for 3G, EDGE, GPRS, wifi broadband, why not build a framework which detects connection speed and bootstraps your site up to the appropriate level of bandwidth? That way you would get appropriate results on slow 3G / wifi, and it would naturally scale to the next generation of wireless broadband (e.g. WiMax and 802.11n) with a minimal amount of effort / disruption.
For example, you could determine different bandwidth "checkpoints" (which may correspond to 3G, EDGE, etc.), then you could do something like transfer some small bit of data or cache a small image (such as an icon) common to all bandwidth levels, benchmark the download speed in the background and set the bandwidth level accordingly.
File only
I like Wedge's answer. I'm not sure that the file wouldn't be cached by ISPs though. You could always keep generating a new file name or choose one big enough that you only test for long enough to get a result.
Simple latency
The idea of using latency is close but as Shivan mentioned it's inaccurate. A user in Australia to UK will get a latency of around 350ms vs the local user who could see it as low as 30-40ms
Solution: Mean deviation
If you ping your server with 3 packets and then look at the mean deviation (mdev) under 3G it's usually under 50ms. With 2G/EDGE it's almost always over 100ms. I got one outlier at 65ms to AUS.
My tests found a range of 4ms-38ms, with only one exception on a test to Australia from Belgium at 202ms.
Hope that's useful to someone..
I plan on writing an automated bot for a game.
The tricky part is figuring out how they encoded their protocol... To make the bot run around is easy, simply make the character run and record what it does in wireshark. However, interpreting the environment is more difficult... It recieves about 5 packets each second if you are idle, hence lots of garbarge.
My plan: Because the game runs under TCP, I will use freecap (http://www.freecap.ru/eng) to force the game to connect to a proxy running on my machine. I will need this proxy to be capable of packet injection, or perhaps a server that is capable of resending captured packets. This way I can recreate and tinker around with what the server sends, and understand their protocol encoding.
Does anyone know where I can get a proxy that allows packet injection or where I can perform packet injection (not via hardware, as is the case with wireless or anything!)
Where of if I can find a server/proxy that resends captured packets (ie: replays a connection).
Any better tools or methodologies for pattern matching? Something which can highlight patterns from mutliple messages would be GREAT.
OR, is there a better way to decipher this here? Possibly a dissasembly strategy (via hooking a winsock function and starting the dissassembly from there) ? I have not done this before so I am not sure. OR , any other ideas?
Network traffic interception and protocol analysis is generally a less favored method to accomplish your goal here. For most modern games, encryption is a serious factor, and there are serious headaches associated with the protocol analysis for any but trivial factors of the most common gameplay scenarios.
Most modern implementations* of what you are trying to do rely on reading and manipulating the memory space and process of a running client. The client will have already done all the hard parts for you, including decrypting the traffic and sorting it into far more easy to read data structures. For interacting with the server you can call functions built into the client instead of crafting entire series of packets from scratch. The plus to this approach is that you have to do far less work to interpret the data and produce activity. The minus is that there is often some data in the network traffic that would be useful to a bot but is discarded by the client, or that you may want to send traffic to the server that the client cannot produce (which, in my own well-developed hierarchy for such, is a few steps farther down the "cheating" slope).
*...I say this having seen the evolution of the majority of MMORPG botting/hacking communities from network protocol analyzers like ShowEQ and Odin's Eye / Excalibur to memory-based applications like MacroQuest and InnerSpace. On that note, InnerSpace provides an excellent extensible framework for the memory/process-based variant of what you are attempting, and you should look into it as a basis for your project if you abandon the network analysis approach.
As I've done a few game bots in the past (for fun, not profit or griefing of course - writing game bots is a lot of fun), I recommend the following:
If you can code and there isn't cheat protection preventing you from doing it, I highly recommend writing an injected DLL for the following reasons:
Your DLL will be able to access the game's memory space directly, and once you reverse-engineer the data structures (either by poking around memory or by code disassembly), you'll have access to lots of data. This will also allow you to bypass any network encryption the game may have. The downside of accessing process memory directly is that offsets and data structures change between versions - however, data structures don't change very often with a stable game, and you can compensate offset changes by searching for code patterns instead of using fixed offsets.
Either way, you'll still be able to hook WinSock functions using API hooks (check out Microsoft Detours and the excellent but now-commercial madCodeHook).
otherwise, I can only advise that you give live/interactive packet editors like WPE Pro a try.
In most scenarios, the coolest methods (code reverse-engineering and direct memory access) tend to be the least productive. They require a lot of skill (to understand the code) and time, both initially (to go through all the code and develop code to interact with the data structure) and for maintainance (in case the game is being updated). (Of course, they sometimes do allow doing cool stuff which is impossible to do with the official client, but most of the time this is obvious as blatant cheating, and likely to attract the GMs quickly). Most of the time bots are made by replacing game graphics/textures with solid colours, and creating simple "pixel" bots which search for certain colours on the screen and react accordingly (e.g. click them).
Hope this helps, and remember - cheating is only fun when it doesn't make the game less fun for everyone else ;)
There are probably a few reasonable assumptions you can make that should simplify your task enormously. However, to make the best use of them you will probably need greater comfort with sleeves-rolled-up programming than it sounds like you have.
First, it's a safe bet that the encryption they are using falls into one of three categories:
None
Cheesy
Far better than you are likely to crack
With the odds of the middle case being very low.
Next, it's a safe bet that the packets are encrypted / decrypted close to the edge of the program (right as they come in, right before they go out) and that the body of the game deals with them in decrypted form.
Finally, the protocol they are using most likely consists of either
ascii with data blocks
binary goo
So do a little packet sniffing with a card set in promiscuous mode for unencrypted ascii. If you see some, great, you're ahead of the game. But if you don't give up the whole tapping-the-line idea and instead start following the code as it returns from the sending data out by breakpointing and stepping with a debugger. Figure the outermost layer or three will be standard network stuff, then will come the encryption layer, and beyond that the huge mass of stuff that deals with the protocol unencrypted.
You should be able to get this far in an hour if you're hot, a weekend if you're reasonably skilled, motivated, and diligent, and never if you are hopeless. But it is possible in principle (and doubtlessly far easier in practice) to do it this way.
Once you get to where something that looks like unencrypted goo comes in, gets mungled, and the mungled form goes out, then start worrying about what it means.
-- MarkusQ
A) I play a MMO and do not support bots, voting down...
B) Download Backtrack v.3, run an arpspoof on your default gateway and your host. There is an application that will spoof the remote host's SSL cert sslmitm (I believe is the name) which will then allow you to create a full connection through your host. Then fireup tcpdump/ethereal/wireshark (choose your pcap poison) and move around do random stuff to find out what packet is doing what. That will be your biggest challenge; but proxying with a Man in the Middle attack on yourself is the way to go.
C) I do not condone this activity, this information is only being provided as free information.
Sounds like there is not encryption going on, so you could do a network approach.
A great place to start would be to find the packet ID's - most of the time, something near the front of the packet is going to be an ID of the type of the packet. For example move could be 1, shoot fired could be "2", chat could be "4".
You can write your own proxy that listens on one port for your game to connect, and then connects to the server. You can make keypresses to your proxy fire off commands, or you can make your proxy write out debugging info to help you go further.
(I've written a bot for an online in game in PHP - of all things.)
If you have a situation where a TCP connection is potentially too slow and a UDP 'connection' is potentially too unreliable what do you use? There are various standard reliable UDP protocols out there, what experiences do you have with them?
Please discuss one protocol per reply and if someone else has already mentioned the one you use then consider voting them up and using a comment to elaborate if required.
I'm interested in the various options here, of which TCP is at one end of the scale and UDP is at the other. Various reliable UDP options are available and each brings some elements of TCP to UDP.
I know that often TCP is the correct choice but having a list of the alternatives is often useful in helping one come to that conclusion. Things like Enet, RUDP, etc that are built on UDP have various pros and cons, have you used them, what are your experiences?
For the avoidance of doubt there is no more information, this is a hypothetical question and one that I hoped would elicit a list of responses that detailed the various options and alternatives available to someone who needs to make a decision.
What about SCTP. It's a standard protocol by the IETF (RFC 4960)
It has chunking capability which could help for speed.
Update: a comparison between TCP and SCTP shows that the performances are comparable unless two interfaces can be used.
Update: a nice introductory article.
It's difficult to answer this question without some additional information on the domain of the problem.
For example, what volume of data are you using? How often? What is the nature of the data? (eg. is it unique, one off data? Or is it a stream of sample data? etc.)
What platform are you developing for? (eg. desktop/server/embedded)
To determine what you mean by "too slow", what network medium are you using?
But in (very!) general terms I think you're going to have to try really hard to beat tcp for speed, unless you can make some hard assumptions about the data that you're trying to send.
For example, if the data that you're trying to send is such that you can tolerate the loss of a single packet (eg. regularly sampled data where the sampling rate is many times higher than the bandwidth of the signal) then you can probably sacrifice some reliability of transmission by ensuring that you can detect data corruption (eg. through the use of a good crc)
But if you cannot tolerate the loss of a single packet, then you're going to have to start introducing the types of techniques for reliability that tcp already has. And, without putting in a reasonable amount of work, you may find that you're starting to build those elements into a user-space solution with all of the inherent speed issues to go with it.
ENET - http://enet.bespin.org/
I've worked with ENET as a reliable UDP protocol and written an asynchronous sockets friendly version for a client of mine who is using it in their servers. It works quite nicely but I don't like the overhead that the peer to peer ping adds to otherwise idle connections; when you have lots of connections pinging all of them regularly is a lot of busy work.
ENET gives you the option to send multiple 'channels' of data and for the data sent to be unreliable, reliable or sequenced. It also includes the aforementioned peer to peer ping which acts as a keep alive.
We have some defense industry customers that use UDT (UDP-based Data Transfer) (see http://udt.sourceforge.net/) and are very happy with it. I see that is has a friendly BSD license as well.
Anyone who decides that the list above isn't enough and that they want to develop their OWN reliable UDP should definitely take a look at the Google QUIC spec as this covers lots of complicated corner cases and potential denial of service attacks. I haven't played with an implementation of this yet, and you may not want or need everything that it provides, but the document is well worth reading before embarking on a new "reliable" UDP design.
A good jumping off point for QUIC is here, over at the Chromium Blog.
The current QUIC design document can be found here.
RUDP - Reliable User Datagram Protocol
This provides:
Acknowledgment of received packets
Windowing and congestion control
Retransmission of lost packets
Overbuffering (Faster than real-time streaming)
It seems slightly more configurable with regards to keep alives then ENet but it doesn't give you as many options (i.e. all data is reliable and sequenced not just the bits that you decide should be). It looks fairly straight forward to implement.
As others have pointed out, your question is very general, and whether or not something is 'faster' than TCP depends a lot on the type of application.
TCP is generally as fast as it gets for reliable streaming of data from one host to another. However, if your application does a lot of small bursts of traffic and waiting for responses, UDP may be more appropriate to minimize latency.
There is an easy middle ground. Nagle's algorithm is the part of TCP that helps ensure that the sender doesn't overwhelm the receiver of a large stream of data, resulting in congestion and packet loss.
If you need the reliable, in-order delivery of TCP, and also the fast response of UDP, and don't need to worry about congestion from sending large streams of data, you can disable Nagle's algorithm:
int opt = -1;
if (setsockopt(sock_fd, IPPROTO_TCP, TCP_NODELAY, (char *)&opt, sizeof(opt)))
printf("Error disabling Nagle's algorithm.\n");
If you have a situation where a TCP connection is potentially too slow and a UDP 'connection' is potentially too unreliable what do you use? There are various standard reliable UDP protocols out there, what experiences do you have with them?
The key word in your sentence is 'potentially'. I think you really need to prove to yourself that TCP is, in fact, too slow for your needs if you need reliability in your protocol.
If you want to get reliability out of UDP then you're basically going to be re-implementing some of TCP's features on top of UDP which will probably make things slower than just using TCP in the first place.
Protocol DCCP, standardized in RFC 4340, "Datagram Congestion Control Protocol" may be what you are looking for.
It seems implemented in Linux.
May be RFC 5405, "Unicast UDP Usage Guidelines for Application Designers" will be useful for you.
RUDP. Many socket servers for games implement something similar.
Did you consider compressing your data ?
As stated above, we lack information about the exact nature of your problem, but compressing the data to transport them could help.
It is hard to give a universal answer to the question but the best way is probably not to stay on the line "between TCP and UDP" but rather to go sideways :).
A bit more detailed explanation:
If an application needs to get a confirmation response for every piece of data it transmits then TCP is pretty much as fast as it gets (especially if your messages are much smaller than optimal MTU for your connection) and if you need to send periodic data that gets expired the moment you send it out then raw UDP is the best choice for many reasons but not particularly for speed as well.
Reliability is a more complex question, it is somewhat relative in both cases and it always depends on a specific application. For a simple example if you unplug the internet cable from your router then good luck keeping reliably delivering anything with TCP. And what even worse is that if you don't do something about it in your code then your OS will most likely just block your application for a couple of minutes before indicating an error and in many cases this delay is just not acceptable as well.
So the question with conventional network protocols is generally not really about speed or reliability but rather about convenience. It is about getting some features of TCP (automatic congestion control, automatic transmission unit size adjustment, automatic retransmission, basic connection management, ...) while also getting at least some of the important and useful features it misses (message boundaries - the most important one, connection quality monitoring, multiple streams within a connection, etc) and not having to implement it yourself.
From my point of view SCTP now looks like the best universal choice but it is not very popular and the only realistic way to reliably pass it across the Internet of today is still to wrap it inside UDP (probably using sctplib). It is also still a relatively basic and compact solution and for some applications it may still be not sufficient by itself.
As for the more advanced options, in some of the projects we used ZeroMQ and it worked just fine. This is a much more of a complete solution, not just a network protocol (under the hood it supports TCP, UDP, a couple of higher level protocols and some local IPC mechanisms to actually deliver messages). Since a couple of releases its initial developer has switched his attention to his new NanoMSG and currently the newest NNG libraries. It is not as thoroughly developed and tested and it is not very popular but someday it may change. If you don't mind the CPU overhead and some network bandwidth loss then some of the libraries might work for you. There are some other network-oriented message exchange libraries available as well.
You should check MoldUDP, which has been around for decades and it is used by Nasdaq's ITCH market data feed. Our messaging system CoralSequencer uses it to implement a reliable multicast event-stream from a central process.
Disclaimer: I'm one of the developers of CoralSequencer
The best way to achieve reliability using UDP is to build the reliability in the application program itself( for example, by adding acknowledgment and retransmission mechanisms)