I try to discover what kind of applications work in my network (e.g. Facebook, Youtube, Twitter etc.) . Unfortunatelly I can't do Deep Packet Inspection, everything I have are NetFlow traces. I was thinking about resolving ip addresses using DNS server and check domain names of flows. But what if application use domain that doesn't contain app name? Is that any possibility to find all ip addresses that use specific app/website?
Outside deep packet inspection (in which I include tech like Cisco NBAR) your main tools are probably going to be whois and port/protocol pair. Some commercial NetFlow collectors will do some of the legwork for you, for example by doing autonomous system lookup on incoming IP addresses, or providing the IANA protocol list.
The term "application" is a bit overloaded in this domain, by the way: often it's used to mean HTTP, SSH, POP3 and similar protocols in the OSI Application Layer, which are generally guessed from the port/protocol pair. For Facebook, Hotmail, etc, the whois protocol is probably your best bet. It's a bit better than reverse DNS, but the return formats aren't standardized among the Regional Internet Registries, so your parser is going to need to have some smarts. Get the IP addresses for a few of the major sites and use the command line whois utility with them to get a feel for the output before scripting anything.
Fortunately, most of the big ones are handled by ARIN. Look for "NetName" and "OrgName" in the results (and watch for the RIR names (RIPE, APNIC, etc) to indicate where that IP address isn't handled by ARIN). For example, I see www.stackoverflow.com as 198.252.206.16. whois 198.252.206.16 returns (among other things,
NetName: SE-NET01
OrgName: Stack Exchange, Inc.
You didn't specify whether you were shell scripting or programming; if the latter, the WHOIS protocol is standard and has a number of implementations in most languages.
Related
I can't seem to locate which whois servers this library uses, where in the code does it have the URL of the server(s) it connects too?
I'm not sure I understand how whois actually works, but I am assuming it connects to popular registrars API's correct?
Does this mean that if I try and run a whois for thousands of entries I will be rate limited and the response will begin to fail?
The WHOIS server definition list is available at
https://github.com/weppos/whois/blob/master/data/tld.json
It doesn't connect to popular registrars API, instead the WHOIS itself is a protocol and the library connects to the WHOIS interfaces provides by registries, whenever applicable.
Each registry has custom rate limiting rules. However, if you perform a large amount of requests, there is an high change you will start being rate limited.
I was surprised
A JMAP-supporting email host for the domain example.com SHOULD publish a SRV record _jmaps._tcp.example.com which gives a hostname and port (usually port 443).
The authentication URL is https://hostname/.well-known/jmap (following any redirects).
Other autodiscovery options using autoconfig.example.com or autodiscover.example.com may be added to a future version of JMAP to support clients which can’t use SRV lookup.
It doesn't match the original use cases for the well-known URI registry. Stuff like robots.txt, or dnt / dnt-policy.txt. And IPP / CUPS printing works fine without it, using a DNS TXT record to specify a URL. If you can look up SRV records, you can equally look up TXT. And the autodiscovery protocol involves XML which can obviously include a full URI.
E.g. what chance is there of this being accepted by the registry of well-known URIs? Or is it more likely to remain as something non-standard, like made-up URI schemes?
The idea almost certainly came from CalDav, which is already in the registry of well-known URIs. RFC 6532 defines DNS SRV and both DNS TXT and a well-known URI. So JMAP's proposal is perfectly well-founded.
It might sound strange that the URL is authenticated against, but this too is justified in CalDav. I think it helps shard users between multiple servers.
IMO it's not a good way to use SRV. On the other hand, JMAP is specifically considering clients that don't use SRV. One presumes the CalDav usage is for similar reasons.
It does seem bizarre that presumably web-centric implementations can't manage to discover full URIs (i.e. if they're using the autoconfig protocol).
I think you have to remember that these approaches start from user email addresses. The hallowed Web Architecture using HTTP URIs for everything... well, let's say it doesn't have much to say about mailto: URIs. DNS has got to be the "right" way to bridge the gap from domains to URIs. But in a web-centric world where you don't necessarily know how to resolve DNS, or only how to look up IPs to speak HTTP with? There are going to be some compromises.
I've read everything I could find on verifying e-mail addresses. The widely encountered solution is this, and it doesn't work (for one, actual nslookup output differs significantly from what the article shows, so I don't get an actual address to telnet to).
But then I thought: I don't need to verify the address. I just want to detect clearly bogus address (such an address that sending a message to it will yield "delivery failed" response). Is it possible to do in principle, and implement using C++ sockets or Java networking API in particular?
Depending on which operating system and tools you use, verifying the recipient's domain, and whether it is recorded in the DNS with a meaningful MX (mail exchange), you could use dig in place of nslookup. For foo#bar.com,
$ dig bar.com MX
Possibilities of detecting bogus eMail adresses are typically limited, though. Availability largely depends on how "generously" the MTA offers this information. Most don't, these days. The SMTP protocol includes some verbs you could then use, such as VRFY. On the other hand, spammers could do just that, hence … (That's one reason why a mail loop is run, in order to detect valid eMails fairly reliably; embedding, as I'm sure you know, a verification string to be sent back, or passed via URL to some web service.
SMTP, being a text protocol, would be used via some "transport layers" underlying higher level APIs like JavaMail. I'd look for programmability of these with the programming language used. Typically, there is some socket library, for sending and retrieving lines of text.
I have to automatically find web servers in certain ip range
It should not look like attack so I cannot use ping, curl, lynx. I cannot also use reverse dns.
The other approach is using search engine like google or bing. I can search by putting ip in search box later I can check if address contains ip then I know that is web server.
But google not returns useful data. For example for ip 212.77.100.101 (which is web server) does not return useful results, any of them on results web page does not contain 212.77.100.101 on address (https://www.google.pl/search?q=212.77.100.101).
Is there any other solution to that problem or is there any search engine to use?
This would really depend on a lot of factors. Your going to need some scripting heft to search through straight up google results for the information you want. Plus what do you mean by server? Just like a regular ole website server? You could probably utilize arin whois in some way to query ip addresses and any belonging to google, yahoo, etc etc you could identify as a LIKELY server IP address. If your looking to see if it's a server based on more technical information like OS, ports, etc etc there isn't much you'll likely find on google.
For instance an ARIN WHOIS of a google ip comes to this, http://whois.arin.net/rest/net/NET-74-125-0-0-1/pft . Using your preference of language you could probably make the query to the web page and have it return the DOM or w/e to a variable and then look for the element that would have identifying information such as a google designation under name or something to that effect.
The best way to really tell, AFAIK, is to check ports and other techniques, which you cannot do by the sound of it. I'm not aware of a database you can access either that would have that information by IP address either...
What do you mean by server? That would help narrow down what your looking to accomplish. Just any IP serving up some sort of data? Or anything that comes back to a linux box or something?
More detail! :D
I am using Objective-C and Java for this, but I think the question is language-neutral.
I have an iOS client that talks to a Java server over TCP/IP. Right now I need to tell at least one of the parties the IP address of the other. Is there a standard way that I can "discover" IP addresses (from one side or the other)?
Also, how would switching to UDP affect the answer?
There are many protocols for discovering other devices/servers on the network. One of the most commonly used in the iOS realm is "Bonjour". Look at Apple's sample apps.
Is there a standard way that I can "discover" IP addresses (from one side or the other)?
Yes, it's called "port sniffing" and will certainly get you in trouble since it's a common kind of attack.
You simply try all IP addresses in a range. Many firewall products will consider this an "intrusion" attempt and log you with the intrusion detection software.
We almost never "discover" addresses.
That's what "domain names" are for.
Why can't the server have a well known DNS name?