check dns-blacklist of ip by using namp - nmap

I'm talking about this script for nmap
http://nmap.org/nsedoc/scripts/dns-blacklist.html
User Summary
Checks target IP addresses against multiple DNS anti-spam and open proxy blacklists and returns a list of services for which an IP has been flagged. Checks may be limited by service category (eg: SPAM, PROXY) or to a specific service name.
Does it possible to use Timing and Performance of nmap like make it in parallel and set time out? any example please?
Thank you

The dns-blacklist script uses the dnsbl library to perform queries. That library uses Lua coroutines to issue many requests in parallel. The number of coroutines (including scripts) that can be running at any given time is set by the CONCURRENCY_LIMIT variable in nse_main.lua, and is not settable by the user. A more complete description of NSE parallelism can be found in the online documentation.
For timeouts, the script itself does not accept a timeout script-argument. Fortunately, though, the dnsbl library offloads the DNS query execution to the dns library, which has a function called get_default_timeout:
get_default_timeout = function()
local timeout = {[0] = 10000, 7000, 5000, 4000, 4000, 4000}
return timeout[nmap.timing_level()] or 4000
end
This shows that the dns library will set the timeout for DNS queries to 4000 ms (4 seconds) for -T3 (the default) through -T5, but will be more cautions at lower timing levels.

Related

Limiting the number of times an endpoint of Kubernetes pod can be accessed?

I have a machine learning model inside a docker image. I pushed the docker image to google container registry and then deploy it inside a Kubernetes pod. There is a fastapi application that runs on Port 8000 and this Fastapi endpoint is public
(call it mymodel:8000).
The structure of fastapi is :
app.get("/homepage")
asynd def get_homepage()
app.get("/model):
aysnc def get_modelpage()
app.post("/model"):
async def get_results(query: Form(...))
User can put query and submit them and get results from the machine learning model running inside the docker. I want to limit the number of times a query can be made by all the users combined. So if the query limit is 100, all the users combined can make only 100 queries in total.
I thought of a way to do this:
Store a database that stores the number of times GET and POST method has been called. As soon as the total number of times POST has been called crosses the limit, stop accepting any more queries.
Is there an alternative way of doing this using Kubernetes limits? Such as I can define a limit_api_calls such that the total number of times mymodel:8000 is accessed is at max equal to limit_api_calls.
I looked at the documentation and I could only find setting limits for CPUs, Memory and rateLimits.
There are several approaches that could satisfy your needs.
Custom implementation: As you mentioned, keep in a persistence layer the number of API calls received and deny requests after it has been reached.
Use a service mesh: Istio (for instance) will let you limit the number of requests received and act as a circuit breaker.
Use an external Api Manager: Apigee will also let you limit and even charge your users, however if it is only for internal use (not pay per use) I definitely won't recommend it.
The tricky part is what you want to happen after the limit has been reached, if it is just a pod you may exit the application to finish and clear it.
Otherwise, if you have a deployment with its replica set and several resources associated with it (like configmaps), you probably want to use some kind of asynchronous alert or polling check to clean up everything related to your deployment. You may want to have a deep look at orchestrators like Airflow (Composer) and use several tools such as Helm for keeping deployments easy.

Kubernetes Service: IPVS load balancing algorithm

As discovered here, there is a new kind of kube service that are IPVS and have many algorithms for load balancing.
The only problem is I didn't find where those algorithms are specified.
My understanding:
rr: round-robin
-> call backend pod one after another in a loop
lc: least connection
-> group all pod with the lowest number of connection, and send message to it. Which kind of connection? only the ones from this service ?
dh: destination hashing
-> ?something based on url?
sh: source hashing
-> ?something based on url?
sed: shortest expected delay
-> either the backend with less ping or some logic on the time a backend took to respond in the past
nq: never queue
-> same as least connection? but refusing messages at some points ?
If anyone has the documentation link (not provided in the official page and still saying IPVS is beta whereas it is stable sync 1.11) or the real algorithm behind all of them, please help.
I tried: Google search with the terms + lookup in the official documentation.
They are defined in the code
https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/apis/config/types.go#L193
rr round robin : distributes jobs equally amongst the available real servers
lc least connection : assigns more jobs to real servers with fewer active jobs
sh source hashing : assigns jobs to servers through looking up a statically assigned hash table by their source IP addresses
dh destination hashing : assigns jobs to servers through looking up a statically assigned hash table by their destination IP addresses
sed shortest expected delay : assigns an incoming job to the server with the shortest expected delay. The expected delay that the job will experience is (Ci + 1) / Ui if sent to the ith server, in which Ci is the number of jobs on the ith server and Ui is the fixed service rate (weight) of the ith server.
nq never queue : assigns an incoming job to an idle server if there is, instead of waiting for a fast one; if all the servers are busy, it adopts the ShortestExpectedDelay policy to assign the job.
All those come from IPVS official documentation : http://www.linuxvirtualserver.org/docs/scheduling.html
Regards

How to enforce single threaded requests?

I've started rate limiting my API using HAProxy, but my biggest problem is not so much the rate of requests, but when multi-threaded requests overlap.
Even within my legal per-second limits, big problems are occurring when clients don't wait for a response before issuing another request.
Is it possible (say, per IP address) to queue requests and pass them one at at time to the back end for sequential processing?
Here is a possible solution to enforce one connection at a time per src IP.
You need to put the following HAProxy conf in the corresponding frontend:
frontend fe_main
mode http
stick-table type ip size 1m store conn_cur
tcp-request connection track-sc0 src
tcp-request connection reject if { src_conn_cur gt 1 }
This will create a stick table that stores concurrent connection counts per source IP. Then rejects new connections if there is already one established from the same src IP.
Browsers imitating multiple connections to your API or clients behind a NAT will not be able to efficiently use you API.

vxworks 6.3 active sockets maxs out at 255?

I have a LPD server running on vxworks 6.3. The client application (over which I have no control) is sending me a LPQ query every tenth of a second. After 235 requests, the client receives a RST when trying to connect. After a time device will again accept some queries (about 300), until it again starts sending out RST.
I have confirmed that it is the TCP stack that is causing the RST. There are some things that I have noticed.
1) I can somewhat change the number of sockets that will accepted if I change the number of other applications that are running. For example, I freed up 4 sockets thereby changing the number accepted from 235 to 239.
2) If I send requests to lpr (port 515) and another port (say, port 80), the total number of connections that are accepted before the RST start happening stays constant at 235.
3) There are lots of sockets sitting TIME_WAIT.
4) I have a mock version of the client. If I slow the client down to one request every quarter second, the server doesn't reject the connections.
5) If I slow down the server's responses, I don't have any connections rejected.
So my theory is that there is some share resource (my top guess is total number of socket handles) that VxWorks can have consumed at a given time. I'm also guessing that this number tops out at 255.
Does anyone know how I can get VxWorks to accept more connections, and leave them in TIME_WAIT when closed? I have looked through the kernel configuration and changed all the values that looked remotely likely, but I have not been able change the number.
We know that we could set SO_LINGER but this not an acceptable solution. However, this does prevent the client connections from getting rejected. We have also tried changed the timeout value for SO_LINGER. This does not appear to be supported in VxWorks. It's either on or off.
Thanks!
Gail
To me it sounds like you are making a new connection for every LPQ query, and after the query is done you aren't closing the connection. In my opinion the correct thing to do is to accept one TCP connection and then use that to get all of the LPQ queries, however that may require mods to the client application. To avoid mods to the client, you should just close the TCP connection after each LPQ query.
Furthermore you can set the max number of FDs open in vxworks by adjusting the #define NUM_FILES config.h (or configall.h or one of those files), but that will just postpone an error if you have a FD leak, which you probably do.

Is there a maintained alternative for libnids?

As libnids seems to be two years old and there are no current updates, do some one know any alternative solution for libnids or better library than it, as it seems to drop packets in higher speeds more than 1G/per sec
And more over it has no support for 64 bit ip addresses.
An alternative to libnids is Bro. It comes with a robust TCP reassembler which has been thoroughly tested and used by the network security monitoring community over the years. It ships with a bunch of protocol analyzers for common protocols, such as HTTP, DNS, FTP, SMTP, and SSL.
Bro is "the Python of network processing:" it has its own domain-specific scripting language with first-class types and functions for IP addresses (both v4 and v6), subnets, ports. The programming style has an asynchronous event-based flavor: users write callback functions for events that reflect network activity. The analysis operates at connection granularity. Here is an example:
event connection_established(c: connection)
{
if ( c$id$orig_h == 1.2.3.4 && c$id$resp_p == 31337/udp )
// IP 1.2.3.4 successfully connected to remote host at port 31337.
}
Moreover, Bro supports a cluster mode that allows for line-rate monitoring of 10 Gbps links. Because most analyses do not require sharing of inter-connection state, Bro scales very well across cores (using PF_RING) as well as multiple nodes. There exist Bro installations with >= 140 nodes. A typical deployment looks as follows:
(source: bro.org)
Due to the high scalability, there is typically no more need to grapple with low-level details and fine-tune C implementations. Or put differently, with Bro you spend your time working on the analysis and not the implementation.