I am using ZAP docker image to perform API scans. I have disabled some passive scan rules in zap_started hook python script. Still, these are listed as part of the final report.
Script to disable passive scan rules:
pscan_id_list = [10003, 10015, 10017, 10020, 10021, 10028, 10029, 10031, 10032, 10039, 10040, 10041, 10041, 10043, 10052, 10050, 10056, 10061, 10108, 10109, 90001, 90026, 90029, 90030]
for pscan_id in pscan_id_list:
zap.pscan.set_scanner_alert_threshold(id=pscan_id, alertthreshold='OFF')
Logs confirm rules are disabled:
2021-06-17 06:44:10,269 Starting new HTTP connection (1): localhost:51008
2021-06-17 06:44:10,281 http://localhost:51008 "GET http://zap/JSON/pscan/action/setScannerAlertThreshold/?id=10003&alertThreshold=OFF&apikey= HTTP/1.1" 200 15
2021-06-17 06:44:10,288 Starting new HTTP connection (1): localhost:51008
2021-06-17 06:44:10,298 http://localhost:51008 "GET http://zap/JSON/pscan/action/setScannerAlertThreshold/?id=10015&alertThreshold=OFF&apikey= HTTP/1.1" 200 15
2021-06-17 06:44:10,304 Starting new HTTP connection (1): localhost:51008
2021-06-17 06:44:10,313 http://localhost:51008 "GET http://zap/JSON/pscan/action/setScannerAlertThreshold/?id=10017&alertThreshold=OFF&apikey= HTTP/1.1" 200 15
2021-06-17 06:44:10,320 Starting new HTTP connection (1): localhost:51008
2021-06-17 06:44:10,327 http://localhost:51008 "GET http://zap/JSON/pscan/action/setScannerAlertThreshold/?id=10020&alertThreshold=OFF&apikey= HTTP/1.1" 200 15
2021-06-17 06:44:10,333 Starting new HTTP connection (1): localhost:51008
2021-06-17 06:44:10,343 http://localhost:51008 "GET http://zap/JSON/pscan/action/setScannerAlertThreshold/?id=10021&alertThreshold=OFF&apikey= HTTP/1.1" 200 15
2021-06-17 06:44:10,350 Starting new HTTP connection (1): localhost:51008
2021-06-17 06:44:10,361 http://localhost:51008 "GET http://zap/JSON/pscan/action/setScannerAlertThreshold/?id=10028&alertThreshold=OFF&apikey= HTTP/1.1" 200 15
Passive scan rules are still listed in final report:
PASS: Vulnerable JS Library [10003]
PASS: Cookie No HttpOnly Flag [10010]
PASS: Cookie Without Secure Flag [10011]
PASS: Incomplete or No Cache-control and Pragma HTTP Header Set [10015]
PASS: Cross-Domain JavaScript Source File Inclusion [10017]
PASS: Content-Type Header Missing [10019]
PASS: X-Frame-Options Header [10020]
PASS: X-Content-Type-Options Header Missing [10021]
PASS: Information Disclosure - Sensitive Information in URL [10024]
PASS: Information Disclosure - Sensitive Information in HTTP Referrer Header [10025]
PASS: HTTP Parameter Override [10026]
PASS: Information Disclosure - Suspicious Comments [10027]
PASS: Open Redirect [10028]
PASS: Cookie Poisoning [10029]
PASS: User Controllable Charset [10030]
PASS: User Controllable HTML Element Attribute (Potential XSS) [10031]
Try using alertThreshold='OFF' (capital T) as per https://www.zaproxy.org/docs/api/#pscanactionsetscanneralertthreshold
Related
I'm sending GET requests via proxy that only supports HTTP (no HTTPS support). When I use that proxy (or any other http-only proxy) to request HTTPS it returns 403 when I use curl (and it seems to be the correct status). But if I just use CONNECT and GET I get 200.
Curl - 403 Forbidden:
curl -x proxyhost:proxyport -I https://example.com -vvv
* Trying PROXYHOST:8080...
* TCP_NODELAY set
* Connected to PROXYHOST (PROXYHOST) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to www.example.com:443
> CONNECT www.example.com:443 HTTP/1.1
> Host: www.example.com:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 403 Forbidden
HTTP/1.1 403 Forbidden
< Date: Fri, 15 Oct 2021 15:37:31 GMT
Date: Fri, 15 Oct 2021 15:37:31 GMT
< Server: Apache
Server: Apache
< Content-Length: 202
Content-Length: 202
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1
<
* Received HTTP code 403 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 403 from proxy after CONNECT
Pure HTTP - 200 OK:
CONNECT PROXYHOST:PROXYPORT HTTP/1.0
GET https://www.example.com:443 HTTP/1.0
HTTP/1.0 200 OK
Why do I get 200 with CONNECT + GET?
Additional info:
PROXYHOST is just a random host without any proxy setting or software. It just happens so that when you use it as proxy for http GET requests it returns correct status (200 if requested page exists, 404 if it does not, etc) and its own html instead of requested body. At the same time it always returns 403 if you try to use it as proxy for requesting https via anything but CONNECT + GET.
I also tried python requests and got this result (with verbose logs):
proxy = {'https': 'http://PROXYHOST:8080', 'http': 'http://PROXYHOST:8080'}
requests.get('https://example.com', proxies=proxy)
# logs
send: b'CONNECT example.com:443 HTTP/1.0\r\n'
send: b'\r\n'
# exception
File "/usr/lib/python3.8/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response / (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response')))
requests.get('http://example.com', proxies=proxy)
# logs
send: b'GET http://example.com/ HTTP/1.1\r\nHost: example.com\r\nUser-Agent: python-requests/2.25.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Fri, 15 Oct 2021 17:10:00 GMT
header: ...
<Response [200]>
I am facing following situations when configuring haproxy with node/express application. I am trying to
achieve following.
(https) (http)
browser ======> haproxy =====> node application
When loading the node application through the browser I am getting http 504 gateway time-out error.
Below is my haproxy configurtions.
haproxy configurations
Following are the haproxy logs.
vm-2 haproxy[21255]: 127.0.0.1:45948 [23/Dec/2019:10:57:51.411] https-in~ servers/server1 0/0/0/-1/100001 504 194 - - sH-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
vm-2 haproxy[21255]: 127.0.0.1:45948 [23/Dec/2019:10:57:51.411] https-in~ servers/server1 0/0/0/-1/100001 504 194 - - sH-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
vm-2 haproxy[21255]: 127.0.0.1:46122 [23/Dec/2019:10:59:31.435] https-in~ servers/server1 0/0/0/-1/100002 504 194 - - sH-- 1/1/0/0/0 0/0 "GET /favicon.ico HTTP/1.1"
Any help would be appreciated.
You're haproxy logs indicate that it's taking over 100 seconds (ie 100001/100002) for the request to complete and that it's being aborted (ie -1) before your backend server can send the full response.
If you're looking for a strictly haproxy solution (ie. you can't/won't tune your application) then you would need to play with haproxy timeout settings.
We faced the same problem, the client requests to the server were 504s sent by the HAProxy. We found out that the defaults configurations in the haproxy.cfg file had the timeout server property that defined the 504 response (setting it to a lower value, 1s in our case, would automatically result in a 504). Increasing that value is a way to have a longer connection between the proxy and the backend.
My server is load balanced to the backend server via gcp https lb and the backend server uses pm2 start -i options different ports and distribute them to those nodes using haproxy.
connection log
Log taken from server using code.
io.on('connection', (socket) => {
console.debug("connection!", socket.id);
}
Once every two times, the server connection fails.
Below is Log through haproxy -d.
fail
success
this log is success after failure
00000041:http-in.clireq[000a:ffffffff]: GET /socket.io/?EIO=3&transport=websocket&sid=rf3JvyUz2KCKV4KDAABS HTTP/1.1
00000041:http-in.clihdr[000a:ffffffff]: User-Agent: websocket-sharp/1.0
00000041:http-in.clihdr[000a:ffffffff]: Host: mydomain.com
00000041:http-in.clihdr[000a:ffffffff]: Upgrade: websocket
00000041:http-in.clihdr[000a:ffffffff]: Sec-WebSocket-Key: 9JCkV46YNC66nIUaZQZl9w==
00000041:http-in.clihdr[000a:ffffffff]: Sec-WebSocket-Version: 13
00000041:http-in.clihdr[000a:ffffffff]: X-Cloud-Trace-Context: 1210ae7f3bb6e56c817e7f5ad30e1d24/17748396103009389644
00000041:http-in.clihdr[000a:ffffffff]: Connection: Upgrade
00000041:http-in.clihdr[000a:ffffffff]: Via: 1.1 google
00000041:http-in.clihdr[000a:ffffffff]: X-Forwarded-For: source ip, dest ip
00000041:http-in.clihdr[000a:ffffffff]: X-Forwarded-Proto: https
00000041:websockets.srvrep[000a:000b]: HTTP/1.1 400 Bad Request
00000041:websockets.srvhdr[000a:000b]: Connection: close
00000041:websockets.srvhdr[000a:000b]: Content-type: text/html
00000041:websockets.srvhdr[000a:000b]: Content-Length: 18
00000041:websockets.srvcls[000a:adfd]
00000041:websockets.clicls[adfd:adfd]
00000041:websockets.closed[adfd:adfd]
00000042:http-in.accept(0007)=000a from [130.211.3.23:53189] ALPN=<none>
00000042:http-in.clireq[000a:ffffffff]: GET /socket.io/?EIO=3&transport=websocket&sid=xmZEqtHokEmBfs2QAABT HTTP/1.1
00000042:http-in.clihdr[000a:ffffffff]: User-Agent: websocket-sharp/1.0
00000042:http-in.clihdr[000a:ffffffff]: Host: mydomain.com
00000042:http-in.clihdr[000a:ffffffff]: Upgrade: websocket
00000042:http-in.clihdr[000a:ffffffff]: Sec-WebSocket-Key: 8PKkFxEv3c3KqIUW8dQbLA==
00000042:http-in.clihdr[000a:ffffffff]: Sec-WebSocket-Version: 13
00000042:http-in.clihdr[000a:ffffffff]: X-Cloud-Trace-Context: de0fa56d1689317bb9212879da8edfcb/11012650454108009103
00000042:http-in.clihdr[000a:ffffffff]: Connection: Upgrade
00000042:http-in.clihdr[000a:ffffffff]: Via: 1.1 google
00000042:http-in.clihdr[000a:ffffffff]: X-Forwarded-For: source ip, dest ip
00000042:http-in.clihdr[000a:ffffffff]: X-Forwarded-Proto: https
00000042:websockets.srvrep[000a:000b]: HTTP/1.1 101 Switching Protocols
00000042:websockets.srvhdr[000a:000b]: Upgrade: websocket
00000042:websockets.srvhdr[000a:000b]: Connection: Upgrade
00000042:websockets.srvhdr[000a:000b]: Sec-WebSocket-Accept: tW86o/zu95tHQayPP7IlGNXi96s=
There is no difference, but two results. i don't know why 400 bad request
haproxy.cfg
global
maxconn 4096
defaults
mode http
balance roundrobin
option redispatch
option forwardfor
timeout connect 5s
timeout queue 5s
timeout client 50s
timeout server 50s
frontend http-in
bind *:80
default_backend servers
# Any URL beginning with socket.io will be flagged as 'is_websocket'
acl is_websocket path_beg /socket.io
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
# The connection to use if 'is_websocket' is flagged
use_backend websockets if is_websocket
backend servers
server server1 10.168.0.50:80
# server server2 [Address]:[Port]
backend websockets
balance source
option http-server-close
option forceclose
cookie io prefix indirect nocache # using the `io` cookie set upon handshake
server ws-server1 10.168.0.50:5000 weight 1 maxconn 1024 cookie ws-server1 check
server ws-server2 10.168.0.50:5001 weight 1 maxconn 1024 cookie ws-server2 check
#server ws-server3 10.168.0.50:5002 weight 1 maxconn 1024 check
i use cookie SRVNAME insert options and server name SA, SB
but socket.io document read change cookie io prefix indirect nocache and server name ws-server1, ws-server2
My Tested:
client used long polling and websocket
testClient give option {transports: ['websocket']} always success.. but real client not use only websocket options
I don't know why it fails.
If only use ws-server1 the connection will always succeed. but ws-server2 use sometime connection failed. I guess sticky session problem. I try haproxy.cfg add cookie option but The problem is not solved.
How can I solve this problem?
I am testing REST API using rest-assured. When I send POST request it looks like rest-assured is making extra call. Here is the output from /var/log/httpd/access_log:
11.31.41.111 - - [26/Nov/2019:19:39:14 +0000] "POST /rest/v1/contact HTTP/1.1" 401 340 "-" "Apache-HttpClient/4.5.3 (Java/1.8.0_221)"
11.31.41.111 - - [26/Nov/2019:19:39:14 +0000] "POST /rest/v1/contact HTTP/1.1" 200 515 "-" "Apache-HttpClient/4.5.3 (Java/1.8.0_221)"
When I send exactly same request using Postman, access log shows only one request comes to the server:
11.31.41.111 - - [26/Nov/2019:19:40:44 +0000] "POST /rest/v1/contact/ HTTP/1.1" 200 529 "-" "PostmanRuntime/7.19.0"
Any idea why this is happening?
You should use Preemptive Authentication when building RestAssured request specification. Here's an example:
RestAssured.given().auth().preemptive().basic("username", "password")
.when().get("http://example.com")
.then().statusCode(200);
I'm looking for a solution with redirects to another domain if the response from HTTP server was 404.
acl not_found status 404
acl found_ceph status 200
use_backend minio_s3 rsprep ^HTTP/1.1\ 404\ (.*)$ HTTP/1.1\ 302\ Found\nLocation:\ / if not_found
use_backend ceph if found_ceph
But still not working, this rule goes to minio_s3 backend.
Thank you for you advice.
When the response from this backend has status 404, first add a Location header that will send the browser to example.com with the original URI intact, then set the status code to 302 so the browser executes a redirect.
backend my-backend
mode http
server my-server 203.0.113.113:80 check inter 60000 rise 1 fall 2
http-response set-header Location http://example.com%[capture.req.uri] if { status eq 404 }
http-response set-status 302 if { status eq 404 }
Test:
$ curl -v http://example.org/pics/funny/cat.jpg
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to example.org (127.0.0.1) port 80 (#0)
> GET /pics/funny/cat.jpg HTTP/1.1
> User-Agent: curl/7.35.0
> Host: example.org
> Accept: */*
The actual back-end returns 404, but we don't see it. Instead...
< HTTP/1.1 302 Moved Temporarily
< Last-Modified: Thu, 04 Aug 2016 16:59:51 GMT
< Content-Type: text/html
< Content-Length: 332
< Date: Sat, 07 Oct 2017 00:03:22 GMT
< Location: http://example.com/pics/funny/cat.jpg
The response body from the back-end's 404 error page will still be sent to the browser, but -- as it turns out -- the browser will not display it, so no harm done. This requires HAProxy 1.6 or later.
#Michael's answer is rather good, but isno't working for me for two reasons:
Mainly because the %[capture.req.uri] tag resolves to empty (HA Proxy 1.7.9 Docker image)
Also due to the fact that the original assumptions are incomplete, due to the fact that the frontend section is missing...
So I struggled for a while, as you find all kinds of answers on the Internet, between those guys who swear the 404 logic should be put in the frontend, vs those who choose the backend, and any possible kind of tags...
This is my answer, which works for me.
My use case is that if an image is not found on the backend behind HA Proxy, then an S3 bucket is checked.
The entry point is: https://myhostname:8080/path/to/image.jpeg
defaults
mode http
global
log 127.0.0.1:514 local0 debug
frontend come_on_over_here
bind :8080
# The following two lines are here to save values while we have access to them. They won't be available in the backend section.
http-request set-var(txn.path) path
http-request set-var(txn.query) query
http-request replace-value Host localhost:8080 dev.local:80
default_backend onprems_or_s3_be
backend onprems_or_s3_be
log global
acl path_photos var(txn.path) -m beg /path/prefix/i/want/to/strip/off
acl p_ext_jpeg var(txn.path) -m end .jpeg
acl is404 status eq 404
http-response set-header Location https://mybucket.s3.eu-west-3.amazonaws.com"%[var(txn.path),regsub(^/path_prefix_i_want_to_strip_off/,/)]?%[var(txn.query)]" if path_photos p_ext_jpeg is404
http-response set-status 301 if is404
server onprems_server dev.local:80 check