Is that possible to testing rest-api via OWASP ZAP ?
Url to attack worked just for GET requests.
For example, my api controllers work with only token. I have TokenController and this controller require POST data via JSON data include password and login. Can I someway testing this controller via OWASP ?
The short answer is yes. The long answer - it's complicated :)
Testing REST API is a bit harder than testing web API - you'll have to give Zap information about your API - which endpoints it has, parameters, etc. Can you share more about you're API? Does it have OpenAPI/Swagger document? Do you have existing tests? You can use either one of those for this task.
I gave a talk about how this can be achieved - you can find the recording here.
It possible to automate API testint with OWASP ZAP, but to perform the tests, I see two options: Offer some usage pattern, for example OpenAPI for ZAP consider extracting the information. And a second option would be to run an automated test to capture ZAP as passive scan information, and after that you can test the session information.
We recommend using the OpenAPI documentation.
The cucumber test would look like this:
Feature: Security
This feature is to test pokemon service security
Scenario: Validate passive and active scan
Given I import context from open API specification "/v2/api-docs"
And I remove alerts
| url |
| http://.*/v2/api-docs* |
And I import scan policy "javaclean" from file "javaclean.policy"
When I run active scan
And I generate security test HTML report with name "java-clean-security-report"
Then the number of risks per category should not be greater than
| low | medium | high | informational |
| 0 | 0 | 0 | 0 |
I am develop step for ZAP, view in the GitHub: https://github.com/osvaldjr/easy-cucumber/wiki/Security-steps
Example step for import OpenAPI docs:
#Given("^I import context from open API specification \"([^\"]*)\"$")
public void iImportContextFromOpenAPISpecification(String path)
throws ClientApiException, InterruptedException {
String url = getTargetUrl() + path;
log.info("Import Open API from url: " + url);
zapProxyApi.openapi.importUrl(url, null);
waitPassiveScanRunning();
verifyThatTheProxyHasCapturedHostInformation();
}
View others steps in: https://github.com/osvaldjr/easy-cucumber/blob/master/src/main/java/io/github/osvaldjr/stepdefinitions/steps/SecuritySteps.java
Related
I am developing secure payment APIs, and I want to avoid replay attacks with manipulation of the parameters in the url. For example in the following API call:
https://api.payment.com/wallet/transfer?from_account=123&to_account=456&amount=100
Once this API call is executed, someone with enough knowledge can execute the same API call by modifying any of the three parameters to his/her own advantage. I have thought of issuing a temporary token (transaction token) for each transaction. But this also doesn't sounds like enough.
Can anyone suggest the best way to mitigate replay attacks with parameters tampering?
THE API SERVER
I am developing secure payment APIs, and I want to avoid replay attacks with manipulation of the parameters in the url.
Before we dive into addressing your concerns it's important to first clarify a common misconception among developers, that relates to knowing the difference between who vs what is accessing the API server.
The difference between who and what is accessing the API server.
This is discussed in more detail in this article I wrote, where we can read:
The what is the thing making the request to the API server. Is it really a genuine instance of your mobile app, or is it a bot, an automated script or an attacker manually poking around your API server with a tool like Postman?
The who is the user of the mobile app that we can authenticate, authorize and identify in several ways, like using OpenID Connect or OAUTH2 flows.
If the quoted text is not enough for you to understand the differences, then please go ahead and read the entire section of the article, because without this being well understood you are prone to apply less effective security measures in your API server and clients.
SECURITY LAYERS AND PARAMETERS IN THE URL
For example in the following API call:
https://api.payment.com/wallet/transfer?from_account=123&to_account=456&amount=100
Security is all about applying as many layers of defence as possible in order to make the attack as harder and laborious as possible, think of it as the many layers in an onion you need to peel to arrive to the center one.
Attackers will always look for the most easy targets, the lower hanging fruit in the tree, because they don't want to resort to use a ladder when they can take the fruit from another tree with lower hanging fruit ;)
So one of the first layers of defense is to avoid using parameters in the url for sensitive calls, thus I would use a POST request with all the parameters in the body of the request, because this type of request cannot be done by simply copy paste the url into the browser or any other tool, thus they require more effort and knowledge to be performed, aka the fruit is more high in the tree for the attacker.
Another reason is that GET requests end up in the logs of the servers, thus can be accidentally exposed and easily replayed.
REPLAY ATTACKS FOR API CALLS
Once this API call is executed, someone with enough knowledge can execute the same API call by modifying any of the three parameters to his/her own advantage.
Yes they can, and they can learn how your API works even if you don't have public documentation for it, they just need to reveres engineer it with the help of any open source tool for mobile apps and web apps.
I have thought of issuing a temporary token (transaction token) for each transaction. But this also doesn't sounds like enough.
Yes it's not enough because this temporary token can be stolen via a MitM attack, just like a show in the article Steal That Api Key With a Man in the Middle Attack:
So, in this article you will learn how to setup and run a MitM attack to intercept https traffic in a mobile device under your control, so that you can steal the API key. Finally, you will see at a high level how MitM attacks can be mitigated.
So after performing the MitM attack to steal the token it's easy to use curl, Postman or any other similar tool to make the requests to the API server just like if you are the genuine who and what the API server expects.
MITIGATE REPLAY ATTACKS
Improving on Existing Security Defence
I have thought of issuing a temporary token (transaction token) for each transaction. But this also doesn't sounds like enough.
This approach is good but not enough as you alreay noticed, but you can improve it, if not have done it already, by making this temporary token usable only one time.
Another important defence measure is to not allow the requests with same amount and same recipients(from_account, to_account) be repeated in sequence, even if they have a new temporary token.
Also don't allow requests from the same source to be made to fast, specially if they are intended to come from human interactions.
This measures on their own will not totally solve the issue, but add some more layers into the onion.
Using HMAC for the One Time Token
In order to try to help the server to be confident about who and what is making the request you can use a Keyed-Hash Message Authentication Code (HMAC) which is designed to prevent hijacking and tampering, and as per Wikipedia:
In cryptography, an HMAC (sometimes expanded as either keyed-hash message authentication code or hash-based message authentication code) is a specific type of message authentication code (MAC) involving a cryptographic hash function and a secret cryptographic key. As with any MAC, it may be used to simultaneously verify both the data integrity and the authenticity of a message.
So you could have the client creating an HMAC token with the request url, user authentication token, your temporary token, and the time stamp that should be also present in a request header. The server would then grab the same data from the request and perform it's own calculation of the HMAC token, and only proceed with the request if it's own result matches the one for the HMAC token header in the request.
For a practical example of this in action you can read part 1 and part 2 of this blog series about API protection techniques in the context of a mobile app, that also features a web app impersonating the mobile app.
So you can see here how the mobile app calculates the HMAC, and here how the Api server calculates and validates it. But you can also see here how the web app fakes the HMAC token to make the API server think that the requests is indeed from who and what it expects to come from, the mobile app.
Mobile App Code::
/**
* Compute an API request HMAC using the given request URL and authorization request header value.
*
* #param context the application context
* #param url the request URL
* #param authHeaderValue the value of the authorization request header
* #return the request HMAC
*/
private fun calculateAPIRequestHMAC(url: URL, authHeaderValue: String): String {
val secret = HMAC_SECRET
var keySpec: SecretKeySpec
// Configure the request HMAC based on the demo stage
when (currentDemoStage) {
DemoStage.API_KEY_PROTECTION, DemoStage.APPROOV_APP_AUTH_PROTECTION -> {
throw IllegalStateException("calculateAPIRequestHMAC() not used in this demo stage")
}
DemoStage.HMAC_STATIC_SECRET_PROTECTION -> {
// Just use the static secret to initialise the key spec for this demo stage
keySpec = SecretKeySpec(Base64.decode(secret, Base64.DEFAULT), "HmacSHA256")
Log.i(TAG, "CALCULATE STATIC HMAC")
}
DemoStage.HMAC_DYNAMIC_SECRET_PROTECTION -> {
Log.i(TAG, "CALCULATE DYNAMIC HMAC")
// Obfuscate the static secret to produce a dynamic secret to initialise the key
// spec for this demo stage
val obfuscatedSecretData = Base64.decode(secret, Base64.DEFAULT)
val shipFastAPIKeyData = loadShipFastAPIKey().toByteArray(Charsets.UTF_8)
for (i in 0 until minOf(obfuscatedSecretData.size, shipFastAPIKeyData.size)) {
obfuscatedSecretData[i] = (obfuscatedSecretData[i].toInt() xor shipFastAPIKeyData[i].toInt()).toByte()
}
val obfuscatedSecret = Base64.encode(obfuscatedSecretData, Base64.DEFAULT)
keySpec = SecretKeySpec(Base64.decode(obfuscatedSecret, Base64.DEFAULT), "HmacSHA256")
}
}
Log.i(TAG, "protocol: ${url.protocol}")
Log.i(TAG, "host: ${url.host}")
Log.i(TAG, "path: ${url.path}")
Log.i(TAG, "Authentication: $authHeaderValue")
// Compute the request HMAC using the HMAC SHA-256 algorithm
val hmac = Mac.getInstance("HmacSHA256")
hmac.init(keySpec)
hmac.update(url.protocol.toByteArray(Charsets.UTF_8))
hmac.update(url.host.toByteArray(Charsets.UTF_8))
hmac.update(url.path.toByteArray(Charsets.UTF_8))
hmac.update(authHeaderValue.toByteArray(Charsets.UTF_8))
return hmac.doFinal().toHex()
}
API server code:
if (DEMO.CURRENT_STAGE == DEMO.STAGES.HMAC_STATIC_SECRET_PROTECTION) {
// Just use the static secret during HMAC verification for this demo stage
hmac = crypto.createHmac('sha256', base64_decoded_hmac_secret)
log.info('---> VALIDATING STATIC HMAC <---')
} else if (DEMO.CURRENT_STAGE == DEMO.STAGES.HMAC_DYNAMIC_SECRET_PROTECTION) {
log.info('---> VALIDATING DYNAMIC HMAC <---')
// Obfuscate the static secret to produce a dynamic secret to use during HMAC
// verification for this demo stage
let obfuscatedSecretData = base64_decoded_hmac_secret
let shipFastAPIKeyData = new Buffer(config.SHIPFAST_API_KEY)
for (let i = 0; i < Math.min(obfuscatedSecretData.length, shipFastAPIKeyData.length); i++) {
obfuscatedSecretData[i] ^= shipFastAPIKeyData[i]
}
let obfuscatedSecret = new Buffer(obfuscatedSecretData).toString('base64')
hmac = crypto.createHmac('sha256', Buffer.from(obfuscatedSecret, 'base64'))
}
let requestProtocol
if (config.SHIPFAST_SERVER_BEHIND_PROXY) {
requestProtocol = req.get(config.SHIPFAST_REQUEST_PROXY_PROTOCOL_HEADER)
} else {
requestProtocol = req.protocol
}
log.info("protocol: " + requestProtocol)
log.info("host: " + req.hostname)
log.info("originalUrl: " + req.originalUrl)
log.info("Authorization: " + req.get('Authorization'))
// Compute the request HMAC using the HMAC SHA-256 algorithm
hmac.update(requestProtocol)
hmac.update(req.hostname)
hmac.update(req.originalUrl)
hmac.update(req.get('Authorization'))
let ourShipFastHMAC = hmac.digest('hex')
// Check to see if our HMAC matches the one sent in the request header
// and send an error response if it doesn't
if (ourShipFastHMAC != requestShipFastHMAC) {
log.error("\tShipFast HMAC invalid: received " + requestShipFastHMAC
+ " but should be " + ourShipFastHMAC)
res.status(403).send()
return
}
log.success("\nValid HMAC.")
Web APP code:
function computeHMAC(url, idToken) {
if (currentDemoStage == DEMO_STAGE.HMAC_STATIC_SECRET_PROTECTION
|| currentDemoStage == DEMO_STAGE.HMAC_DYNAMIC_SECRET_PROTECTION) {
var hmacSecret
if (currentDemoStage == DEMO_STAGE.HMAC_STATIC_SECRET_PROTECTION) {
// Just use the static secret in the HMAC for this demo stage
hmacSecret = HMAC_SECRET
}
else if (currentDemoStage == DEMO_STAGE.HMAC_DYNAMIC_SECRET_PROTECTION) {
// Obfuscate the static secret to produce a dynamic secret to
// use in the HMAC for this demo stage
var staticSecret = HMAC_SECRET
var dynamicSecret = CryptoJS.enc.Base64.parse(staticSecret)
var shipFastAPIKey = CryptoJS.enc.Utf8.parse($("#shipfast-api-key-input").val())
for (var i = 0; i < Math.min(dynamicSecret.words.length, shipFastAPIKey.words.length); i++) {
dynamicSecret.words[i] ^= shipFastAPIKey.words[i]
}
dynamicSecret = CryptoJS.enc.Base64.stringify(dynamicSecret)
hmacSecret = dynamicSecret
}
if (hmacSecret) {
var parser = document.createElement('a')
parser.href = url
var msg = parser.protocol.substring(0, parser.protocol.length - 1)
+ parser.hostname + parser.pathname + idToken
var hmac = CryptoJS.HmacSHA256(msg, CryptoJS.enc.Base64.parse(hmacSecret)).toString(CryptoJS.enc.Hex)
return hmac
}
}
return null
}
NOTE: While the above code is not using the exact same parameters that you would use in your case, it is a good starting pointing for you to understand the basics of it.
As you can see the way the HMAC token is calculated across mobile app, Api server and the Web app are identical in the semantics of the logic, thus resulting in the same HMAC token, and this way the Web app is able to defeat the Api server defense to only accept valid request from the mobile app.
The bottom line here is that anything you place in the client code can be reverse engineered in order to replicate it in another client. So should I use HMAC tokens in my use case?
Yes, because it's one more layer in the onion or a fruit more high in the tree.
Can I do better?
Yes you can do, just keep reading...
Enhance and Strength the Security
Can anyone suggest the best way to mitigate replay attacks with parameters tampering?
Going with the layered defence approach once more, you should look to other layered approaches that will allow your API server to be more confident about who and waht is accessing it.
So if the clients of you API server are only mobile apps, then please read this answer for the question How to secure an API REST for mobile app?.
In the case you need to secure an API that serves both a mobile and web app, then see this another answer for the question Unauthorized API Calls - Secure and allow only registered Frontend app.
GOING THE EXTRA MILE
Now I would like to recommend you the excellent work of the OWASP foundation:
The Web Security Testing Guide:
The OWASP Web Security Testing Guide includes a "best practice" penetration testing framework which users can implement in their own organizations and a "low level" penetration testing guide that describes techniques for testing most common web application and web service security issues.
The Mobile Security Testing Guide:
The Mobile Security Testing Guide (MSTG) is a comprehensive manual for mobile app security development, testing and reverse engineering.
I am using the Twitter API from Matlab, specifically by means of the twitter class from the Datafeed Toolbox.
I have essentially followed the example code from the official documentation. I created a Twitter app in my Twitter developer page, and obtained its API keys and access tokens. With those I can use the Twitter Standard search API from Matlab:
c = twitter(consumerkey,consumersecret,accesstoken,accesstokensecret);
% The variables 'consumerkey' etc are defined as character vectors
s = search(c,tweetquery,'count',100); % this works
Now I want to use the Premium search API. This has two endpoints for accessing Tweets:
30-day endpoint: provides Tweets from the previous 30 days.
Full-archive endpoint: provides complete and instant access to Tweets dating all the way back to the first Tweet in March 2006.
In addition, the Premium API has two tiers of access:
Free Sandbox access that enables initial testing and development.
Paid Premium access that provides increased access.
The link above specifies the restrictions associated to sandbox as compared with paid access.
I am trying to use the full-archive endpoint with sandbox access. For that I had to create a developer environment on Twitter, which I named dev.
The search method in Matlab's twitter class (which worked for the Standard access, as described above) doesn't seem to work with the Premium access. But I noticed that search actually calls getdata, and the latter does work for Premium access as follows. First, the Premium access URL needs to be defined:
c.URL = 'https://api.twitter.com/1.1/tweets/search/fullarchive/dev.json';
and then the following syntax works:
s = getdata(c,c.URL,'query','Jimi Hendrix'); % this works
I have also been able to add operators within the query string, for example to specify a range of geographical positions or to restrict the search to tweets that contain images:
s = getdata(c,c.URL,'query','place:"Palo Alto"'); % this works
s = getdata(c,c.URL,'query','Robert Smith bounding_box:[-0.2 51.4 0.1 51.6]') % this works
However—and this is my question—, I haven't been able to use additional request parameters defined in the Twitter API to refine the search, such as fromDate, toDate or maxResults:
s = getdata(c,c.URL,'query','John Frusciante', 'fromDate', '201708130000') % doesn't work
s = getdata(c,c.URL,'query','Rob Scallon', ...
'fromDate', '201708130000', 'toDate', '201708150000') % doesn't work
s = getdata(c,c.URL,'query','Michael Lemmo', 'maxResults', '20') % doesn't work
All of the above return an HTTP/1.1 422 Unprocessable Entity error.
Is my syntax not correct? Maybe the fromDate etc parameters have to be part of the query string? Or maybe the sandbox tier of the Premium search doesn't support those parameters?
For context, I don't really know what all those terms like endpoint, tier, developer environment and token mean, but still I'd like to make this work.
Going by the description at https://developer.twitter.com/en/docs/tweets/search/api-reference/premium-search#DataParameters, what you call 'addition request parameters' are defined for requests of type POST /search/:product. These are HTTP POST requests, can you try using postdata (https://in.mathworks.com/help/datafeed/twitter.postdata.html) instead of getdata. Their usage is almost identical.
In Analytics > User conversations, I would like to filter out the messages which my Watson Assistant was unable to answer in its conversations, i.e., when it returns the anything_else reply.
But since anything_else is not an intent nor an entity, I can't filter messages by it in the UI. Is it possible to therefore filter messages by the assistant's reply, like you can with user statements? That way I can filter out each time the assistant answered "I don't understand".
IBM Watson Assistant offers the log in the Analytics section of it UI and, IMHO more powerful, via REST API with filter expressions. With that API you can either download all logs and filter "offline" or apply some expressions. I recommend reading the documentation about how to query the "other" fields which includes the user responses.
I have written Python tools for Watson Assistant using the V1 and V2 APIs. The V1 tool is able to download or query the logs. An option is to download all available logs, then use jq to filter them.
python wctool.py -logs -config config.json.event -id 28ac4bcb-2aae-41aa-87ac-2e6074e1356a > logs201904.out
This gives you the information about which nodes were visited:
cat logs201904.out | jq -r '.logs | .[]
| .response.output.nodes_visited '
Filter for log records that visited "Anything else":
cat logs201904.out | jq -r '.logs | .[] | .response | select(
.output.nodes_visited[] | contains("Anything else")) '
anything_else means either an intent was not found, or the confidence was < 0.2.
Filter looking for intents = [], or confidence of first intent is below 20%
Using bing.com, I can do a search like this (click here for link):
history site:berkeley.edu/about/
When I try the same using the API, I get very different results. As far as I can tell, the search results returns webpages that are not hosted in berkeley.edu (see bottom).
this is the HTTP GET request being made to Azure
https://api.cognitive.microsoft.com/bing/v5.0/search?q=history+site:berkeley.edu/about/&count=10&offset=0
This is my HTTP GET code
$.ajax({
url: "https://api.cognitive.microsoft.com/bing/v5.0/search"
, data: { "q":encodeURI("history+site:berkeley.edu/about/"), "count":"10", "offset":"0" }
, beforeSend: function(xhrObj){
xhrObj.setRequestHeader("Ocp-Apim-Subscription-Key","supply-your-key-here");
}
, type: "GET",
})
Any ideas what I could be doing wrong? Thanks
edit1: Seems my "problem" is related to the way AJAX is making the HTTP request. If I supply my key by using a Firefox header plugin and type this (https://api.cognitive.microsoft.com/bing/v5.0/search?q=history+site:berkeley.edu/about/&count=10&offset=0) on my browser URL box, I get the correct response.
search results using API
Environmental Design Library | UC Berkeley Library: A branch of the UC Berkeley Library system, the Environmental Design Library supports the research and teaching of the College of Environmental Design.
Proceedings Template - WORD - ideals.illinois.edu: "(c) ACM, 2007. This is the authors’ version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution.
Trends in metadata practices: A longitudinal study of ...: Trends in metadata practices: A longitudinal study of collection federation. ... A Longitudinal Study of Collection Federation Carole Palmer Oksana ...
http://aerospaceutility.tripod.com/ · GitHub: Clone via HTTPS Clone with Git or checkout with SVN using the repository's web address.
HS RWC Colorado Sample Instructional Units - LiveBinder: Loading Livebinder HS RWC Colorado Sample Instructional Units HS Read Write Communicate Sample Instructional Units provided by the Colorado Department of Education.
Arroyo High School: News Archive: News Archive SIA Awards "As the school year comes to a close, the Students in Action club would like to honor three students for their lasting impact on our ...
English 12 (exp) | Utah Electronic High School: Please be mindful of the fact that this course is not a credit "quick fix." It is a rigorous, college-preparatory class that is both time and labor intensive.
Working SMARTer, not Harder: SOCIAL STUDIES ONLINE ...: SOCIAL STUDIES ONLINE RESOURCES AND LINKS COMPILATION beta List of Social Studies online resources and links to professional development opportunities ...
The Big List -- 20121008 - Grolier: The Big List -- 20121008: 1: EA: http://www.stanford.edu/group/bipolar.clinic/ Stanford Bipolar Disorders Clinic: 2: EA: http://www.mhsource.com/bipolar/
Spreadsheet of Conference Attendees - studylib.net: ÐÏ à¡± á > þÿ ] þÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ...
You can use Bing custom search alternately to make sure you get results only from the domain/webpages you want. Here is the call: https://api.cognitive.microsoft.com/bingcustomsearch/v7.0/search. You will need a different accesskey though, which you can get from customsearch.ai.
I'm getting correct results on both v5.0 and v7.0.
There seems nothing wrong with your query.
https://api.cognitive.microsoft.com/bing/v5.0/search?q=history+site:berkeley.edu/about/&count=10&offset=0
Perhaps you are caching results somewhere in your browser?
Update: Since IE does it, but Firefox doesn't. Have you disabled cache on IE?
I am trying to test an app with RESTful services via Postman (awaiting the front-end to be implemented).
The GET/POST and DELETE requests work as expected, but when it comes to PUT and PATCH I'm completely stuck.
I have a simple form with several inputs and files (pdf & image). With the POST request I simply use the "form-data" body to add all needed parameters. But when I try to test the PUT one, "form-data" detects nothing - with or without a "Content-Type" = "multipart/form-data" set in the header.
The PUT only works with the "x-www-form-urlencoded" option for the body, but then I cannot or don't know how to add both files, as I don't have the ability to choose from "Text/File" anymore via Postman's dropdown. Again, adding Content-Type doesn't help a bit.
I tried simulating POST with a "_method"="PUT" both as a URL parameter and as a form input, but it just creates new item instead of updating existing one (with correct route applied).
# Updated with the routes (standard Laravel routes for RESTful services)
GET | api/v1/Collection | api.v1.collection.index
POST | api/v1/collection | api.v1.collection.store
GET | api/v1/collection/create | api.v1.collection.create
GET | api/v1/collection/{item} | api.v1.collection.show
PUT | api/v1/collection/{item} | api.v1.collection.update
DELETE | api/v1/collection/{item} | api.v1.collection.destroy
GET | api/v1/collection/{item}/edit | api.v1.collection.edit
I realize I have to use a hidden input with _method/PUT as key-value, but I have no idea where to add it in Postman. I can only see two options: text input or file.
What am I missing?
I can provide screenshots upon request. Thanks.