We are trying to connect to Periodic's API using Swift 5. In order to do that, part of the process is encoding a concatenated string in HMAC SHA-256. In our code, we accomplish this by the following code:
let hmac = premac.digest(.sha256, key: "API KEY")
Which is using the SwiftCrypto library to accomplish this. According to Periodic's CTO, we are getting close but the issue is that this function's output is giving us a HEX version when we really need the bytestring level.
We are unsure how to proceed.
Related
I will start with a disclaimer that I am out of my depth here. A colleague was showing me a decryption routine he wrote with pycryptodomex. He had an encrypted file, a key, and a nonce (extracted from the file). He was able to decrypt the file contents in a very straight forward way.
c = Crypto.Cipher.AES.new(key, AES.MODE_GCM, nonce)
c.decrypt(encrypted_data)
You can see a similar implementation in the pycryptodome test for GCM:
cipher = AES.new(self.key_128, AES.MODE_GCM, nonce=self.nonce_96)
pt = get_tag_random("plaintext", 16 * 100)
ct = cipher.encrypt(pt)
cipher = AES.new(self.key_128, AES.MODE_GCM, nonce=self.nonce_96)
pt2 = cipher.decrypt(ct)
Unfortunately, pycryptdomex is an additional dependency that I would need to carry around and I am looking to avoid this. I have a base installation of Anaconda, which brings with it the pyCrypto and pyCA/cryptography packages. It appears that pycryptodomex is a fork of pyCrytpo, which didn't have a stable GCM implementation to begin with. When I look at the implementation for PyCA/cryptography, it looks straight forward:
cipher = Cipher(algorithms.AES(key), modes.GCM(nonce), backend=default_backend())
d = cipher.decryptor()
But when we want to decrypt content we have to call finalize_with_tag and produce an authentication tag:
d.update(encrypted_data) + d.finalize_with_tag(tag)
Unfortunately, I don't have an authentication tag nor do I know where to find it. I can't set the value to None as there is a minimum length requirement. I'm also not sure why I need to produce an authentication tag in the first place for AES GCM decryption with PyCA/Cryptography but I do not need to produce a tag when decrypting with the pycryptodomex. I'm ultimately looking for clarity on the following:
Is it possible to implement AES/GCM decryption with the Anaconda PyCA/cryptography package if I only have access to the key, nonce, and encrypted data?
Why do I need to provide an authentication tag for decryption with one implementation and not the other?
Is pycryptodomex doing something under the hood to determine the tag?
GCM without authentication tag is equivalent to CTR mode. (except the + 1 difference in starting counter value)
Calling decrypt does not verify the tag (as far as I know). You can test this yourself by altering the ciphertext just one byte. It will decrypt just fine (to a plaintext that is off by one byte). Use decrypt_and_verify (see test_invalid_mac test).
See 2.
Apologies as I can't reply to comments. Is it possible to derive the tag from the decrypted data after decryption? This PR associated with PyCA/cryptography seems to imply the exact scenario considered here.
According to the GCM spec (section 7.2: “Algorithm for the
Authenticated Decryption Function”), the tag itself is not needed
until the ciphertext has been decrypted.
Does calling d.update(encrypted_data) decrypt data successfully and d.finalize() is only needed to verify the integrity of the data?
I wrote a gateway application using Spring cloud Greenwich binaries. I'm seeing issues when special characters are present in URL. The request fails with below exception in Spring gateway when request URI contains special characters.
localhost:8080/myresource/WG_splchar_%26%5E%26%25%5E%26%23%25%24%5E%26%25%26*%25%2B)!%24%23%24%25%26%5E_new
When I hit above url, Spring fails with below exception. I'm not able to figure out why it's an invalid sequence and how things like these can be handled.
java.lang.IllegalArgumentException: Invalid encoded sequence "%^&#%$^&%&*%+)!$#$%&^_new"
at org.springframework.util.StringUtils.uriDecode(StringUtils.java:741) ~[spring-core-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.http.server.DefaultPathContainer.parsePathSegment(DefaultPathContainer.java:126) ~[spring-web-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.http.server.DefaultPathContainer.createFromUrlPath(DefaultPathContainer.java:111) ~[spring-web-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.http.server.PathContainer.parsePath(PathContainer.java:76) ~[spring-web-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.cloud.gateway.handler.predicate.PathRoutePredicateFactory.lambda$apply$2(PathRoutePredicateFactory.java:79) ~[spring-cloud-gateway-core-2.1.0.RC3.jar:2.1.0.RC3]
at org.springframework.cloud.gateway.support.ServerWebExchangeUtils.lambda$toAsyncPredicate$1(ServerWebExchangeUtils.java:128) ~[spring-cloud-gateway-core-2.1.0.RC3.jar:2.1.0.RC3]
at org.springframework.cloud.gateway.handler.AsyncPredicate.lambda$and$1(AsyncPredicate.java:35) ~[spring-cloud-gateway-core-2.1.0.RC3.jar:2.1.0.RC3]
at org.springframework.cloud.gateway.handler.RoutePredicateHandlerMapping.lambda$null$2(RoutePredicateHandlerMapping.java:112) ~[spring-cloud-gateway-core-2.1.0.RC3.jar:2.1.0.RC3]
at reactor.core.publisher.MonoFilterWhen$MonoFilterWhenMain.onNext(MonoFilterWhen.java:116) [reactor-core-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2070) [reactor-core-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at reactor.core.publisher.MonoFilterWhen$MonoFilterWhenMain.onSubscribe(MonoFilterWhen.java:103) [reactor-core-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) [reactor-core-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at reactor.core.publisher.MonoFilterWhen.subscribe(MonoFilterWhen.java:56) [reactor-core-3.2.5.RELEASE.jar:3.2.5.RELEASE]
I answered the other question already and don't feel like retyping. The spirit of the answer is the exact same.
Write a unit test exercising this method off of the Spring cloud utils. This is what's breaking. You can try passing in more or less of the string you're concerned about to find where the breakage is. Use a binary search to figure out what's broken. Make sure you don't split the string in the middle of an encoded character or else you'll give yourself a false positive. When it says you have an invalid sequence I would expect you have something like %99 where 99 is does not map to any valid character (I'm just making one up)
https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/util/StringUtils.html#uriDecode-java.lang.String-java.nio.charset.Charset-
As an aside
Where is this encoded string coming from? Did someone at your company create their own solution to encode this string to begin with? Are you accepting user data? It's VERY POSSIBLE that whomever is responsible for producing this string encoded it incorrectly by homerolling their own encoder.
ALTERNATIVELY
spring.cloud.gateway.routes[7].predicates[0]=Path=/test/{testId}/test1/test_%26%5E%26%25%5E%26%25%26*%25%2B)!
When I look at this I see a path that is already encoded. For example, you've taken your ampersand & character and replaced it with %26
Have you tried inputting a path that is NOT already encoded?
For example
spring.cloud.gateway.routes[7].predicates[0]=Path=/test/{testId}/test1/test_&^&%^ < I only partially decoded it by hand using this chart. https://www.w3schools.com/tags/ref_urlencode.asp
I have a string:
RP581147238IN which gets encoded as A3294Fc0Mb0V1Tb4aBK8rw==
and another string:
RP581147239IN which gets encoded as A3294Fc0Mb1BPqxRDrRXjQ==
But after spending a day, I still cannot figure out what is the encoding process.
The encoded string looks like its base64 encoded.
But when I decode it, it looks like:
base64.decodestring("A3294Fc0Mb0V1Tb4aBK8rw==")
\x03}\xbd\xe0W41\xbdA>\xacQ\x0e\xb4W\x8d
The base 64 decoded string now is looking like a zlib compressed string
I've tried to further use zlib decompression methods but none of them worked.
import zlib, base64
rt = 'A3294Fc0Mb1BPqxRDrRXjQ=='
for i in range(-50, 50):
try:
print(zlib.decompress(base64.decodestring(rt), i));
print("{} worked".format(i))
break
except:
pass
But that did not produce any results either.
Can anybody figure out what is the encoding process used here. #Nirlzr, I am looking at you for the heroic answer you provided in Reverse Engineer HTTP request.
The strings seem to be Base64 encoded and the underlying decoded data seems to be encrypted. Encrypted data can not be directly represented as a string and it is common the Base64 encode encrypted data when a string is required.
If this is the case you need to decrypt the decoded data and ignorer to accomplish that you would need the encryption key.
Note: In general it is not productive to compress such short items.
If you put your data strings side by side:
RP581147238IN A3294Fc0Mb0V1Tb4aBK8rw==
RP581147239IN A3294Fc0Mb1BPqxRDrRXjQ==
You can see that source strings have only character difference, but encoded version contains 12 different characters:
----------8-- ----------0V1Tb4aBK8rw--
----------9-- ----------1BPqxRDrRXjQ--
Encoded data has similar paddings at the end as base64, but definitely it is not base64. Probably crypted with some SHA-like algorithm. With the data you provided, I would say that it is not possible to reverse-engineer the encoding process. Probably more data would not help much either.
I am developing ios app which is getting data from Google endpoint ,the data is base 64 encoded on the server to a custom java object, which is then returned by the endpoint method.
On the iOS side I am able to receive the data and print the data using the generated client code.
I am facing a problem and I am unable to decode the data back in to the GTL**** endpoint auto generated class.
The decoded data shows up with some hex numbers:
My Code:
let respo2 = GTLDecodeBase64(responce) as? GTLEndpointStatusCollection
I also tried decoding using the swift classes:
let respo = NSData(base64EncodedString: responce, options: NSDataBase64DecodingOptions(rawValue: 0))
The input is base64 encoded : rO0ABXNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ......
The desired output should have been readable data,
but instead im getting:
<aced0005 73720013 6a617661 2e757469 6c2e4172 7261794c.....
I even tried encoding, decoding the base64 decoded data with NSUTF8
but no use.
What am I doing wrong? Is it possible for data encoded on Server in Java (with custom Java objects) to be decoded back ? (I understand Google endpoint does the serialization/deserialization in between)
Thanks in advance.
You should use JSON for serialization rather than manually converting the object to a bytestring and base64 encoding it. If you are using the Endpoints libraries this is automatically done for you, simply by returning the object in your method. See the docs here for an example and the rest of the Endpoints docs for more details. To consume the API you can use the generated iOS libraries which also do this for you as per the examples here. You won't actually see any JSON unless you inspect the HTTP traffic or use the API Explorer.
It sounds like you might just be doing more work than is needed by pre-encoding the object, rather than just letting Endpoints do it for you. If you really need to manually serialize an object to some property you can use a library on the Endpoints side like Jackson to serialize the object to a string property and NSJSONSerialization on the client to convert it back to an object.
I'm using the GMail API to retrieve an email contents. I am getting the following base64 encoded data for the body: http://hastebin.com/ovucoranam.md
But when I run it through a base64 decoder, it either returns an empty string (error) or something that resembles the HTML data but with a bunch of weird characters.
Help?
I'm not sure if you've solved it yet, but GmailGuy is correct. You need to convert the body to the Base64 RFC 4648 standard. The jist is you'll need to replace - with + and _ with /.
I've taken your original input and did the replacement: http://hastebin.com/ukanavudaz
And used base64decode.org to decode it, and it was fine.
You need to use URL (aka "web") safe base64 decoding alphabet (see rfc 4648), which it doesn't appear you're doing. Using the standard base64 alphabet may work sometimes but not always (2 of the characters are different).
Docs don't seem to consistently mention this important detail. Here's one where it does though:
https://developers.google.com/gmail/api/guides/drafts
Also, if your particular library doesn't support the "URL safe" alphabet then you can do string substitution on the string first ("-" with "+" and "_" with "/") and then do normal base64 decoding on it.
I had the same issue decoding the 'data' fields in the message object response from the Gmail API. The Google Ruby API library wasn't decoding the text correctly either. I found I needed to do a url-safe base64 decode:
#data = Base64.urlsafe_decode64(JSON.parse(#result.data.to_json)["payload"]["body"]["data"])
Hope that helps!
There is an example for python 2.x and 3.x:
decodedContents = base64.urlsafe_b64decode(payload["body"]["data"].encode('ASCII'))
If you only need to decode for displaying purposes, consider using atob to decode the messages in JavaScript frontend (see ref).
I found whilst playing with the API result, once I had drilled down to the body I was given an option to decode in the available methods.
val message = mService!!.users().messages().get(user, id).setFormat("full").execute()
println("Message snippet: " + message.snippet)
if(message.payload.mimeType == "text/plain"){
val body = message.payload.body.decodeData() // getValue("body")
Log.i("BODY", body.toString(Charset.defaultCharset()))
}
The result:-
com.example.quickstart I/BODY: ISOLATE NORMAL: 514471,Fap, South Point Rolleston, 55 Faringdon Boulevard , Rolleston, 30 May 2018 20:59:21
I coped the base64 test to a file (b64.txt), then base64-decoded it using base64 (from coreutils) with the -d option (see http://linux.die.net/man/1/base64) and I got text that was perfectly readable. The command I used was:
cat b64.txt | base64 -d