I am trying to decode a JWT I get from Auth0. When I go to jwt.io, they have a decoder that you can put a JWT in, and it will tell you all the information about each section of the JWT. I can see that all the information is correct. When I try to decode it myself though, I get this error. I'm getting the secret key from my Auth0 registered client information, and there is a note that says: The Client Secret is not base64 encoded. Do I need to base64 encode this secret before using it ?
ValueError: Could not unserialize key data.
Terminal
>>> import jwt
>>> secret = secret
>>> encoded_jwt = encoded_jwt
>>> decoded_jwt = jwt.decode(encoded_jwt, secret, algorithm="RS256")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/site-packages/jwt/api_jwt.py", line 78, in decode
jwt, key=key, algorithms=algorithms, options=options, **kwargs
File "/usr/local/lib/python3.6/site-packages/jwt/api_jws.py", line 140, in decode
key, algorithms)
File "/usr/local/lib/python3.6/site-packages/jwt/api_jws.py", line 204, in _verify_signature
key = alg_obj.prepare_key(key)
File "/usr/local/lib/python3.6/site-packages/jwt/algorithms.py", line 207, in prepare_key
key = load_pem_public_key(key, backend=default_backend())
File "/usr/local/lib/python3.6/site-packages/cryptography/hazmat/primitives/serialization.py", line 24, in load_pem_public_key
return backend.load_pem_public_key(data)
File "/usr/local/lib/python3.6/site-packages/cryptography/hazmat/backends/multibackend.py", line 314, in load_pem_public_key
return b.load_pem_public_key(data)
File "/usr/local/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1110, in load_pem_public_key
self._handle_key_loading_error()
File "/usr/local/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1325, in _handle_key_loading_error
raise ValueError("Could not unserialize key data.")
ValueError: Could not unserialize key data.
As you don't mention a PUBLIC KEY nor a PRIVATE KEY, it looks like you are trying to decode using the "RS256" algorithm, but your token uses "HS256".
try:
decoded_jwt = jwt.decode(encoded_jwt, secret, algorithm="HS256")
instead of:
decoded_jwt = jwt.decode(encoded_jwt, secret, algorithm="RS256")
You are free to encode the key using base64 if you like at this address:
https://www.base64encode.org/
You can verify the encoded key by checking the "secret base64 encoded" check box at jwt.io, under the VERIFY SIGNATURE section.
Related
JWT tokens required signing key to decode them, but in https://jwt.io/ it can be decoded without any signing key, how is this possible.
You do not need a key to open the encoding, you need a key to verify that nobody changed the contents of the JWT. In fact the string you see is just json's base64 with your information, metadata, and "signature" on all content.
The signature is the most important part of a JSON Web Token(JWT). A signature is calculated by encoding the header and payload using Base64url Encoding and concatenating them with a period separator. Which is then given to the cryptographic algorithm.
// signature algorithm
data = base64urlEncode( header ) + “.” + base64urlEncode( payload )
signature = HMAC-SHA256( data, secret_salt )
So when the header or payload changes, the signature has to calculated again. Only the Identity Provider(IdP) has the private key to calculate the signature which prevents the tampering of token.
read more:
https://medium.com/#sureshdsk/how-json-web-token-jwt-authentication-works-585c4f076033
https://jwt.io/introduction/
https://www.youtube.com/watch?v=7Q17ubqLfaM
In my app I get a PEM encoded certificate and need to convert it into a different form to later use it for JWT verification purposes. The result I'm looking for is either a SecKey representation of the public key contained in the certificate, PEM Public Key string or conversion to a DER certificate.
I am very VERY new to this, so I have no idea how to tackle the problem. I've Googled around and found no clear solution, even Apple documentation only mentions DER certificates. If I understand it correctly, one solution would be to use OpenSSL inside my app (is this even possible?) for conversions, but I couldn't find any useful resource on how to implement this so I would appreciate more insight on the right practice.
Making conversions outside the application is not an option for my case.
If you have a certificate in PEM format, you can obtain it's DER encoded data by stripping away the header and footer lines and base64-decoding the text in between them (don't forget to discard line breaks as well). [1, 2]
You can then use SecCertificateCreateWithData from the iOS Security API to create a SecCertificate to use in your app like so:
// example data from http://fm4dd.com/openssl/certexamples.htm
let pem = """
-----BEGIN CERTIFICATE-----
MIIC2jCCAkMCAg38MA0GCSqGSIb3DQEBBQUAMIGbMQswCQYDVQQGEwJKUDEOMAwG
(...)
+tZ9KynmrbJpTSi0+BM=
-----END CERTIFICATE-----
"""
// remove header, footer and newlines from pem string
let certData = Data(base64Encoded: pemWithoutHeaderFooterNewlines)!
guard let certificate = SecCertificateCreateWithData(nil, data as CFData) else {
// handle error
}
// use certificate e.g. copy the public key
let publicKey = SecCertificateCopyKey(certificate)!
[1]
PEM format is simply base64 encoded data surrounded by header lines.
[2]
.pem – Base64 encoded DER certificate, enclosed between "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----"
From what I can understand, it's a straight forward process to validate a JWT signature. But when I use some online tools to do this for me, it doesn't match. How can I manually validate a JWT signature without using a JWT library? I'm needing a quick method (using available online tools) to demo how this is done.
I created my JWT on https://jwt.io/#debugger-io with the below info:
Algorithm: HS256
Secret: hONPMX3tHWIp9jwLDtoCUwFAtH0RwSK6
Header:
{
"alg": "HS256",
"typ": "JWT"
}
Payload:
{
"sub": "1234567890",
"name": "John Doe",
"iat": 1516239022
}
Verify Signature (section):
Secret value changed to above
"Checked" secret base64 encoded (whether this is checked or not, still get a different value)
JWT:
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.wDQ2mU5n89f2HsHm1dluHGNebbXeNr748yJ9kUNDNCA
Manual JWT signature verification attempt:
Using a base64UrlEncode calculator (http://www.simplycalc.com/base64url-encode.php or https://www.base64encode.org/)
If I: (Not actual value on sites, modified to show what the tools would ultimately build for me)
base64UrlEncode("eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9") + "." + base64UrlEncode("eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ")
I get:
ZXlKaGJHY2lPaUpJVXpJMU5pSXNJblI1Y0NJNklrcFhWQ0o5.ZXlKemRXSWlPaUl4TWpNME5UWTNPRGt3SWl3aWJtRnRaU0k2SWtwdmFHNGdSRzlsSWl3aWFXRjBJam94TlRFMk1qTTVNREl5ZlE=
NOTE: there's some confusion on my part if I should be encoding the already encoded values, or use the already encoded values as-is.
(i.e. using base64UrlEncode("eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9") + "." + base64UrlEncode("eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ") vs "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ").
Regardless on which I should do, the end result still doesn't match the signature. I'm leaning towards that I should NOT re-encode the encoded value, whether that's true or not.
Then using a HMAC Generator calculator (https://codebeautify.org/hmac-generator or https://www.freeformatter.com/hmac-generator.html#ad-output)
(Not actual value on sites, modified to show what the tools would ultimately build for me)
HMACSHA256(
"ZXlKaGJHY2lPaUpJVXpJMU5pSXNJblI1Y0NJNklrcFhWQ0o5.ZXlKemRXSWlPaUl4TWpNME5UWTNPRGt3SWl3aWJtRnRaU0k2SWtwdmFHNGdSRzlsSWl3aWFXRjBJam94TlRFMk1qTTVNREl5ZlE=",
"hONPMX3tHWIp9jwLDtoCUwFAtH0RwSK6"
)
Which gets me:
a2de322575675ba19ec272e83634755d4c3c2cd74e9e23c8e4c45e1683536e01
And that doesn't match the signature portion of the JWT:
wDQ2mU5n89f2HsHm1dluHGNebbXeNr748yJ9kUNDNCAM != a2de322575675ba19ec272e83634755d4c3c2cd74e9e23c8e4c45e1683536e01
Purpose:
The reason I'm needing to confirm this is to prove the ability to validate that the JWT hasn't been tampered with, without decoding the JWT.
My clients web interface doesn't need to decode the JWT, so there's no need for them to install a jwt package for doing that. They just need to do a simple validation to confirm the JWT hasn't been tampered with (however unlikely that may be) before they store the JWT for future API calls.
It's all a matter of formats and encoding.
On https://jwt.io you get this token based on your input values and secret:
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.3pIaKksiX9Zv8Jg-hWbrD24VhL36hBIFaNpA4fVx29M
We want to prove that the signature:
3pIaKksiX9Zv8Jg-hWbrD24VhL36hBIFaNpA4fVx29M
is correct.
The signature is a HMAC-SHA256 hash that is Base64url encoded.
(as described in RFC7515)
When you use the online HMAC generator to calculate a hash for
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ
with the secret
hONPMX3tHWIp9jwLDtoCUwFAtH0RwSK6
you get
de921a2a4b225fd66ff0983e8566eb0f6e1584bdfa84120568da40e1f571dbd3
as result, which is a HMAC-SHA256 value, but not Base64url encoded. This hash is a hexadecimal string representation of a large number.
To compare it with the value from https://jwt.io you need to convert the value from it's hexadecimal string representation back to a number and Base64url encode it.
The following script is doing that and also uses crypto-js to calculate it's own hash. This can also be a way for you to verify without JWT libraries.
var CryptoJS = require("crypto-js");
// the input values
var base64Header = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9";
var base64Payload = "eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ";
var secret = "hONPMX3tHWIp9jwLDtoCUwFAtH0RwSK6";
// two hashes from different online tools
var signatureJWTIO = "3pIaKksiX9Zv8Jg-hWbrD24VhL36hBIFaNpA4fVx29M";
var onlineCaluclatedHS256 = "de921a2a4b225fd66ff0983e8566eb0f6e1584bdfa84120568da40e1f571dbd3";
// hash calculation with Crypto-JS.
// The two replace expressions convert Base64 to Base64url format by replacing
// '+' with '-', '/' with '_' and stripping the '=' padding
var base64Signature = CryptoJS.HmacSHA256(base64Header + "." + base64Payload , secret).toString(CryptoJS.enc.Base64).replace(/\+/g,'-').replace(/\//g,'_').replace(/\=+$/m,'');
// converting the online calculated value to Base64 representation
var base64hash = new Buffer.from(onlineCaluclatedHS256, 'hex').toString('base64').replace(/\//g,'_').replace(/\+/g,'-').replace(/\=+$/m,'')
// the results:
console.log("Signature from JWT.IO : " + signatureJWTIO);
console.log("NodeJS calculated hash : " + base64Signature);
console.log("online calulated hash (converted) : " + base64hash);
The results are:
Signature from JWT.IO : 3pIaKksiX9Zv8Jg-hWbrD24VhL36hBIFaNpA4fVx29M
NodeJS calculated hash : 3pIaKksiX9Zv8Jg-hWbrD24VhL36hBIFaNpA4fVx29M
online calulated hash (converted) : 3pIaKksiX9Zv8Jg-hWbrD24VhL36hBIFaNpA4fVx29M
identical!
Conclusion:
The values calculated by the different online tools are all correct but not directly comparable due to different formats and encodings.
A little script as shown above might be a better solution.
I had the same problem until I figured out that I was using plain base64 encoding instead of base64url.
There are also some minor details in between.
Here is the step-by-step manual that will, hopefully, make the whole process much more clear.
Notes
Note 1: You must remove all spaces and newlines from your JSON strings (header and payload).
It is implicitly done on jwt.io when you generate a JWT token.
Note 2: To convert JSON string to base64url string on cryptii.com create the following configuration:
First view: Text
Second view: Encode
Encoding: Base64
Variant: Standard 'base64url' (RFC 4648 §5)
Third view: Text
Note 3: To convert HMAC HEX code (signature) to base64url string on cryptii.com create the following configuration:
First view: Bytes
Format: Hexadecimal
Group by: None
Second view: Encode
Encoding: Base64
Variant: Standard 'base64url' (RFC 4648 §5)
Third view: Text
Manual
You are going to need only two online tools:
[Tool 1]: cryptii.com - for base64url encoding,
[Tool 2]: codebeautify.org - for HMAC calculation.
On cryptii.com you can do both base64url encoding/decoding and also HMAC calculation, but for HMAC you need to provide a HEX key which is different from the input on jwt.io, so I used a separate service for HMAC calculation.
Input data
In this manual I used the following data:
Header:
{"alg":"HS256","typ":"JWT"}
Payload:
{"sub":"1234567890","name":"John Doe","iat":1516239022}
Secret (key):
The Earth is flat!
The secret is not base64 encoded.
Step 1: Convert header [Tool 1]
Header (plain text):
{"alg":"HS256","typ":"JWT"}
Header (base64url encoded):
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9
Step 2: Convert payload [Tool 1]
Payload (plain text):
{"sub":"1234567890","name":"John Doe","iat":1516239022}
Payload (base64url encoded):
eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ
Step 3: Calculate HMAC code (signature) [Tool 2]
Calculate HMAC using SHA256 algorithm.
Input string (base64url encoded header and payload, concatenated with a dot):
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ
Calculated code (HEX number):
c8a9ae59f3d64564364a864d22490cc666c74c66a3822be04a9a9287a707b352
The calculated HMAC code is a HEX representation of the signature.
Note: it should not be encoded to base64url as a plain text string but as a sequence of bytes.
Step 4: Encode calculated HMAC code to base64url [Tool 1]:
Signature (Bytes):
c8a9ae59f3d64564364a864d22490cc666c74c66a3822be04a9a9287a707b352
Signature (base64url encoded):
yKmuWfPWRWQ2SoZNIkkMxmbHTGajgivgSpqSh6cHs1I
Summary
Here are our results (all base64url encoded):
Header:
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9
Payload:
eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ
Signature:
yKmuWfPWRWQ2SoZNIkkMxmbHTGajgivgSpqSh6cHs1I
The results from jwt.io:
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.yKmuWfPWRWQ2SoZNIkkMxmbHTGajgivgSpqSh6cHs1I
As you can see, all three parts are identical.
I am using this code to write a bottrader,in python3 (converted 2to3)
https://pastebin.com/fbkheaRb
except that i have change secret string and post_data string to byte
sign = hmac.new(self.Secret.encode("ASCII"), post_data.encode("ASCII"), hashlib.sha512).hexdigest()
but getting below error
urllib.error.HTTPError: HTTP Error 403: Forbidden
have checked key and secret key multiple time and it is correct
Also deleted the existing key and created new
Also relaxed all IP
then also got the same problem, please help
You don't need to encode to ASCII, so you may try:
sign = hmac.new(self.Secret, post_data, hashlib.sha512).hexdigest()
I want to call a web service method that has a spyne binary type as argument.
But I'm not able to find any python soap client supporting it.
To debug my problem, I made a simple web service method that should print a file:
# #srpc(Attachment, _returns=Unicode)
#srpc(ByteArray, _returns=Unicode)
# #srpc(File, _returns=Unicode)
# #srpc(Unicode, _returns=Unicode)
def print_file(file_content):
logger.info(u"print file:\n{}\ntype:{}".format(file_content, file_content.__class__))
return u''
As you can see, I tried with 3 spyne binary types.
For debug, I also tried with Unicode, and passing the file content in base64, and in this case there is no problem.
Thus the web service is operational.
The server side is a Django application and a spyne application.
My problem is on client side.
With suds, the obtained error is very obscur, and no solution exists according forums.
I tried all SOAP clients described on https://wiki.python.org/moin/WebServices#SOAP, with python 2.7 and 3.3.
They all fail when building the request, when serializing the spyne binary object.
My last try is with zeep.
I instantiate the zeep client with wsdl local url.
Sorry, wsdl is not public.
I call this method with an empty ByteArray:
param = ByteArray()
client.service.print_file(param)
The catched exception is:
File "/usr/lib/python2.7/site-packages/zeep/client.py", line 41, in __call__
self._op_name, args, kwargs)
File "/usr/lib/python2.7/site-packages/zeep/wsdl/bindings/soap.py", line 107, in send
options=options)
File "/usr/lib/python2.7/site-packages/zeep/wsdl/bindings/soap.py", line 65, in _create
serialized = operation_obj.create(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/zeep/wsdl/definitions.py", line 165, in create
return self.input.serialize(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/zeep/wsdl/messages/soap.py", line 48, in serialize
self.body.render(body, body_value)
File "/usr/lib/python2.7/site-packages/zeep/xsd/elements.py", line 333, in render
self._render_value_item(parent, value)
File "/usr/lib/python2.7/site-packages/zeep/xsd/elements.py", line 354, in _render_value_item
return self.type.render(node, value)
File "/usr/lib/python2.7/site-packages/zeep/xsd/types.py", line 356, in render
element.render(parent, element_value)
File "/usr/lib/python2.7/site-packages/zeep/xsd/indicators.py", line 189, in render
element.render(parent, element_value)
File "/usr/lib/python2.7/site-packages/zeep/xsd/elements.py", line 333, in render
self._render_value_item(parent, value)
File "/usr/lib/python2.7/site-packages/zeep/xsd/elements.py", line 354, in _render_value_item
return self.type.render(node, value)
File "/usr/lib/python2.7/site-packages/zeep/xsd/types.py", line 180, in render
parent.text = self.xmlvalue(value)
File "/usr/lib/python2.7/site-packages/zeep/xsd/builtins.py", line 83, in _wrapper
return func(self, value)
File "/usr/lib/python2.7/site-packages/zeep/xsd/builtins.py", line 357, in xmlvalue
return base64.b64encode(value)
File "/usr/lib64/python2.7/base64.py", line 53, in b64encode
encoded = binascii.b2a_base64(s)[:-1]
TypeError: must be convertible to a buffer, not ModelBaseMeta
Does anyone know if there is a solution with zeep ?
Perhaps with a dedicated zeep plugin ?
Or is there another solution ?
Perhaps in C/C++, compiled as a python package ?
Eric
============================================================
== Solution
As I lost a lot of time on this problem, here is the solution for me.
Finally I needed also the filename, not only file data.
The type spyne.model.binary.File would have been perfect, but it is not serialisable into SOAP format.
Thank to discussion with Burak, the final solution is to create a custom ComplexType like:
class File(ComplexModel):
filename = Unicode
data = ByteArray
On client side with suds, replace the ByteArray field directly with the data encoded in base64:
f = File()
data = open({FILENAME}, "rb").read()
f.data = base64.b64encode(data)
On server side, f.data will directly contain the data decoded.
Complete answer is here: http://lists.spyne.io/archives/people/2016-December/000187.html
For completeness; you can use suds if you do base64 operations manually. See https://github.com/arskom/spyne/blob/be222c041837c9f7cd1e6e6e455e6704b5069837/spyne/test/interop/test_suds.py#L111
Zeep should handle bytes fine, I've just tested this against a spyne server.
E.g. service.client.echo_bytearray(b'\x00\x01\x02\x03\x04') works without issue. Do you have more information?
Cheers, Michael (author of zeep)