can't connect to azure file service REST api by python - rest

I want to access my file service by python Request, guided by the list shares page, I'm new to REST and python Request.
my input is
headers= {'x-ms-date': '2018-04-17 06:22:15.181620', 'Authorization': 'SharedKey zedongstorageaccount:codecodecodeFiTzubX9tvC3G3PcDYzR2cX/TMjkOu4JhsvQffS+xTDDBQ==', 'x-ms-version': '2017-07-29'}
url = 'https://zedongstorageaccount.file.core.windows.net/?comp=list'
r=requests.get(url,headers=headers)
But get error, the output of r.content:
b'\xef\xbb\xbf<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:ef98f282-f01a-0042-3e24-d62397000000\nTime:2018-04-17T08:16:21.9635335Z</Message><AuthenticationErrorDetail>The Date header in the request is incorrect.</AuthenticationErrorDetail></Error>'
I have also try the header {'x-ms-date': '2018-04-17 06:22:15.181620', 'Authorization': 'SharedKey zedongnewstorageaccount:NJYYtabOIj5D1R+xb0PpmXrjCldf6NA6oLkYReAKFiTzubX9tvC3G3PcDYzR2cX/TMjkOu4JhsvQffS+xTDDBQ==', 'x-ms-version': '2017-07-29'}, because i don't know if the Authorization and account in the same line. I also tried many version of x-ms-version.
But there all the 400 or 403 respond.
I have read the Authentication for the Azure Storage Services but confused.
Where is the error? And is there case I can learn to build my app by Azure REST API?(I use the keyword to google, all the pages are about build REST API and azure official docs)

I also recommend to use Python SDK but for educational purposes:
import requests
import datetime
import hmac
import hashlib
import base64
storage_account_name = 'teststorageaccount'
storage_account_key = 'D4x6YChpyfqWk9a.......'
api_version = '2016-05-31'
request_time = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
string_params = {
'verb': 'GET',
'Content-Encoding': '',
'Content-Language': '',
'Content-Length': '',
'Content-MD5': '',
'Content-Type': '',
'Date': '',
'If-Modified-Since': '',
'If-Match': '',
'If-None-Match': '',
'If-Unmodified-Since': '',
'Range': '',
'CanonicalizedHeaders': 'x-ms-date:' + request_time + '\nx-ms-version:' + api_version + '\n',
'CanonicalizedResource': '/' + storage_account_name + '/\ncomp:list'
}
string_to_sign = (string_params['verb'] + '\n'
+ string_params['Content-Encoding'] + '\n'
+ string_params['Content-Language'] + '\n'
+ string_params['Content-Length'] + '\n'
+ string_params['Content-MD5'] + '\n'
+ string_params['Content-Type'] + '\n'
+ string_params['Date'] + '\n'
+ string_params['If-Modified-Since'] + '\n'
+ string_params['If-Match'] + '\n'
+ string_params['If-None-Match'] + '\n'
+ string_params['If-Unmodified-Since'] + '\n'
+ string_params['Range'] + '\n'
+ string_params['CanonicalizedHeaders']
+ string_params['CanonicalizedResource'])
signed_string = base64.b64encode(hmac.new(base64.b64decode(storage_account_key), msg=string_to_sign.encode('utf-8'), digestmod=hashlib.sha256).digest()).decode()
headers = {
'x-ms-date' : request_time,
'x-ms-version' : api_version,
'Authorization' : ('SharedKey ' + storage_account_name + ':' + signed_string)
}
url = ('https://' + storage_account_name + '.file.core.windows.net/?comp=list')
r = requests.get(url, headers = headers)
print(r.content)
Output:
<?xml version="1.0" encoding="utf-8"?><EnumerationResults ServiceEndpoint="https://teststorageaccount.file.core.windows.net/"><Shares><Share><Name>myfileshare</Name><Properties><Last-Modified>Tue, 17 Apr 2018 13:43:00 GMT</Last-Modified><Etag>"0x8D5A46922CA8BDA"</Etag><Quota>10</Quota></Properties></Share></Shares><NextMarker /></EnumerationResults>

Related

Selecting sql data with multiple rows and exporting as xml file

I have this JavaScript in my transformer
var dbConn;
try {
dbConn = DatabaseConnectionFactory.createDatabaseConnection($gc('DataWarehouseXMLDataConfig').sqlDriver,
$gc('DataWarehouseXMLDataConfig').sqlConnString,
$gc('DataWarehouseXMLDataConfig').sqlUsername,
$gc('DataWarehouseXMLDataConfig').sqlPassword);
var data = dbConn.executeCachedQuery('EXEC [dbo].[GetDataWarehouseData]');
var xml = "";
var xml2 = "";
var row = "";
while(data.next()) {
row = '<referral>' + '\n' +
'<careid>' + data.getString('careid') + '</careid>' + '\n' +
'<patientid>' + data.getString('patientid') + '</patientid>' + '\n' +
'<dateofreceipt>' + data.getString('dateofreceipt') + '</dateofreceipt>' + '\n' +
'<datefirstseen>' + data.getString('datefirstseen') + '</datefirstseen>' + '\n' +
'<nhsnumber>' + data.getString('nhsnumber') + '</nhsnumber>' + '\n' +
'<hospitalnumber>' + data.getString('hospitalnumber') + '</hospitalnumber>' + '\n' +
'<surname>' + data.getString('surname') + '</surname>' + '\n' +
'<forename>' + data.getString('forename') + '</forename>' + '\n' +
'<dateofbirth>' + data.getString('dateofbirth') + '</dateofbirth>' + '\n' +
'</referral>';
xml = xml + row;
xml2 = '<?xml version="1.0" encoding="utf-8"?>' + '\n' + '<results>' + xml + '</results>';
}
channelMap.put('xml2', xml2);
} finally {
if (dbConn) {
dbConn.close();
}
}
I'm using this outbound message template
<referral>
<careid></careid>
<patientid></patientid>
<dateofreceipt></dateofreceipt>
<datefirstseen></datefirstseen>
<nhsnumber></nhsnumber>
<hospitalnumber></hospitalnumber>
<surname></surname>
<forename></forename>
<dateofbirth></dateofbirth>
</referral>
I have a File Writer destination and the File Name has a .xml extension.
My template in the File Writer is ${XmlUtil.prettyPrint(${message.encodedData})}.
The SQL data is being retrieved ok.
I don't think I'm a million miles away but I'm doing something fundamentally wrong.
What do I need to do to send the required xml file please?
I fixed this by adding a mapping step to the source transformer and using JavaScript, like
tmp['referral']['dateofbirth'] = msg['dateofbirth'].toString();

twilio API issues when submitting whastapp query

I'm using HttpURLConnection to post to https://api.twilio.com/2010-04-01/Accounts/****/Messages.json
this works for SMS - but when the txtTo and txtFrom are changed to To=whatsapp:+12345567&From=whatsapp:+123456778&Body=sacsacsac
it stops working ..... any ideas?
String message = "To=" + txtTo + "&From=" + txtFrom + "&Body=" + txtBody;
byte[] postData = message.getBytes( StandardCharsets.UTF_8 );
String message = "From=" + URLEncoder.encode("whatsapp:" + txtFrom, "UTF-8") +"&Body=" + URLEncoder.encode(txtBody, "UTF-8") + "&To=" + URLEncoder.encode("whatsapp:" + txtTo, "UTF-8");

AWS Cloudfront files' invalidation via REST API and Powershell

I'm trying to write a PowerShell script to invalidate an object of AWS Cloudfront distribution (just a specific file), and not sure how to produce the "signed URL" they're asking for.
My code so far is:
$authPref = "AWS4-HMAC-SHA256"
$AWSAccessKey = "xxx"
$AWSSecretKey = "xxx"
$awsDateOnly = (Get-Date).AddHours(-3).ToString("yyyyMMdd")
$awsRegion = "us-east-1"
$awsServiceName = "cloudfront"
$awsRequestType = "aws4_request"
$stringToSign = $authPref + " " + $awsCallerReference + " " + $awsDateOnly + "/" + $awsRegion + "/" + $awsServiceName + "/" + $awsRequestType + " SOME_STRING_NOT_SURE_WHAT"
$hmacsha = New-Object System.Security.Cryptography.HMACSHA256
$hmacsha.key = [Text.Encoding]::ASCII.GetBytes($stringToSign)
$awsHMAC = $hmacsha.ComputeHash([Text.Encoding]::ASCII.GetBytes($AWSSecretKey))
$awsHMAC = [Convert]::ToBase64String($awsHMAC)
$awsSignedToken = $authPref + " Credential=" + $AWSAccessKey + "/" + $awsDateOnly + "/" + $awsRegion + "/" + $awsServiceName + "/" + $awsRequestType + ", SignedHeaders=content-type;host;x-amz-date, Signature=" + $awsHMAC
#POST /2017-03-25/distribution/$awsDistributionID/invalidation HTTP/1.1
$awsDistributionID = "xxx"
$awsCallerReference = (Get-Date).AddHours(-3).ToString("yyyyMMdd'T'HHmmss'Z'")
$invalidateObjectXML = #"
<?xml version="1.0" encoding="UTF-8"?>
<InvalidationBatch xmlns="http://cloudfront.amazonaws.com/doc/2017-03-25/">
<CallerReference>$awsCallerReference</CallerReference>
<Paths>
<Items>
<Path>/</Path>
</Items>
<Quantity>1</Quantity>
</Paths>
</InvalidationBatch>
"#
[xml]$invalidateObjectXML = $invalidateObjectXML
$awsCFuri = "https://cloudfront.amazonaws.com/2017-03-25/distribution/$awsDistributionID/invalidation"
Invoke-WebRequest -Method POST `
-Uri $awsCFuri `
-Headers #{"content-type"="text/xml";
"x-amz-date"="$awsCallerReference";
"authorization"="$awsSignedToken";
"host"="cloudfront.amazonaws.com"} `
-Body $invalidateObjectXML
The response is:
<ErrorResponse xmlns="http://cloudfront.amazonaws.com/doc/2017-03-25/"><Error><Type>Sender</Type><Code>SignatureDoesNotMatch</Code><Message>The request
signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation
for details.
The Canonical String for this request should have been
'POST
/2017-03-25/distribution/blabla/invalidation
content-type:text/xml
host:cloudfront.amazonaws.com
x-amz-date:20170503T203650Z
content-type;host;x-amz-date
blabla'
The String-to-Sign should have been
'AWS4-HMAC-SHA256
20170503T203650Z
20170503/us-east-1/cloudfront/aws4_request
blabla'
</Message></Error><RequestId>123-123</RequestId></ErrorResponse>
At line:1 char:1
So obviously I'm doing something wrong with the signed URL string that I do, but what it is?
Couldn't find any examples on the internet (not AWS docs nor any other blog) that demonstrates it in Powershell.
Thanks
AWS has developed a PowerShell module to interact with the AWS API called the AWS Tools for PowerShell. This module specifically handles request building and signing for you, so this method of calling to the raw API becomes unnecessary.
You can use this specifically to invalidate objects in a CloudFront distribution with the New-CFInvalidation cmdlet. Write the paths you want to invalidate to the Paths_Item parameter.
Signature:
New-CFInvalidation
-DistributionId <String>
-InvalidationBatch_CallerReference <String>
-Paths_Item <String[]>
-Paths_Quantity <Int32>
-Force <SwitchParameter>
Further Reading
AWS Documentation - Invalidating Objects and Displaying Information about Invalidations
AWS Documentation - New-CFInvalidation Cmdlet

How do you list out all output / generate files from a Gradle Build?

Currently I am using the following to generate list of output files which I want for another plugin which does Github releases.
[
"${buildDir}/libs/${project.name}-${project.version}.jar",
"${buildDir}/libs/${project.name}-${project.version}-javadoc.jar",
"${buildDir}/libs/${project.name}-${project.version}-sources.jar"
]
I am not very satisfied with this as I am manually specifying each type of file (doc, source, class). I am looking to make this more generic where I can reuse without any modification for any project which might have more or less outputs and also may not be in the default location and some outputs may not even be Java outputs. I.e. changing output location, the tasks to run or plugging elsewhere in the script should not break this part of the script and should be reusable without editing. The question is how do you get a list of all the outputs as a List of String like above.
More detailed code snippet:
...
plugins {
id 'co.riiid.gradle' version '0.4.2'
}
...
ext {
org = 'MicroTrader'
repo = org + '/' + project.name
releaseNotesFile = 'RELEASE_NOTES.md'
host = 'github.com'
hostUrl = 'https://' + host + '/'
orgUrl = hostUrl + org
websiteUrl = hostUrl + repo
vcsUrl = websiteUrl + '.git'
scmUrl = 'scm:git#' + host + ':' + repo + '.git'
issueTrackerUrl = websiteUrl + '/issues'
ossrhUsername = ((project.hasProperty('ossrhUsername') ? ossrhUsername : System.getenv('OSSRH_USER')) ?: System.getProperty('ossrh.user')) ?: ' '
ossrhPassword = ((project.hasProperty('ossrhPassword') ? ossrhPassword : System.getenv('OSSRH_PASSWORD')) ?: System.getProperty('ossrh.password')) ?: ' '
bintray_user = ((project.hasProperty('bintray_user') ? bintray_user : System.getenv('BINTRAY_USER')) ?: System.getProperty('bintray.user')) ?: ' '
bintray_api_key = ((project.hasProperty('bintray_api_key') ? bintray_api_key : System.getenv('BINTRAY_KEY')) ?: System.getProperty('bintray.apikey')) ?: ' '
artifactory_user = ((project.hasProperty('artifactory_user') ? artifactory_user : System.getenv('ARTIFACTORY_USER')) ?: System.getProperty('artifactory.user')) ?: ' '
artifactory_password = ((project.hasProperty('artifactory_password') ? artifactory_password : System.getenv('ARTIFACTORY_PASSWORD')) ?: System.getProperty('artifactory.password')) ?: ' '
github_api_key = ((project.hasProperty('github_api_key') ? github_api_key : System.getenv('GITHUB_KEY')) ?: System.getProperty('github.apikey')) ?: ' '
}
...
github {
owner = project.org
repo = project.name
token = project.github_api_key
tagName = project.version
targetCommitish = 'master'
name = 'v' + project.version
body = Files.exists(Paths.get('' + projectDir + '/' + releaseNotesFile)) ?
new File('' + projectDir + '/' + releaseNotesFile).text :
(Files.exists(Paths.get('' + baseDir + '/' + releaseNotesFile)) ?
new File('' + baseDir + '/' + releaseNotesFile).text : '')
assets = [
"${buildDir}/libs/${project.name}-${project.version}.jar",
"${buildDir}/libs/${project.name}-${project.version}-javadoc.jar",
"${buildDir}/libs/${project.name}-${project.version}-sources.jar"
]
}
...
Try reading the outputs property on all of the tasks in your build used to generate the artifacts:
project(':my-sub-project').jar.outputs
Another option is to use the archives configuration from all of your sub-projects:
project(':my-sub-project').configurations.archives.allArtifacts

Yet another "Failed to validate oauth signature and token"

I've this problem that many others have been through. I'm doing everything correct but still i get this annoying "Failed to validate oauth signature and token" error :)
Well, something got to be wrong I guess..
I'm trying to obtain a request_token by making a post to "https://api.twitter.com/oauth/request_token" with headers:
Authorization:
OAuth oauth_consumer_key="MYVq....................ywj2g",
oauth_nonce="m8NG0s4oc87AOIpuILafAeI1YoMv5Mu9",
oauth_signature="Bxb%252FFIfOG9KLVj%252FUNdV%252FycVlGPs%253D",
oauth_signature_method="HMAC-SHA1",
oauth_timestamp="1378976842",
oauth_version="1.0"
But it complains about signature and token.
Is my signature invalid somehow?
And for this request I dont need a token right??
I can't figure out whats wrong.
Here's some of my getRequestToken code:
val oauth_consumer_key: String = CONSUMER_KEY
val oauth_nonce: String = generateNonce()
val oauth_timestamp: String = (System.currentTimeMillis / 1000).toString
var oauth_signature: String = ""
val oauth_signature_method: String = "HMAC-SHA1"
val oauth_version: String = "1.0"
val PARAMETER_STRING: String =
"oauth_consumer_key=" + oauth_consumer_key + "&" +
"oauth_nonce=" + oauth_nonce + "&" +
"oauth_signature_method=" + oauth_signature_method + "&" +
"oauth_timestamp=" + oauth_timestamp + "&" +
"oauth_version=" + oauth_version
val BASE_STRING: String =
"POST&" + URLEncoder.encode("https://api.twitter.com/oauth/request_token", "UTF-8") + "&" + URLEncoder.encode(PARAMETER_STRING, "UTF-8")
oauth_signature = getSignature(CONSUMER_SECRET, BASE_STRING, "HmacSHA1")
val AUTHORIZATION = "OAuth " +
"oauth_consumer_key=\"" + URLEncoder.encode(oauth_consumer_key, "UTF-8") +
"\", oauth_nonce=\"" + URLEncoder.encode(oauth_nonce, "UTF-8") +
"\", oauth_signature=\"" + URLEncoder.encode(oauth_signature, "UTF-8") +
"\", oauth_signature_method=\"" + URLEncoder.encode(oauth_signature_method, "UTF-8") +
"\", oauth_timestamp=\"" + URLEncoder.encode(oauth_timestamp, "UTF-8") +
"\", oauth_version=\"" + URLEncoder.encode(oauth_version, "UTF-8") + "\""
WS.url("https://api.twitter.com/oauth/request_token").withHeaders("Authorization" -> AUTHORIZATION).post(Results.EmptyContent()).map(response => {
if(response.status != 200) Logger.error(response.body) //THIS IS WHERE I GET THE ERROR
else {
if((response.json \ "oauth_callback_confirmed").as[String] == "true") {
REQUEST_TOKEN = (response.json \ "oauth_token").as[String]
REQUEST_SECRET = (response.json \ "oauth_token_secret").as[String]
requestDone.success(true)
}
}
})
Ok so I've got everyting to work (without the oauth_callback parameter, because if I add this I get the error again).
I get the Request_token, which is valid because when I manually paste the authenticate url in the browser together with the generated request token I get redirected to twitter authenticate page and then a correct callback is made and the result is correct also. (token, token_secret, user_id and screen_name)
But my code seem to ignore my redirect to this authorize page.
requestToken_future.map { result =>
Redirect("https://api.twitter.com/oauth/authenticate?oauth_token="+REQUEST_TOKEN)
}
If I put a Logger inside the brackets it shows the log in my terminal window. But that Redirect seems to just be ignored. Never goes off.
You haven't included the oauth_callback parameter which is required. See the documentation here.