How do you list out all output / generate files from a Gradle Build? - github

Currently I am using the following to generate list of output files which I want for another plugin which does Github releases.
[
"${buildDir}/libs/${project.name}-${project.version}.jar",
"${buildDir}/libs/${project.name}-${project.version}-javadoc.jar",
"${buildDir}/libs/${project.name}-${project.version}-sources.jar"
]
I am not very satisfied with this as I am manually specifying each type of file (doc, source, class). I am looking to make this more generic where I can reuse without any modification for any project which might have more or less outputs and also may not be in the default location and some outputs may not even be Java outputs. I.e. changing output location, the tasks to run or plugging elsewhere in the script should not break this part of the script and should be reusable without editing. The question is how do you get a list of all the outputs as a List of String like above.
More detailed code snippet:
...
plugins {
id 'co.riiid.gradle' version '0.4.2'
}
...
ext {
org = 'MicroTrader'
repo = org + '/' + project.name
releaseNotesFile = 'RELEASE_NOTES.md'
host = 'github.com'
hostUrl = 'https://' + host + '/'
orgUrl = hostUrl + org
websiteUrl = hostUrl + repo
vcsUrl = websiteUrl + '.git'
scmUrl = 'scm:git#' + host + ':' + repo + '.git'
issueTrackerUrl = websiteUrl + '/issues'
ossrhUsername = ((project.hasProperty('ossrhUsername') ? ossrhUsername : System.getenv('OSSRH_USER')) ?: System.getProperty('ossrh.user')) ?: ' '
ossrhPassword = ((project.hasProperty('ossrhPassword') ? ossrhPassword : System.getenv('OSSRH_PASSWORD')) ?: System.getProperty('ossrh.password')) ?: ' '
bintray_user = ((project.hasProperty('bintray_user') ? bintray_user : System.getenv('BINTRAY_USER')) ?: System.getProperty('bintray.user')) ?: ' '
bintray_api_key = ((project.hasProperty('bintray_api_key') ? bintray_api_key : System.getenv('BINTRAY_KEY')) ?: System.getProperty('bintray.apikey')) ?: ' '
artifactory_user = ((project.hasProperty('artifactory_user') ? artifactory_user : System.getenv('ARTIFACTORY_USER')) ?: System.getProperty('artifactory.user')) ?: ' '
artifactory_password = ((project.hasProperty('artifactory_password') ? artifactory_password : System.getenv('ARTIFACTORY_PASSWORD')) ?: System.getProperty('artifactory.password')) ?: ' '
github_api_key = ((project.hasProperty('github_api_key') ? github_api_key : System.getenv('GITHUB_KEY')) ?: System.getProperty('github.apikey')) ?: ' '
}
...
github {
owner = project.org
repo = project.name
token = project.github_api_key
tagName = project.version
targetCommitish = 'master'
name = 'v' + project.version
body = Files.exists(Paths.get('' + projectDir + '/' + releaseNotesFile)) ?
new File('' + projectDir + '/' + releaseNotesFile).text :
(Files.exists(Paths.get('' + baseDir + '/' + releaseNotesFile)) ?
new File('' + baseDir + '/' + releaseNotesFile).text : '')
assets = [
"${buildDir}/libs/${project.name}-${project.version}.jar",
"${buildDir}/libs/${project.name}-${project.version}-javadoc.jar",
"${buildDir}/libs/${project.name}-${project.version}-sources.jar"
]
}
...

Try reading the outputs property on all of the tasks in your build used to generate the artifacts:
project(':my-sub-project').jar.outputs
Another option is to use the archives configuration from all of your sub-projects:
project(':my-sub-project').configurations.archives.allArtifacts

Related

Selecting sql data with multiple rows and exporting as xml file

I have this JavaScript in my transformer
var dbConn;
try {
dbConn = DatabaseConnectionFactory.createDatabaseConnection($gc('DataWarehouseXMLDataConfig').sqlDriver,
$gc('DataWarehouseXMLDataConfig').sqlConnString,
$gc('DataWarehouseXMLDataConfig').sqlUsername,
$gc('DataWarehouseXMLDataConfig').sqlPassword);
var data = dbConn.executeCachedQuery('EXEC [dbo].[GetDataWarehouseData]');
var xml = "";
var xml2 = "";
var row = "";
while(data.next()) {
row = '<referral>' + '\n' +
'<careid>' + data.getString('careid') + '</careid>' + '\n' +
'<patientid>' + data.getString('patientid') + '</patientid>' + '\n' +
'<dateofreceipt>' + data.getString('dateofreceipt') + '</dateofreceipt>' + '\n' +
'<datefirstseen>' + data.getString('datefirstseen') + '</datefirstseen>' + '\n' +
'<nhsnumber>' + data.getString('nhsnumber') + '</nhsnumber>' + '\n' +
'<hospitalnumber>' + data.getString('hospitalnumber') + '</hospitalnumber>' + '\n' +
'<surname>' + data.getString('surname') + '</surname>' + '\n' +
'<forename>' + data.getString('forename') + '</forename>' + '\n' +
'<dateofbirth>' + data.getString('dateofbirth') + '</dateofbirth>' + '\n' +
'</referral>';
xml = xml + row;
xml2 = '<?xml version="1.0" encoding="utf-8"?>' + '\n' + '<results>' + xml + '</results>';
}
channelMap.put('xml2', xml2);
} finally {
if (dbConn) {
dbConn.close();
}
}
I'm using this outbound message template
<referral>
<careid></careid>
<patientid></patientid>
<dateofreceipt></dateofreceipt>
<datefirstseen></datefirstseen>
<nhsnumber></nhsnumber>
<hospitalnumber></hospitalnumber>
<surname></surname>
<forename></forename>
<dateofbirth></dateofbirth>
</referral>
I have a File Writer destination and the File Name has a .xml extension.
My template in the File Writer is ${XmlUtil.prettyPrint(${message.encodedData})}.
The SQL data is being retrieved ok.
I don't think I'm a million miles away but I'm doing something fundamentally wrong.
What do I need to do to send the required xml file please?
I fixed this by adding a mapping step to the source transformer and using JavaScript, like
tmp['referral']['dateofbirth'] = msg['dateofbirth'].toString();

Flink SQL Client connect to secured kafka cluster

I want to execute a query on Flink SQL Table backed by kafka topic of secured kafka cluster. I'm able to execute the query programmatically but unable to do the same through Flink SQL client. I'm not sure on how to pass JAAS config (java.security.auth.login.config) and other system properties through Flink SQL client.
Flink SQL query programmatically
private static void simpleExec_auth() {
// Create the execution environment.
final EnvironmentSettings settings = EnvironmentSettings.newInstance()
.inStreamingMode()
.withBuiltInCatalogName(
"default_catalog")
.withBuiltInDatabaseName(
"default_database")
.build();
System.setProperty("java.security.auth.login.config","client_jaas.conf");
System.setProperty("sun.security.jgss.native", "true");
System.setProperty("sun.security.jgss.lib", "/usr/libexec/libgsswrap.so");
System.setProperty("javax.security.auth.useSubjectCredsOnly","false");
TableEnvironment tableEnvironment = TableEnvironment.create(settings);
String createQuery = "CREATE TABLE test_flink11 ( " + "`keyid` STRING, " + "`id` STRING, "
+ "`name` STRING, " + "`age` INT, " + "`color` STRING, " + "`rowtime` TIMESTAMP(3) METADATA FROM 'timestamp', " + "`proctime` AS PROCTIME(), " + "`address` STRING) " + "WITH ( "
+ "'connector' = 'kafka', "
+ "'topic' = 'test_flink10', "
+ "'scan.startup.mode' = 'latest-offset', "
+ "'properties.bootstrap.servers' = 'kafka01.nyc.com:9092', "
+ "'value.format' = 'avro-confluent', "
+ "'key.format' = 'avro-confluent', "
+ "'key.fields' = 'keyid', "
+ "'value.fields-include' = 'EXCEPT_KEY', "
+ "'properties.security.protocol' = 'SASL_PLAINTEXT', 'properties.sasl.kerberos.service.name' = 'kafka', 'properties.sasl.kerberos.kinit.cmd' = '/usr/local/bin/skinit --quiet', 'properties.sasl.mechanism' = 'GSSAPI', "
+ "'key.avro-confluent.schema-registry.url' = 'http://kafka-schema-registry:5037', "
+ "'key.avro-confluent.schema-registry.subject' = 'test_flink6', "
+ "'value.avro-confluent.schema-registry.url' = 'http://kafka-schema-registry:5037', "
+ "'value.avro-confluent.schema-registry.subject' = 'test_flink4')";
System.out.println(createQuery);
tableEnvironment.executeSql(createQuery);
TableResult result = tableEnvironment
.executeSql("SELECT name,rowtime FROM test_flink11");
result.print();
}
This is working fine.
Flink SQL query through SQL client
Running this giving the following error.
Flink SQL> CREATE TABLE test_flink11 (`keyid` STRING,`id` STRING,`name` STRING,`address` STRING,`age` INT,`color` STRING) WITH('connector' = 'kafka', 'topic' = 'test_flink10','scan.startup.mode' = 'earliest-offset','properties.bootstrap.servers' = 'kafka01.nyc.com:9092','value.format' = 'avro-confluent','key.format' = 'avro-confluent','key.fields' = 'keyid', 'value.avro-confluent.schema-registry.url' = 'http://kafka-schema-registry:5037', 'value.avro-confluent.schema-registry.subject' = 'test_flink4', 'value.fields-include' = 'EXCEPT_KEY', 'key.avro-confluent.schema-registry.url' = 'http://kafka-schema-registry:5037', 'key.avro-confluent.schema-registry.subject' = 'test_flink6', 'properties.security.protocol' = 'SASL_PLAINTEXT', 'properties.sasl.kerberos.service.name' = 'kafka', 'properties.sasl.kerberos.kinit.cmd' = '/usr/local/bin/skinit --quiet', 'properties.sasl.mechanism' = 'GSSAPI');
Flink SQL> select * from test_flink11;
[ERROR] Could not execute SQL statement. Reason:
java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is /tmp/jaas-6309821891889949793.conf
There is nothing in /tmp/jaas-6309821891889949793.conf except the following comment
# We are using this file as an workaround for the Kafka and ZK SASL implementation
# since they explicitly look for java.security.auth.login.config property
# Please do not edit/delete this file - See FLINK-3929
SQL client run command
bin/sql-client.sh embedded --jar flink-sql-connector-kafka_2.11-1.12.0.jar --jar flink-sql-avro-confluent-registry-1.12.0.jar
Flink cluster command
bin/start-cluster.sh
How to pass this java.security.auth.login.config and other system properties (that I'm setting in the above java code snippet), for SQL client?
flink-conf.yaml
security.kerberos.login.use-ticket-cache: true
security.kerberos.login.principal: XXXXX#HADOOP.COM
security.kerberos.login.use-ticket-cache: false
security.kerberos.login.keytab: /path/to/kafka.keytab
security.kerberos.login.principal: XXXX#HADOOP.COM
security.kerberos.login.contexts: Client,KafkaClient
I haven't really tested whether this solution is feasible, you can try it out, hope it will help you.

How to escape curly brackets in powershell

I have this
$content = "loose text{{comments}} = {{yolo}}{{comments}} = {{yolo2}}{{comment}} = {{yolo3}}{{comment}} = {{yolo4}}loose text2";
and I want to escape the brackets. I read somewhere that to escape it, you double them, but even then it is not working.
I get
Cannot convert value "loose text{{comments}} = {{yolo}}{{comments}} = {{yolo2}}{{comment}} = {{yolo3}}{{comment}} = {{yolo4}}loose text2" to type "System.Xml.XmlDocument". Error: "The specified node cannot be inserted as the valid child of this node,
because the specified node is the wrong type."
At line:1 char:1
+ $content = "loose text{{comments}} = {{yolo}}{{comments}} = {{yolo2}}{{comment}} ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : MetadataError: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : RuntimeException
The escape character in PS is the "back-tic" = `. However, in the code presented you should not have any issue just having the {} in your string.
In any case that you need to escape it just use a the sample below:
$content = "loose text`{comments`} = `{yolo`}`{comments`} = `{yolo2`}`{comment`} = `{yolo3`}`{comment`} = `{yolo4`}loose text2"

can't connect to azure file service REST api by python

I want to access my file service by python Request, guided by the list shares page, I'm new to REST and python Request.
my input is
headers= {'x-ms-date': '2018-04-17 06:22:15.181620', 'Authorization': 'SharedKey zedongstorageaccount:codecodecodeFiTzubX9tvC3G3PcDYzR2cX/TMjkOu4JhsvQffS+xTDDBQ==', 'x-ms-version': '2017-07-29'}
url = 'https://zedongstorageaccount.file.core.windows.net/?comp=list'
r=requests.get(url,headers=headers)
But get error, the output of r.content:
b'\xef\xbb\xbf<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:ef98f282-f01a-0042-3e24-d62397000000\nTime:2018-04-17T08:16:21.9635335Z</Message><AuthenticationErrorDetail>The Date header in the request is incorrect.</AuthenticationErrorDetail></Error>'
I have also try the header {'x-ms-date': '2018-04-17 06:22:15.181620', 'Authorization': 'SharedKey zedongnewstorageaccount:NJYYtabOIj5D1R+xb0PpmXrjCldf6NA6oLkYReAKFiTzubX9tvC3G3PcDYzR2cX/TMjkOu4JhsvQffS+xTDDBQ==', 'x-ms-version': '2017-07-29'}, because i don't know if the Authorization and account in the same line. I also tried many version of x-ms-version.
But there all the 400 or 403 respond.
I have read the Authentication for the Azure Storage Services but confused.
Where is the error? And is there case I can learn to build my app by Azure REST API?(I use the keyword to google, all the pages are about build REST API and azure official docs)
I also recommend to use Python SDK but for educational purposes:
import requests
import datetime
import hmac
import hashlib
import base64
storage_account_name = 'teststorageaccount'
storage_account_key = 'D4x6YChpyfqWk9a.......'
api_version = '2016-05-31'
request_time = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
string_params = {
'verb': 'GET',
'Content-Encoding': '',
'Content-Language': '',
'Content-Length': '',
'Content-MD5': '',
'Content-Type': '',
'Date': '',
'If-Modified-Since': '',
'If-Match': '',
'If-None-Match': '',
'If-Unmodified-Since': '',
'Range': '',
'CanonicalizedHeaders': 'x-ms-date:' + request_time + '\nx-ms-version:' + api_version + '\n',
'CanonicalizedResource': '/' + storage_account_name + '/\ncomp:list'
}
string_to_sign = (string_params['verb'] + '\n'
+ string_params['Content-Encoding'] + '\n'
+ string_params['Content-Language'] + '\n'
+ string_params['Content-Length'] + '\n'
+ string_params['Content-MD5'] + '\n'
+ string_params['Content-Type'] + '\n'
+ string_params['Date'] + '\n'
+ string_params['If-Modified-Since'] + '\n'
+ string_params['If-Match'] + '\n'
+ string_params['If-None-Match'] + '\n'
+ string_params['If-Unmodified-Since'] + '\n'
+ string_params['Range'] + '\n'
+ string_params['CanonicalizedHeaders']
+ string_params['CanonicalizedResource'])
signed_string = base64.b64encode(hmac.new(base64.b64decode(storage_account_key), msg=string_to_sign.encode('utf-8'), digestmod=hashlib.sha256).digest()).decode()
headers = {
'x-ms-date' : request_time,
'x-ms-version' : api_version,
'Authorization' : ('SharedKey ' + storage_account_name + ':' + signed_string)
}
url = ('https://' + storage_account_name + '.file.core.windows.net/?comp=list')
r = requests.get(url, headers = headers)
print(r.content)
Output:
<?xml version="1.0" encoding="utf-8"?><EnumerationResults ServiceEndpoint="https://teststorageaccount.file.core.windows.net/"><Shares><Share><Name>myfileshare</Name><Properties><Last-Modified>Tue, 17 Apr 2018 13:43:00 GMT</Last-Modified><Etag>"0x8D5A46922CA8BDA"</Etag><Quota>10</Quota></Properties></Share></Shares><NextMarker /></EnumerationResults>

Error C#.NET 3.0: The best overloaded method match for 'CreatePageSection(ref string, ref string, ref object)' has some invalid arguments

I have got the following problem. I encountered an error regarding the default parameters. I added a simple piece of code for an overload. Now I get in the new code (line 3: CreatePageSection(sItemID, "", null);) gives the error mentioned in the title.
I looked for the answer in the other topics, but I can't find the problem. Can someone help me?
The code is found here:
public void CreatePageSection(ref string sItemID)
{
CreatePageSection(sItemID, "", null);
}
public void CreatePageSection(ref string sItemID, ref string sFrameUrl, ref object vOnReadyState)
{
if (Strings.InStr(msPresentPageSections, "|" + sItemID + "|", 0) > 0) {
return;
}
msPresentPageSections = msPresentPageSections + sItemID + "|";
string writeHtml = "<div class=" + MConstants.QUOTE + "PageSection" + MConstants.QUOTE + " id=" + MConstants.QUOTE + "Section" + sItemID + "Div" + MConstants.QUOTE + " style=" + MConstants.QUOTE + "display: none;" + MConstants.QUOTE + ">";
this.WriteLine_Renamed(ref writeHtml);
//UPGRADE_WARNING: Couldn't resolve default property of object vOnReadyState. Click for more: 'ms-help://MS.VSCC.v90/dv_commoner/local/redirect.htm?keyword="6A50421D-15FE-4896-8A1B-2EC21E9037B2"'
//UPGRADE_NOTE: IsMissing() was changed to IsNothing_Renamed(). Click for more: 'ms-help://MS.VSCC.v90/dv_commoner/local/redirect.htm?keyword="8AE1CB93-37AB-439A-A4FF-BE3B6760BB23"'
writeHtml = " <iframe id=" + MConstants.QUOTE + sItemID + "Frame" + MConstants.QUOTE + " name=" + MConstants.QUOTE + sItemID + "Frame" + MConstants.QUOTE + " frameborder=" + MConstants.QUOTE + "0" + MConstants.QUOTE + (!string.IsNullOrEmpty(sFrameUrl) ? " src=" + MConstants.QUOTE + sFrameUrl + MConstants.QUOTE : "") + ((vOnReadyState == null) ? "" : " onreadystatechange=" + MConstants.QUOTE + Convert.ToString(vOnReadyState) + MConstants.QUOTE) + ">";
this.WriteLine_Renamed(ref writeHtml);
writeHtml = " </iframe>";
this.WriteLine_Renamed(ref writeHtml);
writeHtml = "</div>";
this.WriteLine_Renamed(ref writeHtml);
}
you must pass params by reference
public void CreatePageSection(ref string sItemID)
{
var missingString = String.Empty;
object missingObject = null;
CreatePageSection(ref sItemID, ref missingString, ref missingObject);
}
Since your are not manipulating sFrameUrl and vOnReadyState, remove the ref keyword from those parameters.
See: http://msdn.microsoft.com/en-us/library/14akc2c7(v=vs.71).aspx
hth
Mario