Writing gradle plugin with nested extension objects - plugins

I am working on a gradle plugin (i dont know groovy, and just started using gradle) that integrates with Asgard (from netflix). I want my extension object to look like this for the user:
asgard {
url = "http://asgard"
regions {
"us-east-1" {
autoScaling {
{
devPhase = "test"
min = 3
max = 6
availabilityZones = ["us-east-1a", "us-east-1b", "us-east-1c"]
ami = "Base AMI 2013-07-11"
instanceType = "m3.xlarge"
securityGroups = ["base", "test-application"]
}
}
}
}
}
Or something close to that. I have been able to get close to this by making autoScaling a List, but when ever i try to get a property from that class it seems to return a dynamic property and never the value. Here is the below starter plugin:
import org.gradle.internal.reflect.Instantiator
class AsgardPlugin implements Plugin<Project> {
void apply(Project project) {
project.extensions.create("asgard", AsgardPluginExtension, project)
project.asgard.extensions.regions = project.container(AsgardRegion) {String name ->
AsgardRegion region = project.gradle.services.get(Instantiator).newInstance(AsgardRegion, name)
assert region instanceof ExtensionAware
region.extensions.add("autoScaling", project.container(AsgardAutoScaling))
return region
}
project.task('displayConfigs') << {
if(project.asgard.applicationName == null) project.asgard.applicationName = project.name
println "Asgard URL $project.asgard.url"
println "Application name $project.asgard.applicationName"
println "Runs on regions..."
project.asgard.regions.each() {region ->
println "\tRegion $region.name"
println "\tAutoScaling groups..."
region.autoScaling.each() {asg ->
println "\t\tdevPhase $asg"
println "\t\tdevPhase $asg.devPhase"
println "\t\tdevPhase $asg.get('devPhase')"
}
}
}
}
}
class AsgardPluginExtension {
String url = "http://asgard.sisa.samsung.com"
String applicationName
AsgardPluginExtension(Project project) {
applicationName = project.name
}
}
class AsgardRegion {
def String name
List<AsgardAutoScaling> autoScaling
AsgardRegion(String name) { this.name = name }
}
class AsgardAutoScaling {
String devPhase
int min
int max
List<String> availabilityZones
String ami
String instanceType
String sshKey
List<String> securityGroups
}
When I run the task, this is what I see in the logs:
$ ./gradlew displayConfigs
:displayConfigs
Asgard URL http://asgard
Application name gradle-asgard-plugin
Runs on regions...
Region us-east-1
AutoScaling groups...
Creating properties on demand (a.k.a. dynamic properties) has been deprecated and is scheduled to be removed in Gradle 2.0. Please read http://gradle.org/docs/current/dsl/org.gradle.api.plugins.ExtraPropertiesExtension.html for information on the replacement for dynamic properties.
Deprecated dynamic property: "devPhase" on "AsgardRegion_Decorated#65087be0", value: "test".
Deprecated dynamic property: "min" on "AsgardRegion_Decorated#65087be0", value: "3".
Deprecated dynamic property: "max" on "AsgardRegion_Decorated#65087be0", value: "6".
Deprecated dynamic property: "availabilityZones" on "AsgardRegion_Decorated#65087be0", value: "[us-east-1a, us-east-1...".
Deprecated dynamic property: "ami" on "AsgardRegion_Decorated#65087be0", value: "Base AMI 2013-07-11".
Deprecated dynamic property: "instanceType" on "AsgardRegion_Decorated#65087be0", value: "m3.xlarge".
Deprecated dynamic property: "securityGroups" on "AsgardRegion_Decorated#65087be0", value: "[base, test-application]".
devPhase
devPhase test
devPhase AsgardRegion_Decorated#14860315('devPhase')
BUILD SUCCESSFUL
Total time: 1.996 secs
Am I going about this all wrong? If not how can I get the value from the dynamic property (tried get and value, but those don't seem to work).
Thanks for your time reading this.
EDIT based off #peter-niederwieser feedback
Thanks peter-niederwieser for your hints. After taking your advice and looking at how the closures work I think i have a better understanding of how to set out to do what I wanted to try out.
Here is the syntax of the plugin now:
asgard {
url = "http://asgard"
regions {
"us-east-1" {
autoScaling {
devPhase = "test"
min = 3
max = 6
availabilityZones = ["a", "b", "c"]
ami = "Base AMI 2013-07-11"
instanceType = "m3.xlarge"
securityGroups = ["base", "test-application"]
}
autoScaling {
devPhase = "stage"
}
}
"us-west-1" {
autoScaling {
devPhase = "test"
}
autoScaling {
devPhase = "stage"
}
}
}
}
The code to support this can be found here:
import org.gradle.internal.reflect.Instantiator
class AsgardPlugin implements Plugin<Project> {
void apply(Project project) {
project.extensions.create("asgard", AsgardPluginExtension, project)
project.asgard.extensions.regions = project.container(AsgardRegion) {String name ->
AsgardRegion region = project.gradle.services.get(Instantiator).newInstance(AsgardRegion, name, project)
assert region instanceof ExtensionAware
return region
}
project.task('displayConfigs') << {
if(project.asgard.applicationName == null) project.asgard.applicationName = project.name
println "Asgard URL $project.asgard.url"
println "Application name $project.asgard.applicationName"
println "Runs on regions..."
project.asgard.regions.each() {region ->
println "\tRegion $region.name"
println "\tAutoScaling groups..."
region.autoScaling.each() {asg ->
println "\t\tdevPhase $asg.devPhase"
}
}
}
}
}
class AsgardPluginExtension {
String urlgg
String applicationName
AsgardPluginExtension(Project project) {
applicationName = project.name
}
}
class AsgardRegion {
String name
Project project
List<AsgardAutoScaling> autoScaling = []
AsgardRegion(String name, Project project) {
this.name = name
this.project = project
}
void autoScaling(Closure closure) {
def asg = new AsgardAutoScaling()
project.configure(asg, closure)
autoScaling.add(asg)
}
}
class AsgardAutoScaling {
String devPhase
int min
int max
List<String> availabilityZones
String ami
String instanceType
String sshKey
List<String> securityGroups
}
Here is the output I see when I run:
$ ./gradlew displayConfigs
:displayConfigs
Asgard URL http://asgard
Application name gradle-asgard-plugin
Runs on regions...
Region us-east-1
AutoScaling groups...
devPhase test
devPhase stage
Region us-west-1
AutoScaling groups...
devPhase test
devPhase stage
BUILD SUCCESSFUL
Total time: 1.929 secs

Some hints:
There is no need to add an autoScaling extension to region because the AsgardRegion class already has an autoScaling property. You just need to initialize that property and perhaps add a void autoScaling(Closure closure) { project.configure(autoScaling, closure) } convenience method to support the autoScaling { ... } syntax.
project.container creates a NamedDomainObjectContainer. The literal syntax for this container is someName { ... }; otherName { ... }. There is no "plain list" syntax.
Your test code sets devPhase etc. for the autoScaling container, which doesn't have these properties. That's why you get the dynamic properties warnings. The innermost { ... } is a block (not a closure) and is redundant.
To solve the list problem, either define a name property for AsgardAutoScaling and use the regular NamedDomainObjectContainer literal syntax, or don't use NamedDomainObjectContainer and implement your own syntax. For example, you could declare an autoScaling method on AsgardRegion that accepts a closure, creates an instance of AsgardAutoScaling, and adds it to the list.

The previous answer helped me a lot to resolve my issue. I wanted to add nested extension objects into the build.gradle file. Like this
family {
father {
firstName = "Ivan"
lastName = "Karamazon"
}
children {
son {
firstName = "Alesha"
lastName = "Rakitka"
}
}
}
Finally, I was able to do that and created the sample project. However, the requirements were slightly different so sharing my experience here. For the details see please SO answer https://stackoverflow.com/a/41491729/3223198 and the Github project https://github.com/andriipanasiuk/family-gradle-plugin

Related

How to import multiple terraform resource instances?

I am trying to import already existing resources, these cannot be recreated, and there are many.
There is configuration that is basic to all resources and for some there are small changes.
I would like to import all the resources with a single command, doing it one by one is tedious and prone to mistakes.
Currently importing single resources with:
terraform import 'github_repository.repo_config["repo2"]' repo2
What would the import command look like if it were to import all of the resources?
The configuration is as follows:
terraform {
required_providers {
github = {
source = "integrations/github"
version = "~> 5.0"
}
}
}
provider "github" {
owner = "medecau"
}
variable "repo_config" {
type = map(object({
description = string
homepage_url = string
topics = list(string)
}))
default = {
"repo1" = {
description = "Repo 1"
homepage_url = "https://medecau.github.io/repo1/"
topics = ["topic1", "topic2", "topic3"]
}
"repo2" = {
description = "Repo 2"
homepage_url = null
topics = null
}
}
}
variable "default_repo_config" {
type = object({
description = string
homepage_url = string
topics = list(string)
})
default = {
description = ""
homepage_url = ""
topics = []
}
}
data "github_repositories" "medecau_repos" {
query = "user:medecau"
include_repo_id = true
}
resource "github_repository" "repo_config" {
# cast to set to remove duplicates
for_each = toset(data.github_repositories.medecau_repos.names)
name = each.value
description = lookup(var.repo_config, each.value, var.default_repo_config).description
homepage_url = lookup(var.repo_config, each.value, var.default_repo_config).homepage_url
topics = lookup(var.repo_config, each.value, var.default_repo_config).topics
has_issues = true
has_projects = false
has_wiki = false
vulnerability_alerts = true
security_and_analysis {
advanced_security {
status = "enabled"
}
secret_scanning {
status = "enabled"
}
secret_scanning_push_protection {
status = "enabled"
}
}
}
Importing multiple resources with single terraform import .. call is not supported natively as of now by terraform.
However terraformer could be something which can reduce your effort. Please go through the Terraformer Github documentation to verify if this works for you.
Note: Supports only organizational resources.

Inner destructured property not received by ELK stack as destructured, but as escaped string

I'm not sure if this is a Serilog, ELK, Service Fabric, code or config issue.
I am writing a Service Fabric Stateless Service. My logging configuration line is this:
Logger = new LoggerConfiguration()
.WriteTo.EventFlow(loggingOptions.DiagnosticPipeline)
.Destructure.With<JsonNetDestructuringPolicy>()
.Enrich.FromLogContext()
.CreateLogger()
.ForContext(properties);
...where properties is an array of PropertyEnricher and the Policy is from Destructurama's JsonNetDestructuringPolicy.
I have a custom object base class that is successfully being destructured, so if I call it Data, then in ELK's JSON tab I see:
"payload": {
"Data": {
"Property1": "Prop Value",
"Property2": "Prop2 Value"
}
}
However, when one of the inner objects is also destructured, it is sent as an escaped JSON string instead of being destructured, with no quotes around the property names:
"payload": {
"Data": {
"Property1": "Prop Value",
"DestructuredProp": "{InnerProperty: \"Inner Value\"}"
}
}
What I expected was:
"payload": {
"Data": {
"Property1": "Prop Value",
"DestructuredProp": {
"InnerProperty": "Inner Value"
}
}
}
I don't know why the inner property names are not given quotes, or why the entire value is being escaped and quoted instead of destructured.
I have verified that my destructuring code is being executed. I can manually add quotes around the property name, for example, but it just results in more escaped quotes in the inner value.
My own code was destructuring it directly from C#. I thought it might have been a bug in my destructuring code, so I looked around some more and found Destructurama's JsonNetDestructuringPolicy, so I tried that, converting my object with JObject.fromObject(), but the same thing happens with it.
I'm pretty sure I should be able to do this with Serilog. I don't think there would be a depth limit setting if it couldn't do more than one layer deep. Why doesn't this work? I have tried refreshing the field index in Kibana, but the JSON view shows the escaped string, so I'm pretty sure it is being sent incorrectly, and isn't an ELK issue.
---EDIT---
Here is a destructuring policy I tried. My initial object is JsonEvent, and it has a Dictionary that is not destructuring, even though the Dictionayr policy is successfully being invoked.
public class JsonEventDestructuringPolicy : IDestructuringPolicy
{
public bool TryDestructure(object value, ILogEventPropertyValueFactory propertyValueFactory, out LogEventPropertyValue result)
{
if (value is JsonEvent jsonEvent)
{
var properties = new List<LogEventProperty>();
foreach (var property in value.GetType().GetProperties())
{
var propertyValue = property.GetValue(value);
var isCollection = propertyValue is ICollection<Dictionary<string,string>>;
var isDictionary = propertyValue is Dictionary<string,string>;
if (isCollection)
LoggingContext.Message("Found collection of dictionary: " + property.Name);
else if (isDictionary)
LoggingContext.Message("Found dictionary: " + property.Name);
else if (property.Name.Equals("Parameters"))
LoggingContext.Message("Found Parameters: " + propertyValue.GetType());
if (propertyValue != null)
properties.Add(new LogEventProperty(property.Name, propertyValueFactory.CreatePropertyValue(propertyValue, isCollection || isDictionary)));
}
result = new StructureValue(properties);
return true;
}
if (value is Dictionary<string, string> dictionary)
{
var properties = new List<LogEventProperty>();
foreach (var kvp in dictionary)
{
if (!string.IsNullOrWhiteSpace(kvp.Value))
properties.Add(new LogEventProperty("\"" + kvp.Key + "\"", propertyValueFactory.CreatePropertyValue(kvp.Value)));
}
result = new StructureValue(properties);
return true;
}
result = null;
return false;
}
}
It is being invoked like this:
public static void Message(JsonEvent message)
{
Logger.ForContext(GetEnrichers(message))
.Information(message.Event);
}
private static IEnumerable<ILogEventEnricher> GetEnrichers(JsonEvent message)
{
return new List<ILogEventEnricher>()
.Add("Data", message, true)
.Add("CorrelationId", ServiceTracingContext.CorrelationId)
.Add("CorrelationDateTime", ServiceTracingContext.CorrelationDateTime)
.Add("RouteTemplate", ServiceTracingContext.RouteTemplate)
.ToArray();
}

Grails: how to set a parameter for the whole group in UrlMappings

Our Grails 2.5 web application has REST API.
We need to start a new version of the API so a group was created in UrlMappings.groovy like this:
group ("/v2") {
"/documents"(resources: 'document') {
"/meta"(resource: 'documentInfo', includes: ['show', 'update'])
}
"/folders"(resources: 'folder')
apiVersion = 2
}
The thing is that parameter apiVersion is not defined in params in controller actions.
The parameter is properly set if it is defined in each of the resources like this:
group ("/v2") {
"/documents"(resources: 'document') {
"/meta"(resource: 'documentInfo', includes: ['show', 'update']) {
apiVersion = 2
}
apiVersion = 2
}
"/folders"(resources: 'folder') {
apiVersion = 2
}
}
How can I properly define a parameter on the group level?
Workaround will be using namespace... although my attempt to define it at group level wont work but you can give a try. But defining at resource level works even for nested resources.
group ("/v2") {
"/documents"(resources: 'document', namespace:'v2') {
"/meta"(resource: 'documentInfo', includes: ['show', 'update'])
}
"/folders"(resources: 'folder', , namespace:'v2')
}

Resolve a standalone string/file using typesafe config

I'm looking for a way to say:
val c: Config = ConfigFactory.parseString("a=fox,b=dog")
val s: String = """This is a "quick" brown ${a}.\nThat is a lazy, lazy ${b}."""
println(c.resolveString(s))
// Should print:
// > This is a "quick" brown fox.
// > That is a lazy lazy dog.
My two ideas:
Just find the placeholders with regex and replace from config one by one
convert s to config with single value and use resolveWith - but it seems quoting can be really tricky
Maybe there is an easier way?
A naive solution:
class Resolver(vars: Config) {
private lazy val placeholderRegex = "(?<=\\$\\{).*?(?=\\})".r
def resolveString(s: String): String = {
placeholderRegex.findAllIn(s).foldLeft(s) { (str, v) =>
if (vars.hasPath(v)) str.replaceAll("\\Q${" + v + "}\\E", vars.getString(v)) else str
}
}
It should be fine if the string is not huge and there are no insane numbers of distinct placeholders in it.
I have a similar situation where I automatically push elastic search index definitions before
I launch the job which uses those index definitions.
In my case the string which contains variable references is the
JSON index/template definition, which ALSO comes from the typesafe config.
(see es.template_or_index.json, below.)
I resolve these refs using the utility method I wrote on top
of apache.commons StrSubstitutor.
(see: VariableReferenceResolver.resolveReferences(String template), below)
Below is a sample of my config. Note how the property 'es.animal' is injected into the index/template json.
This can't be done with out-of-the-box features of the typesafe config library (but I think this would
be a wonderful feature for them to add !)
es {
// user and password credentials should not be checked in to git. deployer is expected to
// set these parameters into the their environment in whatever way is convenient -- .bashrc or whatever.
user = dummy
user = ${?ES_USER}
password = dummy.passwd
password = ${?ES_PASSWORD}
hostPort = "localhost:9200"
hostPort = ${?ES_HOST_PORT}
protocol = http
protocol = ${?ES_PROTOCOL}
animal = horse // This gets injected into the configuration property es.template_or_index.json
// Note: for template json there is a second round of interpolation performed so the json can reference any defined
// property of this configuration file (or anything it includes)
template_or_index {
name = test_template
json = """
{
"template": "${es.animal}_sql-*",
"settings": {
"number_of_shards": 50,
"number_of_replicas": 2
},
"mappings": {
"test_results" : {
"date_detection": false,
"properties" : {
"timestamp" : { "type" : "date"},
"yyyymmdd" : { "type" : "string", "index" : "not_analyzed"}
}
}
}
}
"""
}
}
package com.foo
import com.typesafe.config.Config;
import org.apache.commons.lang.text.StrLookup;
import org.apache.commons.lang.text.StrSubstitutor;
public class VariableReferenceResolver {
final StrSubstitutor substitutor;
static class ConfigStrLookup extends StrLookup {
private final Config config;
ConfigStrLookup(Config config) {
this.config = config;
}
public String lookup(String key) {
return config.getString(key);
}
}
public VariableReferenceResolver (Config config) {
substitutor=new StrSubstitutor(new ConfigStrLookup(config));
}
public String resolveReferences(String template) {
return substitutor.replace(template);
}
}
public class OtherClass {
private static void getIndexConfiguration(String path) throws IOException {
System.setProperty("config.file", path);
Config config = ConfigFactory.load();
String user = config.getString("es.user");
String password = config.getString("es.password");
String protocol = config.getString("es.protocol");
String hostPort = config.getString("es.hostPort");
String indexOrTemplateJson = config.getString("es.template_or_index.json");
String indexOrTemplateName = config.getString("es.template_or_index.name");
VariableReferenceResolver resolver = new VariableReferenceResolver(config);
String resolvedIndexOrTemplateJson = resolver.resolveReferences(indexOrTemplateJson);
File jsonFile = File.createTempFile("index-or-template-json", indexOrTemplateName);
Files.write(Paths.get(jsonFile.getAbsolutePath()), resolvedIndexOrTemplateJson.getBytes());
curlIndexOrTemplateCreateCommand =
String.format(
"curl -XPUT -k -u %s:%s %s://%s/_template/%s -d #%s",
user, password, protocol, hostPort, indexOrTemplateName, jsonFile.getAbsolutePath());
}
....
}

Copy Groovy class properties

I want to copy object properties to another object in a generic way (if a property exists on target object, I copy it from the source object).
My code works fine using ExpandoMetaClass, but I don't like the solution. Are there any other ways to do this?
class User {
String name = 'Arturo'
String city = 'Madrid'
Integer age = 27
}
class AdminUser {
String name
String city
Integer age
}
def copyProperties(source, target) {
target.properties.each { key, value ->
if (source.metaClass.hasProperty(source, key) && key != 'class' && key != 'metaClass') {
target.setProperty(key, source.metaClass.getProperty(source, key))
}
}
}
def (user, adminUser) = [new User(), new AdminUser()]
assert adminUser.name == null
assert adminUser.city == null
assert adminUser.age == null
copyProperties(user, adminUser)
assert adminUser.name == 'Arturo'
assert adminUser.city == 'Madrid'
assert adminUser.age == 27
I think the best and clear way is to use InvokerHelper.setProperties method
Example:
import groovy.transform.ToString
import org.codehaus.groovy.runtime.InvokerHelper
#ToString
class User {
String name = 'Arturo'
String city = 'Madrid'
Integer age = 27
}
#ToString
class AdminUser {
String name
String city
Integer age
}
def user = new User()
def adminUser = new AdminUser()
println "before: $user $adminUser"
InvokerHelper.setProperties(adminUser, user.properties)
println "after : $user $adminUser"
Output:
before: User(Arturo, Madrid, 27) AdminUser(null, null, null)
after : User(Arturo, Madrid, 27) AdminUser(Arturo, Madrid, 27)
Note: If you want more readability you can use category
use(InvokerHelper) {
adminUser.setProperties(user.properties)
}
I think your solution is quite good and is in the right track. At least I find it quite understandable.
A more succint version of that solution could be...
def copyProperties(source, target) {
source.properties.each { key, value ->
if (target.hasProperty(key) && !(key in ['class', 'metaClass']))
target[key] = value
}
}
... but it's not fundamentally different. I'm iterating over the source properties so I can then use the values to assign to the target :). It may be less robust than your original solution though, as I think it would break if the target object defines a getAt(String) method.
If you want to get fancy, you might do something like this:
def copyProperties(source, target) {
def (sProps, tProps) = [source, target]*.properties*.keySet()
def commonProps = sProps.intersect(tProps) - ['class', 'metaClass']
commonProps.each { target[it] = source[it] }
}
Basically, it first computes the common properties between the two objects and then copies them. It also works, but I think the first one is more straightforward and easier to understand :)
Sometimes less is more.
Another way is to do:
def copyProperties( source, target ) {
[source,target]*.getClass().declaredFields*.grep { !it.synthetic }.name.with { a, b ->
a.intersect( b ).each {
target."$it" = source."$it"
}
}
}
Which gets the common properties (that are not synthetic fields), and then assigns them to the target
You could also (using this method) do something like:
def user = new User()
def propCopy( src, clazz ) {
[src.getClass(), clazz].declaredFields*.grep { !it.synthetic }.name.with { a, b ->
clazz.newInstance().with { tgt ->
a.intersect( b ).each {
tgt[ it ] = src[ it ]
}
tgt
}
}
}
def admin = propCopy( user, AdminUser )
assert admin.name == 'Arturo'
assert admin.city == 'Madrid'
assert admin.age == 27
So you pass the method an object to copy the properties from, and the class of the returned object. The method then creates a new instance of this class (assuming a no-args constructor), sets the properties and returns it.
Edit 2
Assuming these are Groovy classes, you can invoke the Map constructor and set all the common properties like so:
def propCopy( src, clazz ) {
[src.getClass(), clazz].declaredFields*.grep { !it.synthetic }.name.with { a, b ->
clazz.metaClass.invokeConstructor( a.intersect( b ).collectEntries { [ (it):src[ it ] ] } )
}
}
Spring BeanUtils.copyProperties will work even if source/target classes are different types. http://docs.spring.io/autorepo/docs/spring/3.2.3.RELEASE/javadoc-api/org/springframework/beans/BeanUtils.html