Implemented a Resource Type: How does Concourse use the output of the check, in, and out scripts? - concourse

Reading the Concourse documentation about Implementing a Resource Type, in regards to what the check, in, and out scripts must emit, it is not clear why this output is needed or how Concourse uses it. My questions are:
1) How does Concourse use the output of the check script, the in script, and the out script?
2) And, why is it required that the in and out script emit the version? What happens if you don't?
For context, here is the relevant parts of the documentation:
1) For the check script:
...[it] must print the array of new versions, in chronological order,
to stdout, including the requested version if it's still valid.
For example:
[
{ "ref": "61cbef" },
{ "ref": "d74e01" },
{ "ref": "7154fe" }
]
2) For the in script:
The script must emit the fetched version, and may emit metadata as a list of key-value pairs. This data is intended for public consumption and will make it upstream, intended to be shown on the build's page.
For example:
{
"version": { "ref": "61cebf" },
"metadata": [
{ "name": "commit", "value": "61cebf" },
{ "name": "author", "value": "Hulk Hogan" }
]
}
3) Similar to the in script, the out script:
The script must emit the resulting version of the resource. For
example, the git resource emits the sha of the commit that it just
pushed.
For example:
{
"version": { "ref": "61cebf" },
"metadata": [
{ "name": "commit", "value": "61cebf" },
{ "name": "author", "value": "Mick Foley" }
]
}

Concourse uses the check result to verify if there is any new resource available. According to your pipeline definition, the presence of a new resource would trigger a job. The in is therefore used to read the specific resource using parameters provided by the pipeline whilst the out would take care of writing them.
As your in is going to use the information provided by the check you may want to use a similar structure, but you're not obliged to. It is useful to echo the same version information in your check/in/out in order to be able to log it and understand each resource in your pipeline is belonging to which version.

Related

In TeamCity, trigger build with 2 VCS roots with 2 custom revisions using REST API

looking for collective wisdom on this problem I am tackling.
I need to trigger build using REST API on TeamCity. The build configuration has 2 VCS roots and I need to pass specific revision for both of them. In documentation, I found example for single VCS, but I am stuck on setting revisions for both of them.
I am trying to set up something like this:
{
"buildType": {
"id": "ExampleConfiguration"
},
"defaultBranch": "false",
"lastChanges": {
"change": [
{
"locator": "combination of locators that will set VCS 1 - branch + revision"
},
{
"locator": "combination of locators that will set VCS 2 - branch + revision"
}
]
}
}
In principle, I am simulating execution of the build, where both revisions are manually set - this is possible in the UI.
https://www.jetbrains.com/help/teamcity/rest/start-and-cancel-builds.html#Advanced+Build+Run
TC version: 2022.04.3 (build 108706)
Edit:
I was able to detect the changes and resolve their id on both VCS roots. For the changes field in the body I can now use locator of the form {"id": 123456}. But another obstacle on the road, if I specify pair of these changes:
"lastChanges": { "change": [ {"id": 1}, {"id": 2} ] }
I get error
Responding with error, status code: 400 (Bad Request).
Details: jetbrains.buildServer.server.rest.errors.BadRequestException: Several non-personal changes are submitted, only one can be present
Invalid request. Please check the request URL and data are correct.
Same build with the exact revisions is possible to trigger manually from TC UI.
Finally resolved it. The way to go is to use revisions instead of lastChanges. To pass revisions for 2 VCS roots together with branch names, use this:
"revisions": {
"revision": [
{
"vcs-root-instance": {
"id": "111"
},
"vcsBranchName": "branch-name-111",
"version": "rev_on_vcs_111"
},
{
"vcs-root-instance": {
"id": "222"
},
"vcsBranchName": "branch-name-222",
"version": "rev_on_vcs_222"
}
]
}
To resolve VCS root instance id, use API call /app/rest/vcs-root-instances/vcsRoot:(id:(Project_Vcs_Root))"

Is there a way to stop Autorest.Powershell from flattening response objects?

I have a response object in my swagger.json file that includes a nested object as one of its fields. When I use Autorest.Powershell to generate a client for this API, it flattens the nested object. So when the service returns the following response:
{
"code": 200,
"status": "OK",
"data": {
"FileName": "gameserver.zip",
"AssetUploadUrl": "https://example.com"
}
}
my Autorest.Powershell client returns a flattened object like this:
{
"code": 200,
"status": "OK",
"dataFileName": "gameserver.zip",
"dataAssetUploadUrl": "https://example.com"
}
Is there some sort of configuration setting I can use to disable this behavior?
Here are the relevant portions of my swagger.json file, if it helps:
"definitions": {
"GetAssetUploadUrlResponse": {
"type": "object",
"properties": {
"AssetUploadUrl": {
"description": "The asset's upload URL.",
"type": "string"
},
"FileName": {
"description": "The asset's file name to get the upload URL for.",
"type": "string"
}
},
"example": {
"FileName": "gameserver.zip",
"AssetUploadUrl": "https://example.com"
}
}
},
"responses": {
"GetAssetUploadUrlResponse": {
"description": "",
"schema": {
"type": "object",
"properties": {
"code": {
"type": "integer",
"description": "The Http status code. If X-ReportErrorAsSuccess header is set to true, this will report the actual http error code."
},
"status": {
"type": "string",
"description": "The Http status code as a string."
},
"data": {
"$ref": "#/definitions/GetAssetUploadUrlResponse"
}
},
"example": {
"code": 200,
"status": "OK",
"data": {
"FileName": "gameserver.zip",
"AssetUploadUrl": "https://example.com"
}
}
}
}
}
There are several ways, none of which is really straightforward (as, I'm starting to believe, is the case with most things AutoRest-related; sorry, couldn't resist :-P ).
There are three semi-official ways. Semi-official here means they are based on public AutoRest mechanism but are not themselves documented. Being semi-official, they might only work with certain versions of AutoRest components, so, here are the ones I used
(from autorest --info):
#autorest/core (3.0.6369)
#autorest/modelerfour (4.15.414)
#autorest/powershell (3.0.421)
Finally, here are the relevant parts of AutoRest's code base: inline properties plug-in and configuration directive definition
inlining-threshold setting
This setting control the maximum number of properties an inner object could have for it to be considered eligible for inlining. You can set it either on the command line or in the "literate config" .md file.
```yaml
inlining-threshold: 0
```
In theory, setting this to 0 should prevent any inner member's properties from being inlined, however the plug-in has a hard-coded exception that if the inner object is in a property that's itself named properties then the limit is ignored and it's still flattened.
definitions:
SomeSchema:
type: "object"
properties:
detail_info: # <-- threshold honored
$ref: "#/definitions/InfoSchema"
properties: # <-- this is always flattened because of its special name
$ref: "#/definitions/OtherSchema"
no-inline directive
The PowerShell AutoRest plug-in also defines a custom directive that is used to specify that certain schemas should never be inlined. Using "literate config", it goes like
```yaml
directive:
- no-inline:
- OtherSchema
- ThirdSchema
```
The pros of this approach are that the no-inline directive overrides the "always inline properties in a property named properties" exception mentioned above, so it can be used to alleviate the problem.
The cons are that all schema names should be listed explicitly. (It seems the directive should also support Rx name expression but I couldn't get no-inline: ".*" to work)
Low-level transform
This is approach disables inlining unconditionally in all cases, however it is coupled to the specific internal code model used by AutoRest. (In principle, the model should be stable, at least within major versions). It also relies on the PowerShell plug-in using a specific (non-contractual) property to flag schemas excluded from inlining.
```yaml
directive:
- from: code-model-v4-no-tags
where: $.schemas.objects.*
transform: |
$.language.default['skip-inline'] = true;
```

How to increment a parameter in an Azure Data Factory Until Activity?

I am accessing a RESTful API that pages results in groups of 50 using the HTTP connector. The REST connector doesn't seem to support Client Certificates so I can't use the pagination in that.
I have a Pipeline Variable called SkipIndex that defaults to 0. Inside the Until loop I have a Copy Data Activity that works (HTTP source to BLOB sink), then a Set Variable Activity that I am trying to get to increment this Variable.
{
"name": "Add 50 to SkipIndex",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Copy next to temp",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "SkipIndex",
"value": {
"value": "50++",
"type": "Expression"
}
}
}
Everything I have tried results in errors such as "The expression contains self referencing variable. A variable cannot reference itself in the expression." and the one above with 50++ causes a sink error during debug.
How can I get the Until loop to increment this variable after it retrieves data?
Agree that REST Connector does supports pagination but does not for Client Certificates Authentication type.
For the idea of your Until activity scenario,i am tripped by the can't self-reference a variable in an expression limitation also. Maybe you could make a little trick on that: Add one more variable to persist the index number.
For example,i got 2 variables: count and indexValue
Until Activity:
Inside Until Activity:
V1:
V2:
BTW, no usage of 50++ in ADF.

Linking multiple existing work items to queued build (from Azure Devops SDK)

In Azure Devops REST API, I want to link a task\bug\story (that already exists) when triggering a build. How do I do that?
checkInTicket might be a solution, but it is not documented.
Payload based on Merlin's response worked:
var payload = new object[] {
new {
op = "add",
path = "/relations/-",
value =
new {
rel = "ArtifactLink",
url = $"vstfs:///Build/Build/{buildId}",
attributes = new
{
name = "Build"
}
}
}
};
1. Approach 1
Same with the UI operation, to link the exists work item to build, just need to update one option in build definition:
This is the api:
https://dev.azure.com/{org name}/{project name}/_apis/build/definitions/{definition id}?api-version=5.0-preview.6
In its request body, please focus on below script part:
"options": [
{
"enabled": true,
"inputs": {
"branchFilters": "[\"+refs/heads/*\"]",
"additionalFields": "{}"
},
"definition": {
"id": "5d58cc01-7c75-450c-be18-a388ddb129ec"
}
}
]
The enabled represent whether it will create links to work items which linked to associated changes after the build completed. To achieve what you want, here, please set the enabled value as true. The id value is fixed and represent this option, so here do not worry about this id value.
When you updating the build definition by this method, do not forget increment the revision in request body. revision increment means update the definition as a new revision. or the update action would not actually succeed.
But, what you should concerned is this update should finished before the build triggered. Thus it will create link to the exists WIT automatically after the build finished.
Also, this would only add this build link to work item which associate with the changes.
2. Approach 2
If the above is not what you want, and you just want to link work item to build while the build is triggered. Afraid to say, there's no such direct API could finish that.
You may need to use 2 APIs: one of it is queue build, and another API is add this build link to work item.
The request body sample of add build link to work item:
[
{
"op": "test",
"path": "/rev",
"value": "2"
},
{
"op": "add",
"path": "/relations/-",
"value":
{
"rel": "ArtifactLink",
"url": "vstfs:///Build/Build/{the build id that you just queued}"
}
}
]
This method need you get the generated BuildId after you queue the build, and then pass it to the workitem API. Different with the approach 1, in this method, you can customize the work item id which you want to add the build link to.

Create Entities and training phrases for values in functions for google action

I have created a trivia game using the SDK, it takes user input and then compares it to a value in my DB to see if its correct.
At the moment, I am just passing a raw input variable through my conversation, this means that it regularly fails when it mishears the user since the exact string which was picked up is rarely == to the value in the DB.
Specifically I would like it to only pick up numbers, and for example realise that it must extract '10' , from a speech input of 'my answer is 10'.
{
"actions": [
{
"description": "Default Welcome Intent",
"name": "MAIN",
"fulfillment": {
"conversationName": "welcome"
},
"intent": {
"name": "actions.intent.MAIN"
}
},
{
"description": "response",
"name": "Raw input",
"fulfillment": {
"conversationName": "rawInput"
},
"intent": {
"name": "raw.input",
"parameters": [{
"name": "number",
"type": "org.schema.type.Number"
}],
"trigger": {
"queryPatterns":[
"$org.schema.type.Number:number is the answer",
"$org.schema.type.Number:number",
"My answer is $org.schema.type.Number:number"
]
}
}
}
],
"conversations": {
"welcome": {
"name": "welcome",
"url": "https://us-central1-triviagame",
"fulfillmentApiVersion": 2
},
"rawInput": {
"name": "rawInput",
"url": "https://us-central1-triviagame",
"fulfillmentApiVersion": 2
}
}
}
app.intent('actions.intent.MAIN', (conv) => {
conv.data.answers = answersArr;
conv.data.questions = questionsArr;
conv.data.counter = answersArr.length;
var thisQuestion = conv.data.questions;
conv.ask((conv.data.answers)[0]));
});
app.intent('raw.input', (conv, input) => {
if(input == ((conv.data.answers)[0])){
conv.ask(nextQuestion());
}
app.intent('actions.intent.TEXT', (conv,input) => {
//verifying if input and db value are equal
// at the moment input is equal to 'my number is 10' (for example) instead of '10'
//therefore the string verification never works
conv.ask(nextQuestion());
});
In a previous project i used the dialogflow UI and I used this #system.entities number parameter along with creating some training phrases so it understands different speech patterns.
This input parameter I am passing through my conv , is only a raw string where I'd like it to be filtered using some sort of entity schema.
How do I create the same effect of training phrases/entities using the JSON file?
You can't do this using just the Action SDK. You need a Natural Language Processing system (such as Dialogflow) to handle this as well. The Action SDK, by itself, will do speech-to-text, and will use the actions.json configuration to help shape how to interpret the text. But it will only return the entire text from the user - it will not try to determine how it might match an Intent, nor what parameters may exist in it.
To do that, you need an NLP/NLU system. You don't need to use Dialogflow, but you will need something that does the parsing. Trying to do it with simple pattern matching or regular expressions will lead to nightmares - find a good system to do it.
If you want to stick to things you can edit yourself, Dialogflow does allow you to download its configuration files (they're just JSON), edit them, and update or replace the configuration through the UI or an API.