I am new to Concourse
Can someone please help me understand the difference between Resource and Resource Type in simple words?
Apparently, documentation is not helping me.
Think about a resource as an object and a resource type as a class.
Resource types are docker images that can take a bunch of source parameters.
Resources are instantiations of these images with specific parameters.
Resources generate versions which can serve as inputs or outputs of your tasks.
For example, a built-in resource type git can be instantiated into a resource code-repo by taking a specific github URI and branch as params.
From here on out, every time a commit shows up in the code-repo resource it may serve as an input to tasks in a job.
Related
I am wondering what approach in designing serverless functions to take, while taking designing a regular server as a point of reference.
With a traditional server, one would focus on defining collections and then CRUD operations one can run on each of them (HTTP verbs such as GET or POST).
For example, you would have a collection of users, and you can get all records via app.get('/users', ...), get specific one via app.get('/users/{id}', ...) or create one via app.post('/users', ...).
How differently would you approach designing a serverless function? Specifically:
Is there a sense in differentiating between HTTP operations or would you just go with POST? I find it useful to have them defined on the client side, to decide if I want to retry in case of an error (if the operation is idempotent, it will be safe to retry etc.), but it does not seem to matter in the back-end.
Naming. I assume you would use something like getAllUsers() when with a regular server you would define collection of users and then just use GET to specify what you want to do with it.
Size of functions: if you need to do a number of things in the back-end in one step. Would you define a number of small functions, such as lookupUser(), endTrialForUser() (fired if user we got from lookupUser() has been on trial longer than 7 days) etc. and then run them one after another from the client (deciding if trial should be ended on the client - seems quite unsafe), or would you just create a getUser() and then handle all the logic there?
Routing. In serverless functions, we can't really do anything like .../users/${id}/accountData. How would you go around fetching nested data? Would you just return a complete JSON every time?
I have been looking for some comprehensive articles on the matter but no luck. Any suggestions?
This is a very broad question that you've asked. Let me try answering it point by point.
Firstly, the approach that you're talking about here is the Serverless API project approach. You can clone their sample project to get a better understanding of how you can build REST apis for performing CRUD operations. Start by installing the SAM cli and then run the following commands.
$ sam init
Which template source would you like to use?
1 - AWS Quick Start Templates
2 - Custom Template Location
Choice: 1
Cloning from https://github.com/aws/aws-sam-cli-app-templates
Choose an AWS Quick Start application template
1 - Hello World Example
2 - Multi-step workflow
3 - Serverless API
4 - Scheduled task
5 - Standalone function
6 - Data processing
7 - Infrastructure event management
8 - Machine Learning
Template: 3
Which runtime would you like to use?
1 - dotnetcore3.1
2 - nodejs14.x
3 - nodejs12.x
4 - python3.9
5 - python3.8
Runtime: 2
Based on your selections, the only Package type available is Zip.
We will proceed to selecting the Package type as Zip.
Based on your selections, the only dependency manager available is npm.
We will proceed copying the template using npm.
Project name [sam-app]: sample-app
-----------------------
Generating application:
-----------------------
Name: sample-app
Runtime: nodejs14.x
Architectures: x86_64
Dependency Manager: npm
Application Template: quick-start-web
Output Directory: .
Next steps can be found in the README file at ./sample-app/README.md
Commands you can use next
=========================
[*] Create pipeline: cd sample-app && sam pipeline init --bootstrap
[*] Test Function in the Cloud: sam sync --stack-name {stack-name} --watch
Comings to your questions point wise:
Yes, you should differentiate your HTTP operations with their suitable HTTP verbs. This can be configured at the API Gateway and can be checked for in the Lambda code. Check the source code of handlers & the template.yml file from the project you've just cloned with SAM.
// src/handlers/get-by-id.js
if (event.httpMethod !== 'GET') {
throw new Error(`getMethod only accepts GET method, you tried: ${event.httpMethod}`);
}
# template.yml
Events:
Api:
Type: Api
Properties:
Path: /{id}
Method: GET
The naming is totally up to the developer. You can follow the same approach that you're following with your regular server project.
You can define the handler with name getAllUsers or users and then set the path of that resource to GET /users in the AWS API Gateway. You can choose the HTTP verbs of your desire. Check this tutorial out for better understanding.
Again this up to you. You can create a single Lambda that handles all that logic or create individual Lambdas that are triggered one after another by the client based on the response from the previous API. I would say, create a single Lambda and just return the cumulative response to reduce the number of requests. But again, this totally depends on the UI integration. If your screens demand separate API calls, then please, by all means create individual lambdas.
This is not true. We can have dynamic routes specified in the API Gateway.
You can easily set wildcards in your routes by using {variableName} while setting the routes in API Gateway.
GET /users/{userId}
The userId will then be available at your disposal in the lambda function via event.pathParameters.
GET /users/{userId}?a=x
Similarly, you could even pass query strings and access them via event.queryStringParameters in code. Have a look at working with routes.
Tutorial I would recommend for you:
Tutorial: Build a CRUD API with Lambda and DynamoDB
I'm performing an api call in my jenkinsfile that requires specifying a path to file 'A'. Assuming file A is located on the same repo, I am not sure how to refer to file A when running the jenkinsfile.
I feel like this has been done before, but I can't find any resource. Any help is appreciated.
You don't say whether you are using a scripted or declaritive Jenkinsfile, as the details differ. However the principle is the same as far as I am concerned. Basically to do anything with a file you will need to be within a node clause - essentially the controller opens a session on one of the agents and does actions there. You need to checkout your repo on that node:
The scripted Jenkinsfile would look something like (assuming you are not bothered about which node you are running on):
node("") {
checkout scm // "scm" equates to the configuration that the job was run with
// the whole repo will be now available
}
Based on the docs here: https://learn.microsoft.com/en-us/azure/media-services/latest/stream-files-tutorial-with-api
I've created a media services client and am attempting to create the new asset that will have the video file uploaded into it.
When I do this using the infromation provided on the API Access tab of the media service in question, the line: client.Assets.CreateOrUpdateAsync fails with "The resource type is invalid."
Anyone have any idea as to what is causing that and how to fix it? The sample is woefully out of date with credential management and the author is completely non-responsive for over a year.
So I found what this error actually means for anyone that hits this with the same problem:
It's the asset name. It has to conform to Windows file naming standards otherwise you'll get this unhelpful error.
Of course the error message should be updated to "invalid asset name, please ensure that it conforms to windows file naming conventions" but that's a separate bug report to Microsoft.
I just cloned the repo again myself, went to the portal and grabbed the JSON from the API Access blade and replaced the appsettings.json file (hit save), then hit F5 and it is running fine.
Can you try to clean that folder and re-clone the github project again.
git clone https://github.com/Azure-Samples/media-services-v3-dotnet-tutorials.git
Re-grab your credentials and replace them.
Make sure you open vscode from the "UploadEncodeAndStreamFile" folder as well. Seems to be working for me.
Not sure why you would be seeing "resource type is invalid" actually. That sounds like something is named wrong in your appsettings.json file.
Also watch out for Asset Naming conventions, there are character limitations and lengths to be aware of - see here.
https://learn.microsoft.com/en-us/azure/media-services/latest/media-services-apis-overview#naming-conventions
Naming conventions
Azure Media Services v3 resource names (for example, Assets, Jobs, Transforms) are subject to Azure Resource Manager naming constraints. In accordance with Azure Resource Manager, the resource names are always unique. Thus, you can use any unique identifier strings (for example, GUIDs) for your resource names.
Media Services resource names can't include: '<', '>', '%', '&', ':', '', '?', '/', '*', '+', '.', the single quote character, or any control characters. All other characters are allowed. The max length of a resource name is 260 characters.
How is it possible to use Metadata in on_success/on_failure? For example, to send emails via https://github.com/pivotal-cf/email-resource?
I haven't found a way, as I can't change content of files where email resources reside (subject/body), as the metadata is not available to tasks.
And yep, that might be a duplicate for Concourse CI and Build number
But still my question IMHO is a valid use case for notifications.
The metadata you are referring to, I assume, is the environment variables provided to resources, not tasks.
This can be used with the slack resource to provide information about what build failed.
For example:
on_failure:
put: slack-alert
params:
text: |
The `science` pipeline has failed. Please resolve any issues and ensure the pipeline lock was released. Check it out at:
$ATC_EXTERNAL_URL/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME
The email resource, you're referencing has an open PR to support these environment variables. I'd discuss your need for that feature there.
Is it possible to alter a MOF file in DSC after compiling it? I'm trying to have a generic MOF in a pull server and from a client ask for that MOF with specific parameters.
Thanks!
The MOF is just a text file; you could modify it yourself.
But there's no provision in the pull server that would take a parameter and modify the requested MOF on the fly. Additionally you'd have to recalculate the checksum if you did modify it.
What exactly are you trying to do that this would be necessary?
The Guid is used to identify the nodes. If you share the Guid across multiple nodes then it is difficult to identify which node hasn't gotten an update (using the status end point). The correct way of doing it is to use configuration data and compile for the individual nodes. Here is a link to the blog on how to do the same https://blogs.msdn.microsoft.com/powershell/2014/01/09/separating-what-from-where-in-powershell-dsc/