Ansible github_packages - github

There are two releases that can be obtained from a GitHub repo (Binary Releases and Package Releases) as shown below:
I want to use Ansible to retrieve Package Releases from my GitHub Repo
I did some searching on Ansible docs and found a collection community.general.github_release but this gives the latest Release binaries of the repo and not Package Releases.
Can anyone help if they know a collection that can fetch Package Releases from GitHub ?
Appreciate any help. Thanks

You can use Github GraphQL API (as shown in this question and this one) such as:
#
# Tasks that may be included in an Ansible playbook or role depending on your needs
#
# Some variables to define to identify your repository
# They may be set as playbook or role variables as well
# You'll need a Bearer token (see https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token)
- set_fact:
bearer_token: YOUR_BEARER_TOKEN
repository_name: repository-name
repository_owner: repository-owner
- name: Retrieve packages for repository
uri:
url: https://api.github.com/graphql
method: POST
body: '{"query":
"query { repository(name: \"{{ repository_name }}\", owner: \"{{ repository_owner }}\") {
packages(first:10) { nodes { name, packageType, latestVersion {
version, files(first:100) { nodes { url } }
} } }
}
}"'
headers:
Content-Type: application/json
Accept: "application/vnd.github.packages-preview+json"
Authorization: "bearer {{ bearer_token }}"
register: github_packages_json
This will provide an output like:
{
"json": {
"data": {
"repository": {
"packages": {
"nodes": [
{
"latestVersion": {
"files": {
"nodes": [
{
"url": "https://pkg.githubusercontent.com/xxx/some-url"
},
{
"url": "https://pkg.githubusercontent.com/xxx/another-url"
}
]
},
"version": "my-package-1.2.3"
},
"name": "my-package",
"packageType": "DOCKER"
}
]
}
}
}
},
}
Depending on packageType you may need to perform different action. For example, a DOCKER packageType would require you to pull the image such as:
- name: pull docker
shell: docker pull docker.pkg.github.com/{{ repository_owner | lower }}/{{ repository_name }}/{{ docker_image_name }}:{{ docker_image_version }}
vars:
docker_image_name: "{{ github_packages_json.json.data.repository.packages.nodes[0].name }}"
docker_image_version: "{{ github_packages_json.json.data.repository.packages.nodes[0].latestVersion.version }}"

The community.general.github_release Ansible role is part of the ansible-collections/community.general code.
Its source code source_control/github/github_release.py shows that is is using github3.py, the library for using GitHub's REST API.
Specifically, the latest_release endpoint (code here) uses the GET /repos/{owner}/{repo}/releases/latest REST API.
However, a "Package Release" is, for github3.py (used by the Ansible role), an asset, with an ID you can find in a github3.repos.release.Release: the original_assets will give you all the assets id.
You would therefore need to write a role similar to latest_release, using the version returned by latest_release in order to call github3.repos.release.Release, get the assets ID and download the one you need using asset(asset_id)

Related

Referencing REST API resource from CDK in amplify custom resource

I need to customize the method handler for a REST API endpoint and point it to SQS instead of a Lambda function. I'm stalling out trying to get a full reference to the RestApi object...
The RestApi object I get back from RestApi.fromRestApiId is incomplete; I can't do this:
const restApi = apigateway.RestApi.fromRestApiId(this, 'RestApi', dependencies.api.rest.ApiId);
const queueResource = restApi.root.resourceForPath('/webhooks');
...without getting this error:
Error: root is not configured when imported using fromRestApiId(). Use fromRestApiAttributes() API instead.
I can't use RestApi.fromRestApiAttributes as that requires the rootResourceId -- which I can't seem to find a reference to. The documentation for RestApi.fromRestApiAttributes shows this, but I don't have props:
const api = RestApi.fromRestApiAttributes(this, 'RestApi', {
restApiId: props.restApiId,
rootResourceId: props.rootResourceId,
});
Does anyone know how to access the rootResourceId?
The root resource (/) id is a alphanumeric string like 4cfzeywftb, which can be found in the console breadcrumbs:
APIs > API (076t2zozc0) > Resources> / (4cfzeywftb)
or by calling get-resources:
aws apigateway get-resources --rest-api-id 076t2zozc0
{
"items": [
{
"id": "4cfzeywftb",
"path": "/",
"resourceMethods": {
"ANY": {}
}
},
{
"id": "36g7tq",
"parentId": "4cfzeywftb",
"pathPart": "{proxy+}",
"path": "/{proxy+}",
"resourceMethods": {
"ANY": {}
}
}
]
}

Gatsby not working paths deployed on the github pages (adding repo name in the url)

I deployed a gatsby website with the Github pages and I'm having errors like that:
Locally everything works perfectly, errors occur only on the server.
Seems like the server can not resolve paths correctly.
I'm adding unnecessary repository name after domain. How to remove that?
I tried changing some host options and deploying app again and once it worked properly, IDK why, then I made another deploy and it crushed again.
My gatsby.config:
const path = require("path");
const { title, keywords, description, author, defaultLang, trackingId } = require("./config/site");
module.exports = {
pathPrefix: "/lbearthworks",
siteMetadata: {
title,
keywords,
description,
author,
},
plugins: [
{
resolve: "gatsby-plugin-google-analytics",
options: {
trackingId,
},
},
{
resolve: "gatsby-plugin-manifest",
options: {
name: title,
short_name: "Lb",
start_url: "/",
background_color: "#212121",
theme_color: "#fed136",
display: "minimal-ui",
icon: "content/assets/gatsby-icon.png",
},
},
"gatsby-transformer-remark",
{
resolve: "gatsby-source-filesystem",
options: {
name: "markdown",
path: `${__dirname}/content`,
},
},
{
resolve: "gatsby-source-filesystem",
options: {
name: "images",
path: `${__dirname}/content/assets/images`,
},
},
"gatsby-plugin-eslint",
"gatsby-plugin-react-helmet",
"gatsby-transformer-sharp",
"gatsby-plugin-sharp",
"gatsby-plugin-offline",
{
resolve: "gatsby-plugin-sass",
options: {
data: `#import "core.scss";`,
includePaths: [path.resolve(__dirname, "src/style")],
},
},
...
],
};
Live version
Github Reository
When dealing with GitHub Pages you need to add an extra configuration to your build command, since you are adding a pathPrefix variable, you need to allow Netlify how to prefix those paths. Ideally, the build command should look like:
"deploy": "gatsby build --prefix-paths && gh-pages -d public"
In your case, because you are adding a file-based configuration (netlify.toml), your build command is:
[build]
command = "yarn && yarn testbuild"
publish = "public"
Note that testbuild is yarn test && yarn build, according to your repository.
So, one workaround is changing your package.json command to:
"testbuild": "yarn test && yarn build --prefix-paths && gh-pages -d public",
In addition, you should be in gh-pages branch as it shows the Gatsby's documentation:
When you run npm run deploy all contents of the public folder will be
moved to your repository’s gh-pages branch. Make sure that your
repository’s settings has the gh-pages branch set as the source to
deploy from.

How to get rid of "500 Internal Server Error" using ui5-middleware-simpleproxy

I built an SAPUI5 app using UI5 Tooling and now I'm having trouble connecting it to an OData service.
Since the real url to the OData Service is not public, let's just call it:
http://app99.sap.domain.com/sap/opu/odata/sap/Z_DOMAIN_SRV/
and opening it in my browser, adding $metadata?sap-language=DE at the end, works - I get the XML metadata document.
These are the files I tried to change to connect to the OData service:
webapp
manifest.json
package.json
ui5.yaml
Adding a "dataSources"-object and a "models"-object in manifest.json
{
"sap.app": {
...
"dataSources": {
"oDataService": {
"uri": "/sap/opu/odata/sap/Z_DOMAIN_SRV/",
"type": "OData"
}
}
},
"sap.ui5": {
...
"models": {
...
"": {
"dataSource": "oDataService",
"preload": true,
"settings": {
"useBatch": false
}
}
},
...
}
}
I also tried changing the "uri" value to the real link, http://app99.sap.domain.com/sap/opu/odata/sap/Z_DOMAIN_SRV/, but then I expectedly get a Cross-Origin error:
So I tried setting up a proxy by adding this to my package.json:
{
...
"devDependencies": {
"ui5-middleware-simpleproxy": "^0.5.1"
},
"ui5": {
"dependencies": [
"ui5-middleware-simpleproxy"
]
},
...
}
and adding this to my ui5.yaml file:
...
server:
customMiddleware:
- name: ui5-middleware-simpleproxy
mountPath: /sap/opu/odata/sap/Z_DOMAIN_SRV
afterMiddleware: compression
configuration:
baseUri: "http://app99.sap.domain.com/sap/opu/odata/sap/Z_DOMAIN_SRV/"
hoping that now
http://localhost:8080/sap/opu/odata/sap/Z_DOMAIN_SRV/$metadata?sap-language=DE
would turn into
http://app99.sap.domain.com/sap/opu/odata/sap/Z_DOMAIN_SRV/$metadata?sap-language=DE
and it would work, but that isn't the case - I just get the error from the screenshot on top.
Any idea what I am missing?
I really want to make it work with this ui5-middleware-simpleproxy, because I don't want to start up two servers manually (the UI5 Tooling one and another one for the proxy). Currently I just need to do ui5 serve and everything works out of the box.
After changing the config in ui5.yaml it worked.
I don't quite understand how the relationship between mountPath and baseUri works and why it worked this way, but some problems can be solved with try and error like in my case :D
...
server:
customMiddleware:
- name: ui5-middleware-simpleproxy
mountPath: /sap/opu/odata/sap
afterMiddleware: compression
configuration:
baseUri: "http://app99.sap.domain.com/sap/opu/odata/sap"

How can i list all packages in a Github Package Registry repository from the command line?

Let's say that we have the Github package registry repository https://maven.pkg.github.com/someOrganization . How can I cat the list of all packages in this repo into a txt file ?
This can be done using Preview API for GitHub Packages. You can query it in GraphQL using:
query($login: String!) {
organization(login:$login) {
registryPackages(first:10, packageType:MAVEN) {
nodes {
name
}
}
}
}
This will output something like:
{
"data": {
"organization": {
"registryPackages": {
"nodes": [
{
"name": "package1"
},
{
"name": "package2"
}
]
}
}
}
}
At the time of writing this requires both:
Valid Token with org:read and packages:read
Accept header for preview API: application/vnd.github.packages-preview+json
Now because you want to do this over the command line, you could curling it. There's already a good answer on how to use curl to access GitHub's GraphQL API: https://stackoverflow.com/a/42021388/1174076
Hope this helps.

Get all file names from a Github repo through the Github API

Is it possible to get all the file names from repository using the GitHub API?
I'm currently trying to tinker this using PyGithub, but I'm totally ok with manually doing the request as long as it works.
My algorithm so far is:
Get the user repo names
Get the user repo that matches a certain description
??? get repo file names?
This will have to be relative to a particular commit, as some files may be present in some commits and absent in others, so before you can look at files you'll need to use something like List commits on a repository:
GET /repos/:owner/:repo/commits
If you're just interested in the latest commit on a branch you can set the sha parameter to the branch name:
sha string SHA or branch to start listing commits from.
Once you have a commit hash, you can inspect that commit
GET /repos/:owner/:repo/git/commits/:sha
which should return something like this (truncated from GitHub's documentation):
{
"sha": "...",
"...",
"tree": {
"url": "https://api.github.com/repos/octocat/Hello-World/git/trees/691272480426f78a0138979dd3ce63b77f706feb",
"sha": "691272480426f78a0138979dd3ce63b77f706feb"
},
"...": "..."
}
Look at the hash of its tree, which is essentially its directory contents. In this case, 691272480426f78a0138979dd3ce63b77f706feb. Now we can finally request the contents of that tree:
GET /repos/:owner/:repo/git/trees/:sha
The output from GitHub's example is
{
"sha": "9fb037999f264ba9a7fc6274d15fa3ae2ab98312",
"url": "https://api.github.com/repos/octocat/Hello-World/trees/9fb037999f264ba9a7fc6274d15fa3ae2ab98312",
"tree": [
{
"path": "file.rb",
"mode": "100644",
"type": "blob",
"size": 30,
"sha": "44b4fc6d56897b048c772eb4087f854f46256132",
"url": "https://api.github.com/repos/octocat/Hello-World/git/blobs/44b4fc6d56897b048c772eb4087f854f46256132"
},
{
"path": "subdir",
"mode": "040000",
"type": "tree",
"sha": "f484d249c660418515fb01c2b9662073663c242e",
"url": "https://api.github.com/repos/octocat/Hello-World/git/blobs/f484d249c660418515fb01c2b9662073663c242e"
},
{
"path": "exec_file",
"mode": "100755",
"type": "blob",
"size": 75,
"sha": "45b983be36b73c0788dc9cbcb76cbb80fc7bb057",
"url": "https://api.github.com/repos/octocat/Hello-World/git/blobs/45b983be36b73c0788dc9cbcb76cbb80fc7bb057"
}
]
}
As you can see, we have some blobs, which correspond to files, and some additional trees, which correspond to subdirectories. You may want to do this recursively.
You can use Github git trees
https://api.github.com/repos/[USER]/[REPO]/git/trees/[BRANCH]?recursive=1
Repo
https://github.com/deeja/bing-maps-loader
Api Call
https://api.github.com/repos/deeja/bing-maps-loader/git/trees/master?recursive=1
which returns
{
sha: "55382e87889ccb4c173bc99a42cc738358fc253a",
url: "https://api.github.com/repos/deeja/bing-maps-loader/git/trees/55382e87889ccb4c173bc99a42cc738358fc253a",
tree: [
{
path: "README.md",
mode: "100644",
type: "blob",
sha: "41ceefc1262bb80a25529342ee3ec2ec7add7063",
size: 3196,
url: "https://api.github.com/repos/deeja/bing-maps-loader/git/blobs/41ceefc1262bb80a25529342ee3ec2ec7add7063"
},
{
path: "index.js",
mode: "100644",
type: "blob",
sha: "a81c94f70d1ca2a0df02bae36eb2aa920c7fb20e",
size: 1581,
url: "https://api.github.com/repos/deeja/bing-maps-loader/git/blobs/a81c94f70d1ca2a0df02bae36eb2aa920c7fb20e"
},
{
path: "package.json",
mode: "100644",
type: "blob",
sha: "45f24dcb7a457b14fede4cb907e957600882b340",
size: 595,
url: "https://api.github.com/repos/deeja/bing-maps-loader/git/blobs/45f24dcb7a457b14fede4cb907e957600882b340"
}
],
truncated: false
}
Much eaiser now with the graphql api, you can get it all in a single query
first you get your repo:
query {
repository(name: "MyRepo" owner: "mylogin"){
}
}
then you get its defaultBranchRef to make life easy
defaultBranchRef{
}
Now all a branch ref really is, is just a pointer to a commit, and since graphql is strongly typed (and refs can be different things) we need to let it know it is a commit,
target{
...on Commit {
}
}
so target is what our ref is pointing to, and we say "if its a commit, do this"
and what should it do?
it should get the most recent commit (since that will have the latest files in the repo)
so to do that we query history
history(first: 1 until: "2019-10-08T00:00:00"){
nodes{
}
}
now inside of nodes we are inside of our commit and now we can see the files,
the files in a commits pointer are really just a pointer to a tree, and a tree just has entries, which can be objects of either type Tree, or type blob
entries that represent files are known as blobs, but since we dont do anything with them but list their names, you dont even need to know that
but its important to know that trees are also entries, so if you find a tree you need to dig in deeper, but you can only go a pre defined amount of levels deep.
tree{
entries {
name
object {
...on Tree{
entries{
name
object {
...on Tree{
entries{
name
}
}
}
}
}
}
}
}
now to put it all together:
query{
repository(owner: "MyLogin", name: "MyRepo") {
defaultBranchRef {
target {
... on Commit {
history(first: 1 until: "2019-10-08T00:00:00") {
nodes {
tree {
entries {
name
object {
... on Tree {
entries {
name
object{
...on Tree{
entries{
name
object{
...on Tree{
entries{
name
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
As Dan mentioned: github trees
See working example below
import requests
user = "grumbach"
repo = "ft_ping"
url = "https://api.github.com/repos/{}/{}/git/trees/master?recursive=1".format(user, repo)
r = requests.get(url)
res = r.json()
for file in res["tree"]:
print(file["path"])
For the sake of simplicity I omitted error management, velociraptors are extinct anyway…
Use gh api for authenticated HTTP request to the GitHub API
in one line
gh api -X GET /repos/octocat/Hello-World/commits | grep -E -o ".{0,0}\[{\"sha\":\".{0,40}" | sed 's/\[{\"sha\":\"//' | xargs -I {} gh api -X GET /repos/octocat/Hello-World/commits/{} | grep -E -o "\"filename\":\".*?\""
Or in two steps
Get commits sha
gh api -X GET /repos/octocat/Hello-World/commits | grep -E -o ".{0,0}\[{\"sha\":\".{0,40}" | sed 's/\[{\"sha\":\"//' >> ~/commits
List file names
xargs < ~/commits -I {} gh api -X GET /repos/octocat/Hello-World/commits/{} | grep -E -o "\"filename\":\".*?\""