How do I query GitHub Packages to determine whether a package already exists? I want to prevent CI/CD from attempting to publish a Maven package if that package already exists.
I'm trying a query that looks like:
curl -X POST -H "Authorization: bearer <some token>" -H "Accept: application/vnd.github.packages-preview+json" -d <my query> https://api.github.com/graphql
where <my query> looks like:
query {
organization(login: "myorg") {
registryPackagesForQuery(packageType: MAVEN, first: 100) {
edges {
node {
name
id
}
}
}
}
}
I am getting my packages returned:
{
"data": {
"organization": {
"registryPackagesForQuery": {
"edges": [
{
"node": {
"name": "com.example.myorg",
"id": "<some id>"
}
}
]
}
}
}
}
However, there is a query argument (of type String) for registryPackagesForQuery, but I don't know how that query format should look. Any attempt to add in a value seems to result in no data being returned.
What is the correct format of the query, and how does this query vary if I want to check whether a specific version of a package exists?
The base Query docs are here: https://developer.github.com/v4/query/.
Organization: https://developer.github.com/v4/object/organization/
RegistryPackageConnection: https://developer.github.com/v4/object/registrypackageconnection
RegistryPackage: https://developer.github.com/v4/object/registrypackage/
This may be off-topic, but wouldn't it be simpler to use a maven plugin for that? Like https://github.com/chonton/exists-maven-plugin maybe ?
Related
I am trying to create an index_pattern for dashboard in Opensearch programmatically.
Due to the fact that I didn't found anything related to it on Opensearch documentation, I tried the elastic search saved_objects api:
POST /api/saved_objects/index-pattern/my_index_pattern
{
"attributes":{
"title": "my_index"
}
}
But when I call it I got the following error:
{
"error" : "no handler found for uri [/api/saved_objects/index-pattern/my_index_pattern?pretty=true] and method [POST]"
}
How can I solve it? Are there a different call for Opensearch to create an index_pattern?
curl -k -X POST http://<host>:5601/api/saved_objects/index-pattern/logs-* -H "osd-xsrf:true" -H "content-type:application/json" -d '
{
"attributes": {
"title": "logs-*",
"timeFieldName": "#timestamp"
}
}'
this works for me
In a single Deployment Manager template, how do I create a new folder and a new project underneath it? The problem is that the reference to the folder includes a name in the format folders/123456, but the project requires a parent field in the format {'type': 'folder', 'id': 123456}. Using a $(ref.new-folder.name) won't work for the ID field in the parent record for the new project.
It feels like I need to do string manipulation on the $(ref.new-folder.name) like this:
# DOES NOT WORK
# but if it did, I could extract the numeric id from 'folders/123456'
parent_id = '$(ref.new-folder.name)'.replace('folders/', '')
But, of course, that won't work.
Here is my (non-working) attempt:
# Template for new folder & new project
folder_resource = {
'name': 'new-folder',
'type': 'gcp-types/cloudresourcemanager-v2:folders',
'properties': {
'parent': 'organizations/99999',
'displayName': 'new-folder'
}
}
project_resource = {
'name': 'new-project',
'type': 'clouresourcemanager.v1.project',
'metadata': { 'dependsOn': ['new-folder'] },
'properties': {
'name': 'new-project',
'parent': {
'type': 'folder',
# HERE it is -- the problem!
'id': '$(ref.new-folder.name)'
}
}
}
return { 'resources': [folder_resource, project_resource] }
So, to reiterate, I'm getting hung-up on extracting the numeric folder id from the reference to the folder's name. The name is in the format folders/123456 but I just need the 123456 part to use in the parent field for the new project.
This question is specific to folder & project creation, but a more generalized question would be: is there a way to do string-manipulation on the value of references?
For creating and managing folders document [a] might be helpful and folder name must meet the following requirements:
The name may contain letters, digits, spaces, hyphens and underscores.
The folder's display name must start and end with a letter or digit.
The name must be 30 characters or less.
The name must be distinct from all other folders that share its parent.
To create a folder:
Folders can be created with an API request.
The request JSON:
request_json= '{
display_name: "[DISPLAY_NAME]"
}'
The Create Folder curl request:
curl -X POST -H "Content-Type: application/json" \
-H "Authorization: Bearer ${bearer_token}" \
-d "$request_json" \
https://cloudresourcemanager.googleapis.com/v2/folders?parent=[ORGANIZATION_NAME]
Where:
-[DISPLAY_NAME] is the new folder's display name, for example "My Awesome Folder."
-[ORGANIZATION_NAME] is the name of the organization under which you're creating the
folder, for example organizations/123.
The Create Folder response:
{
"name": "operations/fc.123456789",
"metadata": {
"#type": "type.googleapis.com/google.cloud.resourcemanager.v2.FolderOperation",
"displayName": "[DISPLAY_NAME]",
"operationType": "CREATE"
}
}
The Get Operation curl request:
curl -H "Authorization: Bearer ${bearer_token}" \
https://cloudresourcemanager.googleapis.com/v1/operations/fc.123456789
The Get Operation response:
{
"name": "operations/fc.123456789",
"metadata": {
"#type": "type.googleapis.com/google.cloud.resourcemanager.v2.FolderOperation",
"displayName": "[DISPLAY_NAME]",
"operationType": "CREATE"
},
"done": true,
"response": {
"#type": "type.googleapis.com/google.cloud.resourcemanager.v2.Folder",
"name": "folders/12345",
"parent": "organizations/123",
"displayName": "[DISPLAY_NAME]",
"lifecycleState": "ACTIVE",
"createTime": "2017-07-19T23:29:26.018Z",
"updateTime": "2017-07-19T23:29:26.046Z"
}
}
Configuring access to folders
SetsIamPolicy sets the access control policy on a folder, replacing any existing policy. The resource field should be the folder's resource name, for example, folders/1234.
request_json= '{
policy: {
version: "1",
bindings: [
{
role: "roles/resourcemanager.folderEditor",
members: [
"user:email1#example.com",
"user:email2#example.com",
]
}
]
}
}'
The curl request:
curl -X POST -H "Content-Type: application/json" \
-H "Authorization: Bearer ${bearer_token}" \
-d "$request_json" \
https://cloudresourcemanager.googleapis.com/v2/[FOLDER_NAME]:setIamPolicy
Where:
-[FOLDER_NAME] is the name of the folder whose IAM policy is being set, for example folders/123.
Creating a project in a folder
request_json= ‘{
name: “[DISPLAY_NAME]”, projectId: “[PROJECT_ID]”, parent: {id: [PARENT_ID], type: [PARENT_TYPE] }
}’
The curl request:
curl -X POST -H "Content-Type: application/json" \
-H "Authorization: Bearer ${bearer_token}" \
-d "$request_json" \
https://cloudresourcemanager.googleapis.com/v1/projects
Where:
-[PROJECT_ID] is id of the project being created, for e.g., my-awesome-proj-123.
-[DISPLAY_NAME] is the display name of the project being created.
-[PARENT_ID] is the id of the parent being created under, for e.g. 123
-[PARENT_TYPE] is the type of the parent, like “folder” or “Organization”
When we create a reference to a resource, we also create a dependency between resources, document [b] might be helpful for this.
[a]-https://cloud.google.com/resource-manager/docs/creating-managing-folders
[b]-https://cloud.google.com/deployment-manager/docs/configuration/use-references
Is it possible to get all the file names from repository using the GitHub API?
I'm currently trying to tinker this using PyGithub, but I'm totally ok with manually doing the request as long as it works.
My algorithm so far is:
Get the user repo names
Get the user repo that matches a certain description
??? get repo file names?
This will have to be relative to a particular commit, as some files may be present in some commits and absent in others, so before you can look at files you'll need to use something like List commits on a repository:
GET /repos/:owner/:repo/commits
If you're just interested in the latest commit on a branch you can set the sha parameter to the branch name:
sha string SHA or branch to start listing commits from.
Once you have a commit hash, you can inspect that commit
GET /repos/:owner/:repo/git/commits/:sha
which should return something like this (truncated from GitHub's documentation):
{
"sha": "...",
"...",
"tree": {
"url": "https://api.github.com/repos/octocat/Hello-World/git/trees/691272480426f78a0138979dd3ce63b77f706feb",
"sha": "691272480426f78a0138979dd3ce63b77f706feb"
},
"...": "..."
}
Look at the hash of its tree, which is essentially its directory contents. In this case, 691272480426f78a0138979dd3ce63b77f706feb. Now we can finally request the contents of that tree:
GET /repos/:owner/:repo/git/trees/:sha
The output from GitHub's example is
{
"sha": "9fb037999f264ba9a7fc6274d15fa3ae2ab98312",
"url": "https://api.github.com/repos/octocat/Hello-World/trees/9fb037999f264ba9a7fc6274d15fa3ae2ab98312",
"tree": [
{
"path": "file.rb",
"mode": "100644",
"type": "blob",
"size": 30,
"sha": "44b4fc6d56897b048c772eb4087f854f46256132",
"url": "https://api.github.com/repos/octocat/Hello-World/git/blobs/44b4fc6d56897b048c772eb4087f854f46256132"
},
{
"path": "subdir",
"mode": "040000",
"type": "tree",
"sha": "f484d249c660418515fb01c2b9662073663c242e",
"url": "https://api.github.com/repos/octocat/Hello-World/git/blobs/f484d249c660418515fb01c2b9662073663c242e"
},
{
"path": "exec_file",
"mode": "100755",
"type": "blob",
"size": 75,
"sha": "45b983be36b73c0788dc9cbcb76cbb80fc7bb057",
"url": "https://api.github.com/repos/octocat/Hello-World/git/blobs/45b983be36b73c0788dc9cbcb76cbb80fc7bb057"
}
]
}
As you can see, we have some blobs, which correspond to files, and some additional trees, which correspond to subdirectories. You may want to do this recursively.
You can use Github git trees
https://api.github.com/repos/[USER]/[REPO]/git/trees/[BRANCH]?recursive=1
Repo
https://github.com/deeja/bing-maps-loader
Api Call
https://api.github.com/repos/deeja/bing-maps-loader/git/trees/master?recursive=1
which returns
{
sha: "55382e87889ccb4c173bc99a42cc738358fc253a",
url: "https://api.github.com/repos/deeja/bing-maps-loader/git/trees/55382e87889ccb4c173bc99a42cc738358fc253a",
tree: [
{
path: "README.md",
mode: "100644",
type: "blob",
sha: "41ceefc1262bb80a25529342ee3ec2ec7add7063",
size: 3196,
url: "https://api.github.com/repos/deeja/bing-maps-loader/git/blobs/41ceefc1262bb80a25529342ee3ec2ec7add7063"
},
{
path: "index.js",
mode: "100644",
type: "blob",
sha: "a81c94f70d1ca2a0df02bae36eb2aa920c7fb20e",
size: 1581,
url: "https://api.github.com/repos/deeja/bing-maps-loader/git/blobs/a81c94f70d1ca2a0df02bae36eb2aa920c7fb20e"
},
{
path: "package.json",
mode: "100644",
type: "blob",
sha: "45f24dcb7a457b14fede4cb907e957600882b340",
size: 595,
url: "https://api.github.com/repos/deeja/bing-maps-loader/git/blobs/45f24dcb7a457b14fede4cb907e957600882b340"
}
],
truncated: false
}
Much eaiser now with the graphql api, you can get it all in a single query
first you get your repo:
query {
repository(name: "MyRepo" owner: "mylogin"){
}
}
then you get its defaultBranchRef to make life easy
defaultBranchRef{
}
Now all a branch ref really is, is just a pointer to a commit, and since graphql is strongly typed (and refs can be different things) we need to let it know it is a commit,
target{
...on Commit {
}
}
so target is what our ref is pointing to, and we say "if its a commit, do this"
and what should it do?
it should get the most recent commit (since that will have the latest files in the repo)
so to do that we query history
history(first: 1 until: "2019-10-08T00:00:00"){
nodes{
}
}
now inside of nodes we are inside of our commit and now we can see the files,
the files in a commits pointer are really just a pointer to a tree, and a tree just has entries, which can be objects of either type Tree, or type blob
entries that represent files are known as blobs, but since we dont do anything with them but list their names, you dont even need to know that
but its important to know that trees are also entries, so if you find a tree you need to dig in deeper, but you can only go a pre defined amount of levels deep.
tree{
entries {
name
object {
...on Tree{
entries{
name
object {
...on Tree{
entries{
name
}
}
}
}
}
}
}
}
now to put it all together:
query{
repository(owner: "MyLogin", name: "MyRepo") {
defaultBranchRef {
target {
... on Commit {
history(first: 1 until: "2019-10-08T00:00:00") {
nodes {
tree {
entries {
name
object {
... on Tree {
entries {
name
object{
...on Tree{
entries{
name
object{
...on Tree{
entries{
name
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
As Dan mentioned: github trees
See working example below
import requests
user = "grumbach"
repo = "ft_ping"
url = "https://api.github.com/repos/{}/{}/git/trees/master?recursive=1".format(user, repo)
r = requests.get(url)
res = r.json()
for file in res["tree"]:
print(file["path"])
For the sake of simplicity I omitted error management, velociraptors are extinct anyway…
Use gh api for authenticated HTTP request to the GitHub API
in one line
gh api -X GET /repos/octocat/Hello-World/commits | grep -E -o ".{0,0}\[{\"sha\":\".{0,40}" | sed 's/\[{\"sha\":\"//' | xargs -I {} gh api -X GET /repos/octocat/Hello-World/commits/{} | grep -E -o "\"filename\":\".*?\""
Or in two steps
Get commits sha
gh api -X GET /repos/octocat/Hello-World/commits | grep -E -o ".{0,0}\[{\"sha\":\".{0,40}" | sed 's/\[{\"sha\":\"//' >> ~/commits
List file names
xargs < ~/commits -I {} gh api -X GET /repos/octocat/Hello-World/commits/{} | grep -E -o "\"filename\":\".*?\""
I have been using Druid for the past week and wanted to enable javascript for some postAggregations.
I think I followed the outlined steps and updated the common.runtime.properties file in ../con f/druid/_common/ to include druid.javascript.enabled=true. I then stopped the current processes and re-ran the Quickstart procedures, but it still says that JavaScript is disabled:
{
"error" : "Unknown exception",
"errorMessage" : "Instantiation of [simple type, class io.druid.query.aggregation.post.JavaScriptPostAggregator] value failed: JavaScript is disabled. (through reference chain: java.util.ArrayList[0])",
"errorClass" : "com.fasterxml.jackson.databind.JsonMappingException",
"host" : null
}
I am currently running it in the 'Quickstart' configuration - single local machine. Any pointers? Thanks!
JavaScript Query For druid Aggregation. Save the file as .body and hit the curl request.
This is a sample query for Average value.
curl -X POST "http://localhost:8082/druid/v2/?pretty" \ -H
'content-type: application/json' -d #query.body
{
"queryType":"groupBy",
"dataSource":"whirldata",
"granularity":"all",
"dimensions":[],
"aggregations":[{"name":"rows","type":"count","fieldName":"rows"},
{"name":"TargetDOS","type":"doubleSum","fieldName":"Target DOS"}],"postAggregations":[
{
"type": "javascript",
"name": "Target DOS Average",
"fieldNames": ["TargetDOS", "rows"],
"function": "function(TargetDOS, rows) { return Math.abs(TargetDOS) / rows; }"
}], "intervals":[ "2006-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z" ]}
The part you are missing is likely that the quickstart reads configs from conf-quickstart rather than conf. So try editing conf-quickstart/druid/_common/common.runtime.properties.
I couldn't find any docs explain the syntax or grammar for the JSON used by HCatalog REST API.
The office guide (
https://cwiki.apache.org/confluence/display/Hive/WebHCat+Reference+PutTable) only gives a very simple case without saying anything useful about how the JSON part is defined.
I tried the following, but without luck:
curl -X PUT -HContent-type:application/json -d '
{
"format": {
"storedAs": "INPUTFORMAT \"org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat\" OUTPUTFORMAT \"org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat\"",
"rowFormat": {
"serde": {
"name": "org.apache.hadoop.hive.serde2.avro.AvroSerDe"
} } }
"tblproperties" : [
"avro.schema.url": "hdfs://xxxx"
]
} ' \
'http://<host>:50111/templeton/v1/ddl/database/default/table/table1?user.name=hive'
any idea?
Thanks,