Is it possible to wget / curl protected files from Google Cloud Storage without making them public? I don't mind a fixed predefined token. I just want to avoid the case where my public file gets leeched, costing me good dollars.
Another way, if as you say you don't mind getting a token externally, is to use curl to set the 'Authorization' header in your call to GCS like so:
curl -H "Authorization: Bearer 1/fFBGRNJru1FQd44AzqT3Zg" https://www.googleapis.com/storage/v1/b/bucket/o/object?alt=media
The 'alt=media' query string parameter is necessary to download the object data directly instead of receiving a JSON response.
You can copy and paste the token obtained by authorizing with the Cloud Storage JSON API separately in the OAuth 2.0 Playground.
See also:
https://cloud.google.com/storage/docs/access-control
https://cloud.google.com/storage/docs/json_api/v1/objects/get
You can use Signed URLs. This allows you to create a signed URL that can be used to download an object without additional authentication.
You could also use the curlwget chrome extension so whenever you download something on Chrome it will create the url with headers etc to allow you to wget your file.
Related
I am trying to follow these docs to download an artifact from github using githubs API:
https://docs.github.com/en/rest/actions/artifacts#download-an-artifact
I ran the curl command given in the docs, and it gave me the following url from which to download the artifact (I have replaced the specifics with ...)
https://pipelines.actions.githubusercontent.com/serviceHosts/..../_apis/pipelines/1/runs/16/signedartifactscontent?artifactName=my-artifact&urlExpires=....&urlSigningMethod=HMACV2&urlSignature=....
I am able to download the artifact by putting the URL into my browser (it automatically downloads when the URL is visited) however I tried to use wget to download it via console and got this error:
wget https://pipelines.actions.githubusercontent.com/... # the command I ran
HTTP request sent, awaiting response... 400 Bad Request # the error I got
How can I download a zip file to console? Should I use something other than wget?
I'd like to clarify that viewing this link in the browser is possible even when not logged in to github (or when in private browsing). Also, I can download the zip file at the link as many times as I would like before the link expires after 1 minute. Also my repo is private, which is necessary for my work. I need to use an access token when doing the curl command as described in the docs, however the link that is returned to me does not require any authentication when accessed via a browser.
The api docs seem a bit ambiguous here. It is possible that the redirect can only be accessed a single time in which case you should try generating the redirect and first using wget to parse it. You can then unzip the file using the unzip command.
If that is not the case I believe this statement in the api docs is key:
Anyone with read access to the repository can use this endpoint. If the repository is private you must use an access token with the repo scope. GitHub Apps must have the actions:read permission to use this endpoint.
My guess is that your repository is private and you are logged in on the browser to Github which allows you to be authenticated hence why you are able to download from the redirect link. I would suggest trying from incognito mode to test this.
Migrating the repository to public would allow you to bypass this issue. Alternatively you can pass the authentication token as a header to wget like so in order to authenticate with the server to pull the file.
header='--header=Authorization: token <TOKEN>'
wget "$header" https://pipelines.actions.githubusercontent.com/... -O output_file
The problem was that I didn't put quotes around my url. I needed to do this:
wget "https://pipelines.actions.githubusercontent.com/serviceHosts/..../_apis/pipelines/1/runs/16/signedartifactscontent?artifactName=my-artifact&urlExpires=....&urlSigningMethod=HMACV2&urlSignature=...."
Using an API gateway, I created an S3 bucket to copy an image (image/jpg). This website describes how I can upload images using Amazon's API gateway: https://aws.amazon.com/premiumsupport/knowledge-center/api-gateway-upload-image-s3/.
When I type the URL by adding the bucket and object name, I get the following error:
{"message":"Missing Authentication Token"}
I would like to know, first of all, where API can find my image? How can I verify that the image is a local file as stated in the introduction? In addition to that, should I use the curl command to complete the transformation? I am unable to use Potsman.
I have included a note from the link, how can I change the type of header in a put request?
What is the best way to copy files from the API gateway to S3 without using the Lambda function?
Assuming you set up your API and bucket following the instructions in the link, you upload a local file to your bucket using curl command as below.
curl -X PUT -H "Accept: image/jpeg" https://my-api-id.execute-api.us-east-2.amazonaws.com/my-stage-name/my-bucket-name/my-local-file.jpg
Note that the header indicates it will accept jpeg files only. Depending on the file you are uploading to S3, you will change/add this header.
To answer the questions directly:
What is the best way to copy files from the API gateway to S3 without using the Lambda function? - follow steps in this link https://aws.amazon.com/premiumsupport/knowledge-center/api-gateway-upload-image-s3/
where API can find my image? - your image is located in your local host/computer. You use curl command to upload it via the API that you created
should I use the curl command to complete the transformation? - not sure what you meant by transformation. But you use curl command to upload the image file to your S3 bucket via API Gateway
how can I change the type of header in a put request? - you use -H to add headers in your curl command
Background
We are writing some documentation for our support team.
We want to include links to files that are stored in private GitHub repositories.
We do not want the documentation to become stale if somebody refactors the code in the private GitHub repositories, so instead I am setting up a CI job that parses the documentation (with jsoup if you are interested) and finds all the links.
Once we have all the links we start checking them.
NOTE: we have written a custom link checker, because one of the critical set of links we have is for our monitoring solution, and sadly (also understandably) the SaaS we are using returns 404's for any unauthenticated requests on the URLs of the alerts.
The SaaS itself uses a 2FA to access the Web UI, so what we have ended up doing is parsing the URLs and then constructing an equivalent call to the SaaS API to validate the link.
For the monitoring system we use, this is easy: all the URLs are the same format.
Question
Can we validate a random GitHub URL as valid (ideally using only curl - I can translate to my chosen HTTP client from there, and curl gives a more generic answer) using a Personal Access Token? And if so, how?
The URLs could be:
simple direct to repo URLs: https://github.com/<org>/<repo>
direct to branch URLs: https://github.com/<org>/<repo>/tree/<branch>
file URLs: https://github.com/<org>/<repo>/blob/<path/to/file>
diff URLs: https://github.com/<org>/<repo>/compare/[<branch>...]<branch>
other URLs that are based on the presence of the repo and do not vary in child path, e.g. https://github.com/<org>/<repo>/pulls, https://github.com/<org>/<repo>/settings/collaboration, etc
plus who knows what other URLs people will add within the docs...
Things I have tried that didn't work
HTTP Basic authentication with the Personal Access Token as the password, e.g.
curl -I -u stephenc:2....token.redacted....b https://github.com/stephenc/<repo-name>
HTTP/1.1 404 Not Found
HTTP Bearer authentication, e.g.
curl -I -H "Authorization: bearer 2....token.redacted....b" https://github.com/stephenc/<repo-name>
HTTP/1.1 404 Not Found
It looks like it works for some URLs (no idea which ones).
I can access curl -u agentgonzo:$TOKEN https://raw.githubusercontent.com/agentgonzo/repo/path/to/file using the API Token as my username, but the same doesn't work on https://github.com URLs. Not sure if this will help you or not.
I got an answer from GitHub Support: No
Since a personal access token won't work for GitHub web UI URLs, no, there isn't a way to verify all possible GitHub private repo URLs without making API calls in some cases.
Context: We are using Keycloak to secure our APIs by usually passing tokens through Authorization Headers. However, these APIs also allow users to download files (for instance: https://api.service.io/users.xlsx).
To use these "download endpoints", our web client applications pass users' token via query strings. (e.g. https://api.service.io/users.xlsx?accessToken=${bearerToken})).
Problem: Passing tokens via query string has several security flaws (browser history, ...). Therefore we would like to pass a very short-lived token (e.g. lifespan of 15sec) instead of the normal one (lifespan of 300sec by default).
Question: How could we request a different token from Keycloak API (for instance, /realms/#{realm_id}/protocol/openid-connect/token) by:
providing the normal access token (not credentials);
and specifying a different lifespan ?
After reading Keycloak's source code, it appears this is not possible (version 3.4.2.Final) to ask for a specific lifespan at runtime.
However, I developed a Keycloak Custom REST endpoint to do that. https://github.com/looorent/keycloak-configurable-token-api
When this JAR file is deployed in Keycloak, you can ask for a given lifespan at runtime. For example:
$ curl -X POST -d '{ "tokenLifespanInSeconds": 20}' -H "Content-Type: application/json" -H "Authorization: Bearer <user-access-token>" http://auth.service.io/auth/realms/a-realm/configurable-token
I’m trying to use the Github v3 API - I already implemented the required OAuth flow and it works well.
Now I’m trying some of the Repos API endpoints (http://developer.github.com/v3/repos/).
So far, I’m able to get a List of my repos using: GET /user/repos
However, when I try to create a repo using POST /user/repos, I get a 404.
Any thoughts what I might be doing wrong?
Joubert
Can you please tell us how exactly you did the HTTP request? The 404 sounds like you were using a wrong path, probably. But to give a reliable answer instead a wild guess, we need to see your request, including how you are sending your token, just mask it with 'xxx' or something.
I'll show you in the meantime an example request, that is working:
curl -XPOST -H 'Authorization: token S3CR3T' https://api.github.com/user/repos -d '{"name":"my-new-repo","description":"my new repo description"}'
You would need to replace the OAuth token of course: S3CR3T
I had the same issue. The reason why you are getting a 404 with your oauth access token is that when you authorize to github you need to also additionally pass the scopes you want. For example, in the header you should see "X-OAuth-Scopes: repo, user", which means this user has read/write access to his profile and repositories. Once you have set the correct scopes you should be able to do POST/PUT requests just fine.
To see whether or not you have the correct permissions. You can do something like the following. Substitute the XXXXXXX with your access token.
curl -I https://api.github.com/user?access_token=XXXXXXXX
For creating repositories as a user you can use an personal access token and basic auth, which can be much simpler when you are fluffing around on the command line and have 2FA enabled.
curl -d '{"name":"test"}' -u githubuser:personaccesstoken https://api.github.com/user/repos
Create a personal access token here https://github.com/settings/tokens and make sure it has the 'repo' scope.
This script lets you read in in the token and project name as variables so you can use it in a script
#!/usr/bin/env bash -u
#
TOKEN=`cat token_file`
PROJECT=myproject
curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' -d '{"name": "'"$PROJECT"'"}' https://api.github.com/user/repos?access_token=$TOKEN