I am running following command to fetch the content of file from github
curl -H 'Authorization: token XXXXXX'
-H 'Accept: application/vnd.github.v3.raw'
-o file.txt
-L https://github.com/USERNAME/REPONAME/blob/main/file.txt
(line breaks only added for readability here)
But the file downloaded or created contains HTML content not content of required file.
Please suggest the command to get same content of file without using raw link.
You're making a request to the web interface (github.com) and you're getting back an HTML page that contains the data of the file rendered in the web interface. If you want to download a file programmatically, you should use the appropriate REST API, which will be at a different URL under the domain api.github.com.
In general, you should only make programmatic requests to the API and avoid making requests directly to things under github.com.
Related
I have a script that I'd like to be able to access via a curl command against its https://raw.githubusercontent.com/... location. Using git.io, it's really easy to shorten this URL to something like https://git.io/ABCDE.
But there's an issue related to the fact that my script exists in a private repository. If I directly curl against the githubusercontent URL, I get 404: Not Found. I'm able to bypass this by passing an authorization header with the request, e.g.
$ curl -H "Authorization: token <My Github Personal Access Token>" \
https://raw.githubusercontent.com/...
> !#/bin/bash
... # rest of script
However, when I use my shortened URL, I don't get anything back. Not even a 404.
$ curl -H "Authorization: token <My Github Personal Access Token>" \
https://git.io/ABCDE
$
Anyone know what's going on here?
The way a URL shortener works is that it issues some sort of 3xx-series HTTP status code that redirects you to the new location, and then you make your request against that new location. However, by default, curl does not follow redirects, so all you see when you make your request is the output from git.io, which in this case is nothing.
If you want to follow redirects, then you should use the -L option to curl, which will make it follow redirects. Note that this can be insecure in many cases when passing credentials, since any credentials passed with -H will be passed to any remote server that the data is redirected to. In this case, that's what you want, but it can be a security problem in other cases if the credentials were only intended for the original server.
Using an API gateway, I created an S3 bucket to copy an image (image/jpg). This website describes how I can upload images using Amazon's API gateway: https://aws.amazon.com/premiumsupport/knowledge-center/api-gateway-upload-image-s3/.
When I type the URL by adding the bucket and object name, I get the following error:
{"message":"Missing Authentication Token"}
I would like to know, first of all, where API can find my image? How can I verify that the image is a local file as stated in the introduction? In addition to that, should I use the curl command to complete the transformation? I am unable to use Potsman.
I have included a note from the link, how can I change the type of header in a put request?
What is the best way to copy files from the API gateway to S3 without using the Lambda function?
Assuming you set up your API and bucket following the instructions in the link, you upload a local file to your bucket using curl command as below.
curl -X PUT -H "Accept: image/jpeg" https://my-api-id.execute-api.us-east-2.amazonaws.com/my-stage-name/my-bucket-name/my-local-file.jpg
Note that the header indicates it will accept jpeg files only. Depending on the file you are uploading to S3, you will change/add this header.
To answer the questions directly:
What is the best way to copy files from the API gateway to S3 without using the Lambda function? - follow steps in this link https://aws.amazon.com/premiumsupport/knowledge-center/api-gateway-upload-image-s3/
where API can find my image? - your image is located in your local host/computer. You use curl command to upload it via the API that you created
should I use the curl command to complete the transformation? - not sure what you meant by transformation. But you use curl command to upload the image file to your S3 bucket via API Gateway
how can I change the type of header in a put request? - you use -H to add headers in your curl command
I am using Curl to make requests to the Jenkins API - which is being done over https.
Jenkins is setup in IIS using a reverse proxy as advised by Jenkins.
The Jenkins site has windows authentication.
I need to pass a Crumb with my request - 403 error.
However, when I request for the Crumb, it states that I need to provide a Crumb?
I need this to work via the command prompt/batch file.
Thanks
To be able to do API calls to Jenkins, you need to generate a token for a given user in Jenkins. For example, let's do it with user Foo. You'll need to sign in with Foo user and then in the web UI: Foo (right upper corner) > Configure > API token > Add new token.
When you have your token saved somewhere, you can retrieve the breadcrumb with this command:
curl --user Foo:<token> 'https://www.mysuperduperjenkins.com/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)'
What this is doing basically is fetching the XML from Jenkins that contains the breadcrumb and getting the specific element where the breadcrumb is. The result of the command should be something similar to:
Jenkins-Crumb:<someLongLongLongToken>
You can save the breadcrumb into a variable, for example name it breadcrumb and then:
curl -u 'Foo:<token>' -X POST --data '<parameters>' -H '$breadcrumb' www.mysuperduperjenkins.com
In this curl we're using the breadcrumb as a header, obtained in previous step. If the crumb is valid you should be able to perform the request without 403/401 errors.
This page states:
Bamboo's REST APIs provide the following capabilities:
Retrieve the artifacts for a build.
and here I see the documentation:
http://myhost.com:8085/bamboo/rest/api/latest/plan/{projectKey}-{buildKey}/artifact
[GET]
When I try this link with the bamboo server I have, like:
https://my.bamboo.server/rest/api/latest/plan/MY-PLAN/artifact
All I get is:
<artifacts expand="artifacts">
<link href="http://my.bamboo.server/rest/api/latest/plan/MY-PLAN/artifact" rel="self"/>
<artifacts start-index="0" max-result="0" size="0"/>
</artifacts>
So am I understanding the REST documentation completely wrong, or is there something wrong possibly with MY-PLAN and this link is supposed to provide me a war file as I expect?
I'm afraid you are misunderstanding the REST documentation; by "Retrieve the artifacts for a build", it means "retrieves information about the build artifacts defined for a given plan". As you have already seen, all you get back is an XML or JSON document describing the artifacts defined.
If you want to download an actual build artifact, you'll need to write a script that uses /rest/api/latest/result/ to get the latest successful build info and, from that, form an actual download link to the artifact.
There are some issues related to your question: https://jira.atlassian.com/browse/BAM-11706
and BAM-16315 (which was deleted, because it contained customer details)
Here is the rest api documentation
https://docs.atlassian.com/atlassian-bamboo/REST/latest
Search for "/latest/result" documentation
http://myhost.com:8085/rest/api/latest/result/{projectKey}-{buildKey}-{buildNumber : ([0-9]+)|(latest)} [GET]
Example xml request
https://bamboo.server.com/rest/api/latest/result/projectKey-buildKey-buildNumber?expand=artifacts
Example json request
https://bamboo.server.com/rest/api/latest/result/projectKey-buildKey-buildNumber.json?expand=artifacts
Parse the artifacts node in the response. Each artifact should have an href property.
Pass the href to curl to download the artifact. You will probably need to setup a Bamboo token for rest api authentication.
Example curl request
curl -X GET -H "Authorization: Bearer ${BAMBOO_TOKEN}" $ARTIFACT_HREF
Spent a long time looking for this answer. Ended up having to piece together information and here is what I got. As #oallauddin mentions above, you need to get the url of the file from the xml.
curl -H "Authorization: Bearer <KEY>" https://bamboo.server.com/rest/api/latest/result/projectKey-buildKey-buildNumber?expand=artifacts
Alternative you can get it as json format
curl -H "Authorization: Bearer <KEY>" https://bamboo.server.com/rest/api/latest/result/projectKey-buildKey-buildNumber.json?expand=artifacts
I think where a lot of people are getting stuck is trying to use the Authorization header to download it after. BAM-20764 shows that this is NOT possible. You have to remove the header and use basic authentication with -u. Not both. Only the -u by itself.
curl -u username:password https://bamboo.server.com/browse/projectKey-build-Key-buildNumber/artifact/shared/<ARTIFACT_NAME>/<filename> --remote-name
The --remote-name/-O flag or --remote-header-name/-J is preferred here if you want to download the file with the default original name.
You've the link
<link href="http://my.bamboo.server/rest/api/latest/plan/MY-PLAN/artifact" rel="self"/>
Using curl you can download the artifact.
curl --user ${username}:{password} http://my.bamboo.server/rest/api/latest/plan/MY-PLAN/artifact
Is it possible to wget / curl protected files from Google Cloud Storage without making them public? I don't mind a fixed predefined token. I just want to avoid the case where my public file gets leeched, costing me good dollars.
Another way, if as you say you don't mind getting a token externally, is to use curl to set the 'Authorization' header in your call to GCS like so:
curl -H "Authorization: Bearer 1/fFBGRNJru1FQd44AzqT3Zg" https://www.googleapis.com/storage/v1/b/bucket/o/object?alt=media
The 'alt=media' query string parameter is necessary to download the object data directly instead of receiving a JSON response.
You can copy and paste the token obtained by authorizing with the Cloud Storage JSON API separately in the OAuth 2.0 Playground.
See also:
https://cloud.google.com/storage/docs/access-control
https://cloud.google.com/storage/docs/json_api/v1/objects/get
You can use Signed URLs. This allows you to create a signed URL that can be used to download an object without additional authentication.
You could also use the curlwget chrome extension so whenever you download something on Chrome it will create the url with headers etc to allow you to wget your file.