Is there a way to update the metadata of an object in Google Storage when using an OAuth2 authentification token with read/write (but not full) access to the bucket?
My naive attempts (HTTP PUT with x-goog-copy-source set to the same object, and
x-goog-metadata-directive: REPLACE) fail with an AccessDenied error. gsutil seems to have the same problem:
$ gsutil config -r -w
[...]
$ echo hello > foo.txt
$ gsutil -h "x-goog-meta-foo: bar" cp foo.txt gs://nikratio-test/
Copying file://foo.txt [Content-Type=text/plain]...
Uploading gs://nikratio-test/foo.txt: 0 B/6 B
$ gsutil stat gs://nikratio-test/foo.txt
gs://nikratio-test/foo.txt:
Creation time: Sun, 09 Nov 2014 22:34:22 GMT
Content-Language: en
Content-Length: 6
Content-Type: text/plain
Metadata:
foo: bar
Hash (crc32c): NT3Yvg==
Hash (md5): sZRqySSS0jR8YjW00mERhA==
ETag: CID8ka7K7sECEAE=
Generation: 1415572462272000
Metageneration: 1
$ gsutil setmeta -h "x-goog-meta-foo: com" gs://nikratio-test/foo.txt
Setting metadata on gs://nikratio-test/foo.txt...
AccessDeniedException: 403 Forbidden
I presume this is because the setmeta operation would also allow me to change the ACL of the object.
However, it seems to me that if I'm able to delete the object and re-upload it with different metadata using the same authorization, then there really ought to be a way to just change the metadata as well.
Setting metadata works with OAuth2. I suggest running this gsutil command and comparing the protocol output it generates with what you're doing:
gsutil -o GSUtil:prefer_api=xml -d setmeta -h x-goog-meta-data-00:gAJ9cQBYBQAAAGFwcGxlcQFYCAAAAHBvdGF0b2VzcQJzLg gs://your-bucket/your-object
Related
I am trying to download the logs of a particular workflow in github.
I have referenced the following link for the same.
However I am getting 302 as response code.
Not sure what the issue here is. Its not downloading logs as expected
curl -v -u username:$token -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/ORGANIZATION/REPOSITORY/actions/runs/319282523477/logs
.
.
< HTTP/1.1 302 Found
< Date: Wed, 21 Oct 2020 07:47:13 GMT
< Content-Type: text/html;charset=utf-8
.
As per the same documentation that you mentioned:
Gets a redirect URL to download a plain text file of logs for a workflow job. This link expires after 1 minute. Look for Location: in the response header to find the URL for the download
Response
Status: 302 Found
So you might have already got the url to download the logs as the response has a 302 http status. Please check for the Location: response header, which should give you the url that you need to download the logs.
As a side note to Madhu Bhat's answer, you can simply add -L to your curl command to make it follow redirects.
(from curl --help)
-L, --location Follow redirects
I am using cURL to execute commands using the REST interface of BaseX like this:
curl http://localhost:8984/rest/?command=repo+list
There are also commands to manage the XQuery module repository. I am especially interested in REPO INSTALL to install a package. Is it somehow possible to execute this command using cURL and the REST interface but without having the package already on the target server? I want to provide the file in the body of the cURL request, similar to adding a resource to a database (source) which goes like this:
curl -i -X PUT -T "etc/xml/factbook.xml" "http://localhost:8984/rest/factbook"
Trying
curl -i -X PUT -T "/tmp/foo.xar" http://localhost:8984/rest/?command=repo+install
Gives me
HTTP/1.1 404 Not Found
Content-Type: text/plain;charset=UTF-8
Content-Length: 18
Connection: close
Server: Jetty(9.4.18.v20190429)
No path specified.
Adding -H "Content-Type: application/x-xar" does not help either.
And replacing PUT with POST gives me
HTTP/1.1 100 Continue
HTTP/1.1 400 Bad Request
Date: Tue, 03 Mar 2020 09:53:21 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 46
Server: Jetty(9.4.18.v20190429)
"" (Line 1): Content is not allowed in prolog.
The following works in case of standard modules (replace user/pass/server if needed):
$ curl http://admin:admin#localhost:8984/rest/?command=repo+install+http://www.xqueryfunctions.com/xq/functx-1.0.1-doc.xq
I'm using this wget command to download all the .fits files from that URL:
wget -r -np -nd -l inf -A fits https://archive.stsci.edu/missions/tess/ete-6/tid/00/000/000/057/
This is based on an adaptation of this answer.
All I'm getting is a directory structures that mirrors the URI on the website all the way down to /057/, but there's no file.
If I add -nd, then I only get a robot.txt files which isn't very instructive, but still no files.
What am I not getting about how to use wget for this?
EDIT: based on Turgbek's answer below, I do see that the robot.txt file from that website actually has /missions/ in the "Disallow"... maybe this is what is preventing me from using the wget command? Is that the source of the problem? How can I get around that?
In robots.txt there's a statement:
Disallow: /missions/
Which your requested files are in. Since the url builds up as /missions/tess/ete-6/tid/00/000/000/057/ I believe that robots.txt is blocking you.
I've saved two of the files from that URL in my Raspberry Pi and ran a local test without robots.txt. With this command:
wget -r -np -nd -l inf -A fits 192.168.1.250/test/
It worked as intended and I've received both of the files.
--2018-05-03 23:46:51-- http://192.168.1.250/test/tess2019128220341-0000000005712108-0016-s_lc.fits
Reusing existing connection to 192.168.1.250:80.
HTTP request sent, awaiting response... 200 OK
Length: 2090880 (2.0M)
Saving to: `192.168.1.250/test/tess2019128220341-0000000005712108-0016-s_lc.fits'
100%[==============================================================================>] 2,090,880 3.77M/s in 0.5s
2018-05-03 23:46:51 (3.77 MB/s) - `192.168.1.250/test/tess2019128220341-0000000005712108-0016-s_lc.fits' saved [2090880/2090880]
--2018-05-03 23:46:51-- http://192.168.1.250/test/tess2019128220341-0000000005715814-0016-s_lc.fits
Reusing existing connection to 192.168.1.250:80.
HTTP request sent, awaiting response... 200 OK
Length: 2090880 (2.0M)
Saving to: `192.168.1.250/test/tess2019128220341-0000000005715814-0016-s_lc.fits'
100%[==============================================================================>] 2,090,880 4.61M/s in 0.4s
2018-05-03 23:46:52 (4.61 MB/s) - `192.168.1.250/test/tess2019128220341-0000000005715814-0016-s_lc.fits' saved [2090880/2090880]
I am trying to get a CSV file from Redmine in a shell script. WGET is complaining about an unacceptable. Any ideas what the magical incantation is, or how to find it?
$ wget --no-check-certificate --accept csv https://username:password#company.com/redmine/issues.csv?utf8=%E2%9C%93&columns=all&description=1
Resolving company.com (company.com)... 192.168.1.45
Connecting to company.com (company.com)|192.168.1.45|:443... connected.
WARNING: The certificate of ‘company.com’ is not trusted.
WARNING: The certificate of ‘company.com’ hasn't got a known issuer.
HTTP request sent, awaiting response... 406 Not Acceptable
2017-04-04 10:14:20 ERROR 406: Not Acceptable.
You can try to replace --accept csv with --accept "*.csv". See the wget manual: https://www.gnu.org/software/wget/manual/wget.html#Recursive-Accept_002fReject-Options
Trying to mirror a local intranet site and have found previous questions using 'wget'. It works great with sites that are anonymous, but I have not been able to use it against a site that is expecting username\password (IIS with Integrated Windows Authentication).
Here is what I pass in:
wget -c --http-user='domain\user' --http-password=pwd http://local/site -dv
Here is the debug output (note I replaced some with dummy values obviously):
Setting --verbose (verbose) to 1
DEBUG output created by Wget 1.11.4 on Windows-MSVC.
--2009-07-14 09:39:04-- http://local/site
Host `local' has not issued a general basic challenge.
Resolving local... seconds 0.00, x.x.x.x
Caching local => x.x.x.x
Connecting to local|x.x.x.x|:80... seconds 0.00, connected.
Created socket 1896.
Releasing 0x003e32b0 (new refcount 1).
---request begin---
GET /site/ HTTP/1.0
User-Agent: Wget/1.11.4
Accept: */*
Host: local
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 401 Access Denied
Server: Microsoft-IIS/5.1
Date: Tue, 14 Jul 2009 13:39:04 GMT
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
Content-Length: 4431
Content-Type: text/html
---response end---
401 Access Denied
Closed fd 1896
Unknown authentication scheme.
Authorization failed.
NTLM authentication is broken in wget 1.11, use 1.10 instead.
Curl is actually probably a better tool for fetching content from NTLM-authenticated web servers. You can get an equivalent function to your proposed wget command line by using:
curl --anyauth --user username:password http://someserver/site
I've seen references to being able to use the NTLM Authorization Proxy Server to get around these types of problems.
use --auth-no-challenge option (wget 1.11+) (it's now considered unsafe)
I found solution.
It is work-around for Basic auth IIS7.
When auth is successeful it send next http header:
'Authorization: < type > < credentials >'.
So we able to do authorization in browser and
copy this header params from browser (firebug addon) or generate:
$ echo -en 'username:password' | base64
dXNlcm5hbWU6cGFzc3dvcmQK
$ echo 'dXNlcm5hbWU6cGFzc3dvcmQK' | base64 -d
username:password
example:
$ wget --header="Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQK" http://example.com/