I have some problem about OpenStack swift object storage. I want to set an expiration for objects. I use openstack4j. My upload object code
public void add(String objectName, InputStream imageStream) {
OSClientV3 clientV3 = OSFactory.clientFromToken(swiftOS.getToken());
Map<String, String> metaData = new HashMap<>();
metaData.put("X-Delete-After", "120");
ObjectPutOptions objectPutOptions = ObjectPutOptions.create();
objectPutOptions.metadata(metaData);
clientV3.objectStorage().objects().put(container, objectName,
Payloads.create(imageStream), objectPutOptions);
}
But it doesn't work.
Then I tried to use swift command line.
swift stat test test-19b8e4d4-1085-490d-b866-97f0ada0d46c
What I get is
Account: AUTH_01d73f1e49ed4dfd9535c831eed4ccf9
Container: test
Object: test-19b8e4d4-1085-490d-b866-97f0ada0d46c
Content Type: application/octet-stream
Content Length: 2414
Last Modified: Wed, 20 Dec 2017 07:58:56 GMT
ETag: 1cb55838010ed189c0698b6b5cade3ed
Meta X-Delete-After: 120
X-Openstack-Request-Id: tx4f1f57ef08e34d9296bfd-005a3a184b
X-Timestamp: 1513756735.97761
X-Trans-Id: tx4f1f57ef08e34d9296bfd-005a3a184b
Accept-Ranges: bytes
When I upload an object by
swift upload test test.jpg -H "X-Delete-After: 120"
and then
swift stat test test.jpg
What I get is
Account: AUTH_01d73f1e49ed4dfd9535c831eed4ccf9
Container: test
Object: test.jpg
Content Type: application/octet-stream
Content Length: 1688
Last Modified: Wed, 20 Dec 2017 08:03:20 GMT
ETag: 8a2d75ff8db40610a52a492abac09d3b
Meta Mtime: 1513755398.217256
X-Delete-At: 1513757119
Accept-Ranges: bytes
X-Timestamp: 1513756999.02865
X-Trans-Id: txc016e1aff901450aa934b-005a3a194c
X-Openstack-Request-Id: txc016e1aff901450aa934b-005a3a194c
It is like openstack document said.
The X-Delete-After header takes an integer number of seconds.
The proxy server that receives the request will convert this
header into an X-Delete-At header using its current time plus
the value given.
But why openstack4j doesn't work?
Looks like we can use
objectPutOptions.getOptions().put("X-Delete-After", "120");
As the getOptions call returns the headers map straight up.
You should use header instead of metadata:
objectPutOptions.header("X-Delete-After", "120");
Related
I am writing a REST plugin for Foswiki using Perl and I am facing an reliability issue when using File::Find. I have tried my best to write a minimal reproducible example. The plugin uses File::Find to traverse directories and print the filenames in the HTTP response. The REST request is working properly 4 times, but stop to work the 5th time. The HTTP status remain “HTTP/1.1 200 OK” but no file is reported by File::Find anymore.
The webserver is nginx and is configured to use FastCGI. It appear to run 4 working threads managed by foswiki-fcgi-pm:
> ps aux
www-data 16957 0.0 7.7 83412 78332 ? Ss 16:52 0:00 foswiki-fcgi-pm
www-data 16960 0.0 7.5 83960 76740 ? S 16:52 0:00 foswiki-fcgi
www-data 16961 0.0 7.6 84004 76828 ? S 16:52 0:00 foswiki-fcgi
www-data 16962 0.0 7.6 83956 76844 ? S 16:52 0:00 foswiki-fcgi
www-data 16963 0.0 7.5 83960 76740 ? S 16:52 0:00 foswiki-fcgi
Firstly, the plugin initialization simply register the REST handler:
sub initPlugin {
my ( $topic, $web, $user, $installWeb ) = #_;
# check for Plugins.pm versions
if ( $Foswiki::Plugins::VERSION < 2.3 ) {
Foswiki::Func::writeWarning( 'Version mismatch between ',
__PACKAGE__, ' and Plugins.pm' );
return 0;
}
Foswiki::Func::registerRESTHandler(
'restbug', \&RestBug,
authenticate => 0, # Set to 0 if handler should be useable by WikiGuest
validate => 0, # Set to 0 to disable StrikeOne CSRF protection
http_allow => 'GET,POST', # Set to 'GET,POST' to allow use HTTP GET and POST
description => 'Debug'
);
# Plugin correctly initialized
return 1;
}
Secondly, the REST handler is implemented as follow, printing all the files it can possibly find:
sub RestBug {
my ($session, $subject, $verb, $response) = #_;
my #Directories = ("/var/www/foswiki/tools");
sub findfilestest
{
$response->print("FILE $_\n");
}
find({ wanted => \&findfilestest }, #Directories );
}
When I test the REST service with a HTTP request, the first 4 times I get the following HTTP response, which seems quite satisfying:
HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Tue, 22 Nov 2022 09:23:10 GMT
Content-Length: 541
Connection: keep-alive
Set-Cookie: SFOSWIKISID=385db599c5d66bb19591e1eef7f1a854; path=/; secure; HttpOnly
FILE .
FILE foswiki.freebsd.init-script
FILE bulk_copy.pl
FILE dependencies
FILE mod_perl_startup.pl
FILE geturl.pl
FILE extender.pl
FILE extension_installer
FILE configure
FILE lighttpd.pl
FILE foswiki.freebsd.etc-defaults
FILE save-pending-checkins
FILE babelify
FILE upgrade_emails.pl
FILE tick_foswiki.pl
FILE foswiki.defaults
FILE rewriteshebang.pl
FILE fix_file_permissions.sh
FILE foswiki.init-script
FILE convertTopicSettings.pl
FILE mailnotify
FILE html2tml.pl
FILE tml2html.pl
FILE systemd
FILE foswiki.service
The following attempts give this unexpected response:
HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Tue, 22 Nov 2022 09:24:56 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: SFOSWIKISID=724b2c4b1ddfbebd25d0dc2a0f182142; path=/; secure; HttpOnly
Note that if I restart Foswiki with the command systemctl restart foswiki, the REST service work again 4 more times.
How to make this REST service work more than 4 times in a row?
I am trying to get an image of a web page using pageres package, And post the image to slack using hubot. I am able to get the image, but for some reason i am not able to post it to slack using slack upload api. Here is my code, can you tell me what could be wrong? (not a coffee lint issue)
fs = require("fs")
Pageres = require('pageres')
util = require("util")
request = require("request")
module.exports = (robot) ->
robot.respond /screenshot page (\S*)?( at )?(\S*)?/i, (msg) ->
pageres = new Pageres({delay: 30})
domain = msg.match[1].replace("http://", "")
if msg.match[3] == undefined
size = '960x1024'
else
size = msg.match[3]
dest = './screenshots'
msg.send "Acquiring screenshot of #{domain}"
pageres.src(domain, [size]).dest(dest)
pageres.run (err) ->
if err
robot.logger.error err
msg.send "Um..., you better check the log"
else
opts = {
method: 'POST',
uri: 'https://slack.com/api/files.upload',
formData: {
channels: process.env.HUBOT_SCREENSHOT_SLACK_CHANNEL,
initial_comment: "Screenshot of #{domain}",
token: process.env.HUBOT_SLACK_TOKEN,
file: fs.createReadStream("#{dest}/#{domain}.png")
}
}
request.post opts, (error, response, body) ->
if error
robot.logger.error error
else
robot.logger.debug 'screenshot posted to slack'
return
The bot is connected to slack, and receiving messages from slack, parsing them fine and getting the image back to the local destination, but not able to post it to slack, There are no errors as well in the log.
[Wed Apr 11 2018 16:16:47 GMT+0000 (UTC)] DEBUG Received message: '#hubot screenshot page http://www.google.com' in channel: ****, from: ******
[Wed Apr 11 2018 16:16:47 GMT+0000 (UTC)] DEBUG Message '#hubot screenshot page http://www.google.com' matched regex //^\s*[#]?hubot[:,]?\s*(?:screenshot page (\S*)?( at )?(\S*)?)/i/; listener.options = { id: null }
[Wed Apr 11 2018 16:16:47 GMT+0000 (UTC)] DEBUG Executing listener callback for Message '#hubot screenshot page http://www.google.com'
[Wed Apr 11 2018 16:16:47 GMT+0000 (UTC)] DEBUG Sending to *****: Acquiring screenshot of www.google.com
You can use curl command which can be called using child_process to upload a file in the channel.
curl -F file=#dramacat.gif -F channels=C024BE91L,#general -F token=xxxx-xxxxxxxxx-xxxx https://slack.com/api/files.upload
It seems the formData property in your opts variable should be slightly different like this:
formData: {
token: process.env.HUBOT_SLACK_TOKEN,
title: "Screenshot of #{domain}",
filename: "image.png",
filetype: "auto",
channels: channel_id,
file: fs.createReadStream("path_to_your_image"),
}
The channel_id is your slack channel id which you can see in the browser address bar when you access the channel.
I'm trying to use Elixir to access Azure Storage Services via their REST API but I'm having difficulty getting the Authentication Header to work. I am able to connect if I use the ex_azure package (wrapper for erlazure) but not when I try to build the request and use HTTPoison.
Most Recent Error Messages
<?xml version=\"1.0\" encoding=\"utf-8\"?>
<Error>
<Code>AuthenticationFailed</Code>
<Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:00000000-0000-0000-0000-000000000000\nTime:2017-08-02T21:46:08.6488342Z</Message>
<AuthenticationErrorDetail>The MAC signature found in the HTTP request '<signature>' is not the same as any computed signature. Server used following string to sign: 'GET\n\n\nWed, 02 Aug 2017 21:46:08
GMT\nx-ms-date-h:Wed, 02 Aug 2017 21:46:08 GMT\nx-ms-version-h:2017-05-10\n/storage_name/container_name?comp=list'.</AuthenticationErrorDetail>
</Error>
After 1st Edit
<?xml version=\"1.0\" encoding=\"utf-8\"?>
<Error>
<Code>AuthenticationFailed</Code>
<Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:00000000-0000-0000-0000-000000000000\nTime:2017-08-03T03:03:57.1385277Z</Message>
<AuthenticationErrorDetail>The MAC signature found in the HTTP request '<signature>' is not the same as any computed signature. Server used following string to sign: 'GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Thu, 03 Aug
2017 03:03:57 GMT\nx-ms-version:2017-04-17\n/storage_name/container_name\ncomp:list\nrestype:container'.</AuthenticationErrorDetail>
</Error>
Dependencies
# mix.exs
defp deps do
{:httpoison, "~> 0.12"}
{:timex, "~> 3.1"}
end
Code
Am I formatting the Authentication Header (string_to_sign) right?
Am I using encode/decode right?
Am I adding headers correctly to HTTPoison?
Should I be using something else for REST actions instead of HTTPoison?
# account credentials
storage_name = "storage_name"
container_name = "container_name"
storage_key = "storage_key"
storage_service_version = "2017-04-17" # fixed version
request_date =
Timex.now
|> Timex.format!("{RFC1123}") # Wed, 02 Aug 2017 00:52:10 +0000
|> String.replace("+0000", "GMT") # Wed, 02 Aug 2017 00:52:10 GMT
# set canonicalized headers
x_ms_date = "x-ms-date:#{request_date}"
x_ms_version = "x-ms-version:#{storage_service_version}"
# assign values for string_to_sign
verb = "GET\n"
content_encoding = "\n"
content_language = "\n"
content_length = "\n"
content_md5 = "\n"
content_type = "\n"
date = "\n"
if_modified_since = "\n"
if_match = "\n"
if_none_match = "\n"
if_unmodified_since = "\n"
range = "\n"
canonicalized_headers = "#{x_ms_date}\n#{x_ms_version}\n"
canonicalized_resource = "/#{storage_name}/#{container_name}\ncomp:list\nrestype:container" # removed timeout. removed space
# concat string_to_sign
string_to_sign =
verb <>
content_encoding <>
content_language <>
content_length <>
content_md5 <>
content_type <>
date <>
if_modified_since <>
if_match <>
if_none_match <>
if_unmodified_since <>
range <>
canonicalized_headers <>
canonicalized_resource
# decode storage_key
{:ok, decoded_key} =
storage_key
|> Base.decode64
# sign and encode string_to_sign
signature =
:crypto.hmac(:sha256, decoded_key, string_to_sign)
|> Base.encode64
# build authorization header
authorization_header = "SharedKey #{storage_name}:#{signature}"
# build request and use HTTPoison
url = "https://storage_name.blob.core.windows.net/container_name?restype=container&comp=list"
headers = [ # "Date": request_date,
"x-ms-date": request_date, # fixed typo
"x-ms-version": storage_service_version, # fixed typo
# "Accept": "application/json",
"Authorization": authorization_header]
options = [ssl: [{:versions, [:'tlsv1.2']}], recv_timeout: 500]
HTTPoison.get(url, headers, options)
Notes
Some sources I used/tried...
Authentication for the Azure Storage Services
The MAC signature found in the HTTP request is not the same as any computed signature azure integration using php
How to access rest azure blob using cURL
Accessing Azure blob storage using bash, curl
A few issues I noticed:
You included Date request header in your request but it is not included in your string_to_sign. Either include this header in your string_to_sign or remove this header from request headers.
You included timeout:30 in your canonicalized_resource but it is not included in your request URL. Again, either add timeout=30 in your request querystring or remove timeout:30 from canonicalized_resource.
I have not used Elixir as such so I don't know how request headers work there, but you're naming your request headers as x-ms-date-h and x-ms-version-h. Shouldn't they be x-ms-date and x-ms-version respectively?
I am planning to use S3 Cloudstorage in IBM Bluemix but then one strange thing I found is that there is no way to add the custom META-DATA to the objects which are stored in S3 bucket.
Is there a way I can add custom Meta-Data to the objects and if yes then can you please advise on how we can add it and access it.?
Thanks for pointing out a hole in the documentation!
Custom metadata is defined by passing a x-amz-meta-{key} header with a {value} value. As an example request:
PUT /{bucket-name}/{object-name} HTTP/1.1
Authorization: {authorization-string}
x-amz-meta-foo: bar
x-amz-date: 20160825T183001Z
x-amz-content-sha256:{hashed-body}
Content-Type: text/plain; charset=utf-8
Host: s3-api.us-geo.objectstorage.softlayer.net
Content-Length: 18
{
"foo": "bar"
}
A HEAD request to check the metadata would look like:
HEAD /{bucket-name}/{object-name} HTTP/1.1
Authorization: {authorization-string}
x-amz-date: 20160825T183244Z
Host: s3-api.us-geo.objectstorage.softlayer.net
And respond with:
HTTP/1.1 200 OK
Date: Thu, 25 Aug 2016 18:32:44 GMT
X-Clv-Request-Id: da214d69-1999-4461-a130-81ba33c484a6
Accept-Ranges: bytes
Server: Cleversafe/3.9.1.102
X-Clv-S3-Version: 2.5
ETag: {MD5-hash}
Content-Type: text/plain; charset=UTF-8
x-amz-meta-foo: bar
Last-Modified: Thu, 25 Aug 2016 17:49:06 GMT
Content-Length: 18
Using the CLI, the syntax would be:
$ aws --endpoint-url=https://{endpoint} s3 cp ~/new-file s3://bucket-1/ --metadata foo=bar
Hope that helps!
This is possible. I am using this every day . Adding meta data then sending the meta data in database by doing cron calls.
Here is a small example of python script to create/add/change metadata for a a list object :
import sys
import os
import boto3
import pprint
from boto3 import client
from botocore.utils import fix_s3_host
param_1= YOUR_ACCESS_KEY
param_2= YOUR_SECRETE_KEY
param_3= YOUR_END_POINT
param_4= YOUR_BUCKET
#Create the S3 client
s3ressource = client(
service_name='s3',
endpoint_url= param_3,
aws_access_key_id= param_1,
aws_secret_access_key=param_2,
use_ssl=True,
)
# Building a list of object per bucket
def BuildObjectListPerBucket (variablebucket):
global listofObjectstobeanalyzed
listofObjectstobeanalyzed = []
extensions = ['.jpg','.png']
for key in s3ressource.list_objects(Bucket=variablebucket)["Contents"]:
#print (key ['Key'])
onemoreObject=key['Key']
if onemoreObject.endswith(tuple(extensions)):
listofObjectstobeanalyzed.append(onemoreObject)
else :
s3ressource.delete_object(Bucket=variablebucket,Key=onemoreObject)
return listofObjectstobeanalyzed
# for a given existing object, create metadata
def createmetdata(bucketname,objectname):
s3ressource.upload_file(objectname, bucketname, objectname, ExtraArgs={"Metadata": {"metadata1":"ImageName","metadata2":"ImagePROPERTIES" ,"metadata3":"ImageCREATIONDATE"}})
# for a given existing object, add new metadata
def ADDmetadata(bucketname,objectname):
s3_object = s3ressource.get_object(Bucket=bucketname, Key=objectname)
k = s3ressource.head_object(Bucket = bucketname, Key = objectname)
m = k["Metadata"]
m["new_metadata"] = "ImageNEWMETADATA"
s3ressource.copy_object(Bucket = bucketname, Key = objectname, CopySource = bucketname + '/' + objectname, Metadata = m, MetadataDirective='REPLACE')
# for a given existing object, update a metadata with new value
def CHANGEmetadata(bucketname,objectname):
s3_object = s3ressource.get_object(Bucket=bucketname, Key=objectname)
k = s3ressource.head_object(Bucket = bucketname, Key = objectname)
m = k["Metadata"]
m.update({'watson_visual_rec_dic':'ImageCREATIONDATEEEEEEEEEEEEEEEEEEEEEEEEEE'})
s3ressource.copy_object(Bucket = bucketname, Key = objectname, CopySource = bucketname + '/' + objectname, Metadata = m, MetadataDirective='REPLACE')
def readmetadata (bucketname,objectname):
ALLDATAOFOBJECT = s3ressource.get_object(Bucket=bucketname, Key=objectname)
ALLDATAOFOBJECTMETADATA=ALLDATAOFOBJECT['Metadata']
print ALLDATAOFOBJECTMETADATA
# create the list of object on a per bucket basis
BuildObjectListPerBucket (variablebucket)
# Call functions to see the results
for objectitem in listofObjectstobeanalyzed:
readmetadata(param_4,objectitem)
createmetdata(param_4,objectitem)
readmetadata(param_4,objectitem)
ADDmetadata(param_4,objectitem)
readmetadata(param_4,objectitem)
CHANGEmetadata(param_4,objectitem)
readmetadata(param_4,objectitem)
I have written a socket program in C. I used this program as a chat server/client using TCP. I tried to change the chat server to use it as a HTTP server by changing the port to 80. I referred to the HTTP request/response format in http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Example_session , and made my program to reply with sample response. I tried the url
http://127.0.0.1/
in browser. My program read the request and replied with response. At first, I used google-chrome. Chrome didn't load the page correctly until i added the correct data length in Content-Length header. After setting content length header, chrome loaded the page correctly. But, firefox doesn't load the page. Firefox doesn't showed any errors, but still loading the page like it is still waiting for some data. Only When i stop the server or close the socket, complete page is loaded. I tried to follow rfc2616 https://www.rfc-editor.org/rfc/rfc2616 , and made the response exactly , but the still the results are same.
Request:
GET / HTTP/1.1\r\nHost: 127.0.0.1:8080\r\nUser-Agent: Mozilla/5.0
(X11; Ubuntu; Linux x86_64; rv:33.0) Gecko/20100101
Firefox/33.0\r\nAccept:
text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8\r\nAccept-Language:
en-US,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection:
keep-alive\r\n\r\n
For the above request, my program write to the socket with following response & content.
Response:
HTTP/1.1 200 OK\r\nCache-Control : no-cache, private\r\nContent-Length
: 107\r\nDate : Mon, 24 Nov 2014 10:21:21 GMT\r\n\r\n
Content:
<html><head><title></title></head><body>TIME : 1416824843 <br>DATE: Mon Nov 24 15:57:23 2014 </body></html>
This response is loading in Chrome, but not in firefox. Chrome is loading the page instantly whereas firefox is waiting for data. Note that the data length 107 is specified in the header. I donot have any addons enabled in firefox. My firefox version is in the request. Chrome version: Version 38.0.2125.111 (64-bit).
Code:
void *socket_read(void *args)
{
int socket,*s,length;
char buf[1024];
s=(int *)args;
socket=*s;
while(1){
buf[0]='\0';
length=read(socket,buf,1024);
if(length==0)
break;
else if(length==-1){
perror("Read");
return;
}
else{
printf("Request: %s\n",buf);
send_response(socket);
}
}
printf("End of read thread [%d]\n",socket);
}
int start_accept(int port)
{
int socket,csocket;
pthread_t thread;
struct sockaddr_in client;
socklen_t addrlen=sizeof(client);
pthread_attr_t attr;
socket=create_socket(port);
while(1){
if((csocket=accept(socket,(struct sockaddr *)&client,&addrlen))==-1)
{
perror("Accept");
break;
}
pthread_attr_init(&attr);
if(0!=pthread_create(&thread,&attr,socket_read,&csocket))
{
perror("Read thread");
return;
}
usleep(10000);
}
}
void send_response(int socket)
{
char buf1[1024];
int content_length;
char buf2[1024]="<html><head><title></title></head><body>TIME : 1416824843 <br>DATE: Mon Nov 24 15:57:23 2014 </body></html>";
content_length=strlen(buf2);
sprintf(buf1,"HTTP/1.1 200 OK\r\nCache-Control : no-cache, private\r\nContent-Length : %d\r\nDate : Mon, 24 Nov 2014 12:03:43 GMT\r\n\r\n",content_length);
printf("Written: %d \n",write(socket,buf1,strlen(buf1)));
fflush(stdout);
printf("Written: %d \n",write(socket,buf2,content_length));
fflush(stdout);
}
I have found the problem.
The Response is incorrect. There should not be any spaces between the header field name and colon(':'). Found this in http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2 .
My correct response is
HTTP/1.1 200 OK\r\nCache-Control: no-cache, private\r\nContent-Length: 107\r\nDate: Mon, 24 Nov 2014 10:21:21 GMT\r\n\r\n
I had put a space between 'Content-Length' and ':' . That's the reason Firefox ignored the content length header and reading the socket. Chrome accepted the header fields with spaces, so the problem didn't occurred in chrome.
After removing the space, program works fine.
It actually loads the page. If you add content-type header you will see the HTML page (Content-Type: text/html; charset=UTF-8\r\n)
Anyway, both in Chrome and in Firefox you will see the connections never stops because the server doesn't close the socket. If you closed csocket, you would see the HTML page in both browsers but as you said it should be a persistent connection.