I've got a few thousand blobs at Rackspace's Cloud Files which I need to update content type for. However, I can't figure out how I do that using the jclouds API.
How can I go about to update metadata on an existing blob?
Assuming you have the whole set up running for your rackspace, using jclouds is easy:
First initialize with the following details:
BlobStoreContext context = ContextBuilder.newBuilder(provider)
.credentials(username, apiKey)
.buildView(BlobStoreContext.class);
BlobStore blobStore = context.getBlobStore();
You can now build a new blob to to put on rackspace:
Blob blob = blobStore.blobBuilder(key)
.userMetadata(metadata)
.payload(value)
.build();
blobStore.putBlob(container, blob);
value is the input bytes[] and metadata is a hash map of meta data associated with the blob like content type.
If you want to do operations like update:
RegionScopedBlobStoreContext context = ContextBuilder.newBuilder("openstack-swift")
.endpoint(config.getAuthUrl())
.credentials(config.getUser(), config.getPasswd())
.overrides(p)
.buildView(RegionScopedBlobStoreContext.class);
SwiftApi swift = (SwiftApi) ((org.jclouds.rest.internal.ApiContextImpl)context.unwrap()).getApi();
boolean success = swift.objectApiInRegionForContainer(config.getRegion(), container).updateMetadata(filename, metaData);
I know it is an overview but I hope it gives you a good direction.
As of jclouds 2.1.0 (and 1.9.3 at least) the API to change object custom metadata looks like this:
BlobStoreContext context = contextBuilder.buildView(BlobStoreContext.class);
SwiftApi api = (SwiftApi) ((org.jclouds.rest.internal.ApiContextImpl)context.unwrap()).getApi();
ObjectApi objectApi = api.getObjectApi(region, container);
Map<String, String> meta = new HashMap<>();
meta.put('some-meta', value);
objectApi.updateMetadata(blobName, meta);
Content type cannot be updated this way, only metadata with keys starting from X-Object-Meta- can be updated. updateMetadata automatically prefixes all keys passed to it with X-Object-Meta-. In the example above custom data with key X-Object-Meta-some-meta would be added to the blob.
Theoretically updateRawMetadata should be able to update content type (it does not add X-Object-Meta- prefix to keys and passes them verbatim) but due to a bug in jclouds it fails for content type key with error:
configuration error please use request.getPayload().getContentMetadata().setContentType(value) as opposed to adding a content type header
I've checked update content type via curl and it works fine, so it is a bug in jclouds:
curl -X POST -H "X-Auth-Token: $TOKEN" -H "Content-Type: $CONTENT_TYPE" "$PUBLIC_URL/$CONTAINER/$BLOB_NAME"
The workaround for this is to use copy operation to copy the blob into itself as described in the documentation for the API:
You can use COPY as an alternate to the POST operation by copying to the same object
And this can be done using vendor/api independent jclouds API like this:
Blob blob = blobStore.getBlob(container, blobName);
MutableContentMetadata contentMetadata = blob.getPayload().getContentMetadata();
contentMetadata.setContentType(mimeType);
blobStore.copyBlob(getContainer, blobName, getContainer, blobName,
CopyOptions.builder().contentMetadata(contentMetadata).build());
Or via SwiftApi (this does not require fetching of the blob's metadata):
BlobStoreContext context = contextBuilder.buildView(BlobStoreContext.class);
SwiftApi api = (SwiftApi) ((org.jclouds.rest.internal.ApiContextImpl)context.unwrap()).getApi();
ObjectApi objectApi = api.getObjectApi(region, container);
Map<String, String> meta = new HashMap<>();
meta.put(HttpHeaders.CONTENT_TYPE, mimeType);
objectApi.copy(blobName, container, blobName, new HashMap<String, String>(), meta);
Related
I have a dashboard named as "server-plots" and there is another dashboard named as "master-plots". panels under "master-plots" are most updated graphs and I want to add the new panels inside "master-plots" dashboard to "server-plots" as well, everything with Python code (not manually or using curl).
I am able to programmatically take the backup of these plots using Grafana GET APIs ,as JSON. I want to find the new panels inside the "master-plots" dashboard JSON and add those into "server-plots" dashboard , all using Python. I am unable to find any API to do that. Any idea how can I achieve this?
The way I achieved this was by taking the backup of the master-plots as JSON using:
/api/dashboards/uid/<uid of the dashboard>
Then comparing it with the one inside the server-plots (taken similarly), and then updating the server-plots JSON with the diff (basically replacing server-plots JSON with the master-plots JSON), and finally writing that to the following using the POST method:
/api/dashboards/db/
One thing to consider here: The new JSON which is being written into the server-plots should have different uid and overwrite=True:
--snip--
f = open(myjsonfile,) # the updated JSON of server-plots
data = json.load(f)
data["dashboard"]["id"] = None
data["overwrite"] = True
data["folderId"] = 123 # this is the folder ID of the server-plots
data["dashboard"]["uid"] = <logic to generate randum alpha-num>
url = "https://mygrafanaurl.com/api/dashboards/db"
headers = {'Authorization': 'auth', 'Content-Type': 'application/json' }
response = requests.request("POST", url, headers=headers, data=json.dumps(data))
I have created a method using GETFILE() service of azure. Reference: https://learn.microsoft.com/en-us/rest/api/storageservices/get-file
public void getImage(){
string storageKey = 'xxxxStorageKeyxxx';
string storageName = '<storageName>';
Datetime dt = Datetime.now();
string formattedDate = dt.formatGMT('EEE, dd MMM yyyy HH:mm:ss')+ ' GMT';
string CanonicalizedHeaders = 'x-ms-date:'+formattedDate+'\nx-ms-version:2016-05-31';
string CanonicalizedResource = '/' + storageName + '/<shareName>/<dirName>/<File Name>\ntimeout:20';
string StringToSign = 'GET\n\n\n\n\napplication/octet-stream\n\n\n\n\n\n\n' + CanonicalizedHeaders+'\n'+CanonicalizedResource;
Blob temp = EncodingUtil.base64Decode(storageKey);
Blob hmac = Crypto.generateMac('HmacSHA256',Blob.valueOf(StringToSign),temp ); //StringToSign
system.debug('oo-'+EncodingUtil.base64Encode(hmac));
HttpRequest req = new HttpRequest();
req.setMethod('GET');
req.setHeader('x-ms-version','2016-05-31' );
req.setHeader('x-ms-date', formattedDate);
req.setHeader('content-type','application/octet-stream');
string signature = EncodingUtil.base64Encode(hmac);
string authHeader = 'SharedKey <storageName>'+':'+signature;
req.setHeader('Authorization',authHeader);
req.setEndpoint('https://<storageName>.file.core.windows.net/<shareName>/<dirName>/<file Name>?timeout=20');
Http http = new Http();
HTTPResponse res;
res = http.send(req);
}
The above was working fine and giving the 200 as response code. But, my main goal is to display/download the respective image which i retrieved through REST API. How can i achieve that?
So a few things before I answer your question:
File storage is not really suitable for what you're trying to accomplish (it's possible though).
You should look at Blob storage for this as blob storage is more suitable for this kind of scenario.
Assuming you go with Blob storage, there are a few things you could do:
If the blob container (equivalent to a share in file storage) has an ACL is Blob or Container (i.e. blobs in a container are publicly available), you could simply return the blob's URL (Same is your request URL in code above) in your response and then create a link in your application with href set to this URL.
If the blob container has an ACL as Private (i.e. blobs are not publicly available), you would need to create a Shared Access Signature (SAS) token on that blob with at least Read permission and then create a SAS URL. A SAS URL is simply blob URL + SAS token and return this SAS URL in your response and then create a link in your application with href set to this URL.
Since an Azure File Share is always private, if you were to use Azure File service to serve a file, you would do the same thing as 2nd option I listed above. You will create a SAS token on the file with at least Read permission and then return the SAS URL in the response and then create a link in your application with href set to this URL.
To read about Shared Access Signature, you may find this link helpful: https://learn.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1.
To create a Shared Access Signature using REST API, you may find this link helpful: https://learn.microsoft.com/en-us/rest/api/storageservices/Constructing-a-Service-SAS?redirectedfrom=MSDN
I'm using Google Endpoint and for one of my entities I want to create a POST request that adds a map of properties. What is the right way to do it?
I know Google Endpoint can receive a Collection as a parameter, but I want to add a map (unknown key values).
Should I pass a JSON as a parameter or just add the JSON in the body of the request and extract it from the HttpServletRequest object?
I would avoid passing it as a parameter. You can send it in the body of the request and then use json library to get a python object.
https://docs.python.org/3/library/json.html
Every JSON object is a map, so it looks like the most obvious choice. GSON makes it easy, but you can use other parsers too.
Type type = new TypeToken<Map<String, String>>(){}.getType();
Map<String, String> myMap = gson.fromJson("{'k1':'apple','k2':'orange'}", type);
I am building a REST API and facing this issue: How can REST API pass very large JSON?
Basically, I want to connect to Database and return the training data. The problem is in Database I have 400,000 data. If I wrap them into a JSON file and pass through GET method, the server would throw Heap overflow exception.
What methods we can use to solve this problem?
DBTraining trainingdata = new DBTraining();
#GET
#Produces("application/json")
#Path("/{cat_id}")
public Response getAllDataById(#PathParam("cat_id") String cat_id) {
List<TrainingData> list = new ArrayList<TrainingData>();
try {
list = trainingdata.getAllDataById(cat_id);
Gson gson = new Gson();
Type dataListType = new TypeToken<List<TrainingData>>() {
}.getType();
String jsonString = gson.toJson(list, dataListType);
return Response.ok().entity(jsonString).header("Access-Control-Allow-Origin", "*").header("Access-Control-Allow-Methods", "GET").build();
} catch (SQLException e) {
logger.warn(e.getMessage());
}
return null;
}
The RESTful way of doing this is to create a paginated API. First, add query parameters to set page size, page number, and maximum number of items per page. Use sensible defaults if any of these are not provided or unrealistic values are provided. Second, modify the database query to retrieve only a subset of the data. Convert that to JSON and use that as the payload of your response. Finally, in following HATEOAS principles, provide links to the next page (provided you're not on the last page) and previous page (provided you're not on the first page). For bonus points, provide links to the first page and last page as well.
By designing your endpoint this way, you get very consistent performance characteristics and can handle data sets that continue to grow.
The GitHub API provides a good example of this.
My suggestion is no to pass the data as a JSON but as a file using multipart/form-data. In your file, each line could be a JSON representing a data record. Then, it would be easy to use a FileOutputStream to receive te file. Then, you can process the file line by line to avoid memory problems.
A Grails example:
if(params.myFile){
if(params.myFile instanceof org.springframework.web.multipart.commons.CommonsMultipartFile){
def fileName = "/tmp/myReceivedFile.txt"
new FileOutputStream(fileName).leftShift(params.myFile.getInputStream())
}
else
//print or signal error
}
You can use curl to pass your file:
curl -F "myFile=#/mySendigFile.txt" http://acme.com/my-service
More details on a similar solution on https://stackoverflow.com/a/13076550/2476435
HTTP has the notion of chunked encoding that allows you send a HTTP response body in smaller pieces to prevent the server from having to hold the entire response in memory. You need to find out how your server framework supports chunked encoding.
I have a class that models exactly the entity I have in the database. I have a stored procedure that takes in parameters for a new row and returns all the settings in the table which in turn populates my repository. I am able to see the results of GET, PUT and DELETE in the List of type Setting that is in memory. I am noticing first that even when I close Visual Studio and reopen and run the project, sometimes, the List is still in the state it was before. It is not repopulating from the database so I'm not sure why that is first of all... Secondly, I can't seem to get POST to work from Fiddler unlike the other HTTP verbs. I DO see the values from Fiddler show up in the code below but I get the error: Invalid URI: The format of the URI could not be determined. I get the same error if I pass an ID or not.
Here is what I put into Fiddler:
POST localhost:54852/api/settings
Request Headers
User-Agent: Fiddler
Content-type: application/x-www-form-urlencoded
Host: localhost:54852
Content-Length: 149
Request Body
ID=0&Category=Dried%20Goods&Sub_Category=Other&UnitSize=99&UnitOfMeasureID=999&Facings=true&Quantity=true&EverydayPrice=999.99&PromotionPrice=111.11
PostSetting function within my SettingsController
public HttpResponseMessage PostSetting(Setting item)
{
item = repository.Add(item);
var response = new HttpResponseMessage<Setting>(item) { StatusCode = HttpStatusCode.Created };
string uri = Url.Route("DefaultApi", new { id = item.ID });
response.Headers.Location = new Uri(uri);
return response;
}
Should I create a new procedure that gets the MAXID from the database and use that as the NEW ID in the line above where a new ID is created?
You need to create a JSON representation of the Setting class or item that you are wanting to test with use Fiddler (now a Telerik product) and use the Composer tab.
Next you will want to perform a POST to the following URL:
http://[your base url]/api/settings
and pass the JSON formatted setting class.
You can see an example of this here: ASP.NET Web API - Scott Hanselman
Here is a short video showing how to achieve it easily
get and post to webapi from fiddler