Following the installation of RestBase using standard config, I have a working version of summary API.
The problem that the caching mechanism seems strange to me.
The piece of code would decide whether to look at a table cache for fast response. But I cannot make it a server-cache depend on some time-constrain (max-age when the cache is written for example). It means that the decision to use cache or not entirely depend on clients.
Can someone explain the workflow of RestBase caching mechanism?
// Inside key.value.js
getRevision(hyper, req) {
//This one get the header from client request and decide to use cache
or not depend on the value. Does it mean server caching is non-existent?
if (mwUtil.isNoCacheRequest(req)) {
throw new HTTPError({ status: 404 });
}
//If should use cache, below run
const rp = req.params;
const storeReq = {
uri: new URI([rp.domain, 'sys', 'table', rp.bucket, '']),
body: {
table: rp.bucket,
attributes: {
key: rp.key
},
limit: 1
}
};
return hyper.get(storeReq).then(returnRevision(req));
}
Cache invalidation is done by the change propagation service, which is triggered on page edits and similar events. Cache control headers are probably set in the Varnish VCL logic. See here for a full Wikimedia infrastructure diagram - it is outdated but gives you the generic idea of how things are wired together.
Related
All the logic get failed because chrome extensions block one of my js files,
Is there a way to make pre-caching more robust? even if I get an error when caching some files, I can still cache most of them correctly, and I have run time caching.
If you're using workbox-precaching, then the answer is no—it's designed so that service worker installation will only proceed if all of the items in the precache manifest were successfully added to the cache. That way, you're guaranteed to have a functional set of resources available offline. (There's a longstanding feature request for adding support for "optional" precaching, but it's not clear how that would work in practice.)
I'd recommend using runtime caching for URLs that are optional, and might be blocked by browser extensions. If you want to "warm" the cache in this scenario, and you don't care if the cache population fails, you can add your own logic along the lines of:
import {CacheFirst} from 'workbox-strategies';
import {registerRoute} from 'workbox-routing';
import {precacheAndRoute} from 'workbox-precaching';
const OPTIONAL_CACHE_NAME = 'optional-resources';
const OPTIONAL_URLS = [
// Add URLs here that might be blocked.
];
self.addEventListener('install', (event) => {
event.waitUntil((async () => {
const cache = await caches.open(RUNTIME_CACHE_NAME);
for (const url of OPTIONAL_URLS) {
try {
await cache.add(url);
} catch (e) {
// Ignore failures due to, e.g., a content blocker.
}
}
})());
});
// Precache everything in the manifest, which you need to
// configure to exclude your "optional" URLs.
precacheAndRoute(self.__WB_MANIFEST);
// Use a cache-first runtime strategy.
registerRoute(
// Check url.pathname, or url.href, if OPTIONAL_URLS
// contains full URLs.
({url}) => OPTIONAL_URLS.includes(url.pathname),
new CacheFirst({cacheName: OPTIONAL_CACHE_NAME}),
);
I have a system that processes images. Essentially, I provide an ID to it, and it fetches a source image, and then it begins performing transformations on it to resize and reformat it.
This system gets quite a bit of usage, and one of the things that I've noticed is that I tend to get many requests for the same ID simultaneously, but in different requests to the webserver.
What I'd like to do is "batch" these requests. For example, if there's 5 simultaneous requests for the image "user-upload.png", I'd like there to be only one HTTP request to fetch the source image.
I'm using NestJS with default scopes for my service, so the service is shared between requests. Requests to fetch the image are done with the HttpModule, which is using axios internally.
I only care about simultaneous requests. Once the request finishes, it will be cached, and that prevents new requests from hitting the HTTP url.
I've thought about doing something like this (Pseudocode):
#Provider()
class ImageFetcher {
// Store in flight requests as a map between id:promise
inFlightRequests = { }
fetchImage(id: string) {
if (this.inFlightRequests[id]) {
return this.inFlightRequests[id]
}
this.inFlightRequests[id] = new Promise(async (resolve, reject) => {
const { data } = await this.httpService.get('/images' + id)
// error handling omitted here
resolve(data)
delete inFlightRequests[id]
})
return this.inFlightRequests[id]
}
}
The most obvious issue I see is the potential for a memory leak. This is solveable with more custom code, but I thought I'd see if anyone has any suggestions for doing this without writing more code.
In particular, I've also thought about using an axios interceptor, but I'm not entirely sure how to handle that properly. Any pointers here would be really appreciated.
I'm in the middle of upgrading our API from Sails v0.12 -> v1, which was prompted by the use of self-validating machines for controller actions. After finally getting through a ton of headache replacing deprecated code, I've landed in a rough spot...
With v0.12 (rather, with the older "req, res" controller style), one could use custom response handlers across the board. I've taken advantage of this, and have request logging at the end of all our response types (with some additional sugaring of data). This was done to log all requests in the database, so we can get insights into what our production servers are doing (because they are load-balanced, having a central place to view this is a must, and this was an easy route to take).
So now, my problem is moving forward with "Actions2" machine-style actions. How does one use these custom response types in these things? Are we being forced to repeat ourselves in our exists? I can't find any good documentation to help guide this process, nor can I find a consistent way to "hook" into the end of a response using machines as actions. I can't find any documentation on what kind of options machines can give to Sails.
#Nelson yes, I understand that, but at the time, that isn't what I wanted at all. I wanted all of the benefits of Actions2.
EDIT: While the original, crossed-out comment below does still work, the prefered way to use Actions2 and the custom responses folder paradigm, is to do something similar to the following in an Actions2 file:
module.exports = {
friendlyName: 'Human-friendly name of function',
description: 'Long description of function and what it does.',
inputs: {
userCommand: {
type: 'string',
required: true,
description: 'Long, human-readable description of the input'
}
},
exits: {
success: {
responseType: 'chatbotResponse'
}
},
fn: async function(inputs, exits){
// do some crazy stuff with the inputs, which has already been validated.
return exits.success('Woot');
}
}
This ultimately will route through the responses/chatbotResponse.js, which looks something similar to this:
module.exports = async function chatbotResponse(data){
let res = this.res,
req = this.req;
if (!data) {
data = 'Something didn\'t go as planned...';
}
// how to call a Node Machine style helper with named inputs
await sails.helpers.finalizeRequestLog.with({req: req, res: res, body: {plainString: data}});
return res.json(data);
};
ORIGINAL:
As it turns out, in the Actions2 function, you just need to add the env param async function(inputs, exists, env). The env will give you access to the req and res. So, if you have custom responses, that perform special tasks (like request logging), you can just use return await env.res.customResponse('Hurray, you made a successful call!');
I have a project that caters for individuals with poor internet connections in predominantly rural areas. I need to allow for users to download(or any other applicable means), or fill out details offline and then when they are ready and the internet connection is ready the data filled out offline should sync with the online database and give a report.
The offline form also needs the same validation as online, to ensure no time wastage.
What are the options I know that HTML 5 has an offline application ability. I would prefer an open source option, which will allow people with intermittent internet issues to continue filling out a form or series of forms even though internet has dropped, and the data sync when internet reconnects.
So what are the best options? Having the user requiring to download a large application is also not the best case, I would prefer a browser or small download solution. Maybe even a way of downloading a validatable form in some format for re-upload.
This is something I've been muddling through myself as some of the users of the site I am currently tasked with building have poor connections or would like to fill in forms away from a network for various reasons. Depending on your precise needs and your customer's browser compatibility, the solution I've decided to go with is to use the HTML5 cache capability you mention in your post.
The amount of data stored is not that great, and it will mean that the webpage you want them to fill in is available offline.
If you couple this with the localStorage interface you can keep all form submissions until they regain connection.
As an example of my current solution:
The cache.php file, to write the manifest
<?php
header("Content-Type: text/cache-manifest");
echo "CACHE MANIFEST\n";
$pages = array(
//an array of the pages you want cached for later
);
foreach($pages as $page) {
echo $page."\n";
}
$time = new datetime("now");
//this makes sure that the cache is different when the browser checks it
//otherwise the cache will not be rebuilt even if you change a cached page
echo "#Last Build Time: ".$time->format("d m Y H:i:s T");
You can then have a simple ajax script checking for connection
setInterval( function() {
$.ajax({
url: 'testconnection.php',
type: 'post',
data: { 'test' : 'true' },
error: function(XHR, textStatus, errorThrown) {
if(textStatus === 'timeout') {
//update a global var saying connection is down
noCon = true;
}
}
});
if(hasUnsavedData) {
//using the key/value pairs in localstorage, put together a data object and ajax it into the database
//once complete, return unsavedData to false to prevent refiring this until we have new data
//also using localStorage.removeItem(key) to clear out all localstorage info
}
}, 20000 /*medium gap between calls, do whatever works best for you here*/);
Then for your form submission script, use localstorage if that noCon variable is set to true
$(/*submit button*/).on("click", function(event) {
event.preventDefault();
if(noCon) {
//go through all inputs in some way and put to localstorage, your method is up to you
$("input").each( function() {
var key = $(this).attr("name"), val = $(this).val();
localStorage[key] = val;
});
hasUnsavedData = true;
//update a global variable to let the script above know to save information
} else {
//or if there's connection
$("form").submit();
//submit the form in some manner
}
});
I've not tested every script on this page, but they're written based on the skeleton of what my current solution is doing, minus a lot of error checking etc, so hopefully it will give you some ideas on how to approach this
Suggestions for improvements are welcomed
I was wondering if there were any features hidden in Angular or exposed by some 3rd-party libraries to easily create HATEOAS-compliant Restful clients.
On backend side, I am using Spring Data/REST to produce an HATEOAS JSON API.
Consuming it, though, is quite another story.
For instance, I've got those 3 entities:
Company {name, address}
Employee {firstName, lastName, employer[Company]}
Activity {rate, day, employee[Employee], client[Company]}
and requesting an activity (the most complex entity of the model) produces something like this:
{
links: [],
content: [{
rate: 456,
day: 1366754400000,
links: [{
rel: "self",
href: "http://localhost:8080/api/activities/1"
},
{
rel: "activities.activity.client",
href: "http://localhost:8080/api/activities/1/client"
},
{
rel: "activities.activity.employee",
href: "http://localhost:8080/api/activities/1/employee"
}]
}]
}
My API talks in terms of REST (resources identified by links).
An Activity has an Employee for instance. What I really want to use is : {rate: 456, day: 1366754400000, employee: {firstName:"xxx", lastName:"xxx" ...}}.
However, as you can see in the first output, my Activity only contains a link to the employee, not its data. Is there anything in Angular or in a 3rd-party library to resolve those links and embed the resulting data instead?
Any input on this?
Thanks in advance!
Checkout angular-hateoas. ITs an AngularJS module for using $resource with a HATEOAS-enabled REST API.
You could write a Response Transformation that would inspect your returned object, check for links, and resolve them before returning the response. See the section "Transforming Requests and Responses" in the $http service documentation.
Something like this:
transformResponse: function(rawData) {
var json = JSON.parse( rawData );
forEach( json.content.links, function(link) {
// resolve link...
});
return json;
}
Since the "resolve link" step is itself an $http call, sub-references would also be resolved. HOWEVER, since these are asynchronous, you would likely return a promise instead of the real value; I don't know if the transform function is allowed to do this.
As #charlietfl pointed out, however, please note that this will result in several HTTP calls to return a single entity. Even though I like the concept of HATEOAS, this will likely result in sluggishness if too many calls are made. I'd suggest that your server return the data, or some of it, directly, PLUS the link for details.
Based on your comment about wanting to work with data as against links on the client, I think Restangular would be a good fit.
I've been using angular-hal for one of my projects. It was a Spring HATEOAS backend. And I didn't run into any issues. It handles parametrized resources. I suspect it only supports HAL so since you're using Spring Data Rest you probably have to configure it to generate HAL compliant responses.
I think the confusion may be that you are asking for an Angular solution, when what you really want to do is have Spring Data send a more complete JSON response. I.e, you really just want the server to return the employee data as part of the response JSON, rather than having the client perform extra steps to look it up. I don't know what data store you are using in Spring Data, but the general solution would be to add public getEmployee() method to your Activity class and then annotate the method with #RelatedTo and #Fetch (this would be the setup for Neo4J -- may be different annotations for your flavor of Spring Data). This should cause Spring HATEOAS to include the Employee record within the response for /activity. Hope this helps.