TL;DR
What's the best way to handle dependency between types of data that is loaded asynchronously from different backend endpoints?
Problem
My app fetches data from a backend, for each entity I have an endpoint to fetch all instances.
For example api.myserver.com/v1/users for User model and api.myserver.com/v1/things for Thing model.
This data is parsed and placed into data store objects (e.g. UserDataStore and ThingDataStore) that serve these models to the rest of the app.
Question
What should I do if the data that comes from /things depends on data that comes from /users and the fetch operations are async. In my case /things returns the id of a user that created them. This means that if /things returns before /users, then I won't have enough data to create the Thing model.
Options
Have /things return also relevant /users data nested.
This is bad because:
I'll then have multiple model instances User for the same actual user - one that came from /users and one that came nested in /things.
Increases the total payload size transferred.
In a system with some permission policy, data that is returned for /users can be different to /things, and then it'll allow partially populated models to be in the app.
Create an operational dependency between the two data stores, so that ThingsDataStore will have to wait for UserDataStore to be populated before it attempts to load its own data.
This is also bad because:
Design-wise this dependency is not welcome.
Operational-wise, it will very quickly become complicated once you throw in another data stores (e.g. dependency cycles, etc).
What is the best solution for my problem and in general?
This is obviously not platform / language dependent.
I see two possible solutions:
Late initialization of UserDataStore in ThingDataStore. You will have to allow for creation an object that is not fully valid. And you will also need to add method that will give you an information whether UserDataStore is initialized or not. Not perfect, because for some time there will exists an invalid instance.
Create some kind of proxy or maybe a buider object for ThingDataStore that will hold all information about particular thing and will create ThingDataStore object as soon as UserDataStore related with this instance will be received.
Maybe it will help you. Good luck!
Related
I am using Sitecore 8.1 with xDB enabled (MongoDB). I would like to store the user-roles of the visiting users in the xDB, so I can aggregate on these data in my reports. These roles can change over time, so one user could have one set of roles at some point in time and another set of roles at a later time.
I could go and store these user-roles as custom facets on the Contact entity, but as they may change for a user from visit to visit, I will loose historical data if I update the data in the facet every time the user log in (fx. I will not be able to tell which roles a given user had, at some given visit).
Instead, I could create a custom IElement for my facet data, and store the roles along with a timestamp saying when the given roles were registered, but this model may be hard to handle during the reporting phase, where I would need to connect the interaction data with the role-data based on timestamps every time I generate a report.
Is it possible to store these custom data in the xDB in something else than the Contact collection? Can I store custom data in the Interactions collection? There is a property called Tracker.Current.Session.Interaction.CustomValues which sounds like what I need, but if I store data here, will I be able to perform proper aggregation/reporting on the data? Any other approaches I haven't thought about?
CustomValues
Yes, the CustomValues dictionary is what I would use in your case. This dictionary will get serialized to MongoDB as a nested document of every interaction (unless the dictionary is empty).
Also note that, since CustomValues is a member of the base class Sitecore.Analytics.Model.Entity, this dictionary is available in many other data classes of xDB. For example, you can store custom values in PageData and PageEventData objects.
Since CustomValues takes an object of any class, your custom data class needs some extra things for it to be successfully saved to and subsequently loaded from MongoDB:
It has to be marked as [Serializable].
It needs to be registered in the MongoDB driver like this:
using Sitecore.Analytics.Data.DataAccess.MongoDb;
// [...]
MongoDbObjectMapper.Instance.RegisterModelExtension<YourCustomClassName>();
This needs to be done only once per application lifetime - for example, in an initialize pipeline processor.
Your own storage
Of course, you don't have to use Sitecore's API to store your custom data. So the alternative would be to manually save data to a custom MongoDB collection or an SQL table. You can then read that data in your aggregation processor, finding it by the ID of currently processed interaction.
The benefit of this approach is that you can decide where and how your data is stored. The downside is extra work of implementing and maintaining this data storage.
I have the following URI: /articles/:id, where article is a resource on web-service and have associated model/class. Now I need to return only partial data for each resource (to save bandwidth and make for speed) when collection is requested, but when a single item is requested from collection I need to send full data. My question is should I use two models/classes for the same resource on the server and initiate different one depending on collection or single resource is requested? Or maybe there is should be only one model/class but not all fields should be filled with data when a collection is requested? Or maybe there is another approach?
I suggest using the approach suggested here with a fields query parameter.
If the API is going to be open to everyone to use and client usage is going to be unpredictable, then by default you probably need to limit the fields that you return. Just make sure you document in some way all the possible fields that could be used, in case a client actually needs them.
If the API is going to be consumed only by an app or apps you made, then by default you could return all of the fields and then your app can pass that fields parameter to speed things up.
I have a form where users create Person records. Each Person can have several attributes -- height, weight, etc. But they can also have lists of associated data such as interests, favorite movies, etc.
I have a single form where all this data is collected. To me it seems like I should POST all of this data in a single request. But is that RESTful? My reading suggests that the interests, favorite movies and other lists should be added in separate POST requests. But I don't think that makes sense because one of those could fail and then there would be a partial insert of the Person and it may be missing their interests or favorite movies.
I'd say that it depends entirely upon the addressability and uniqueness of the dependent data.
If your user-associated data is dependent upon the user (i.e., a "distinct" string, e.g. an attribute such as a string representing an (unvalidated) name of a movie), then it should be included in the POST creation of the user representation; however, if the data is independent of the user (where the data can be addressed independently of the user, e.g. a reference, such as a movie from a set of movies) then it should be added independently.
The reasoning behind this is that reference addition when bundled with the original POST implies transactionality; that is, if another user deletes the movie reference for the "favorite" movie between when it is chosen on the client and when the POST goes through, the user add will (should by that design) fail, whereas if the "favorite" movie is not associative but is just an attribute, there's nothing to fail on (attributes (presumably) cannot be invalidated by a third party).
And again, this goes very much to your specific needs, but I fall on the side of allowing the partial inserts and indicating the failures. The proper way to handle this sort of thing if you really want to not allow partial inserts is to just implement transactions on the back end; they're the only way to truly handle a situation where a critical associated resource is removed mid-process.
The real restriction in REST is that for a modifiable resource that you GET, you can also turn around and PUT the same representation back to change its state. Or POST. Since it's reasonable (and very common) to GET resources that are big bundles of other things, it's perfectly reasonable to PUT big bundles of things, too.
Think of resources in REST very broadly. They can map one-to-one with database rows, but they don't have to. An addressable resource can embed other addressable resources, or include links to them. As long as you're honoring your representation and the semantics of the underlying protocol's operations (i.e. HTTP GET POST PUT etc.), REST doesn't have anything to say about other design considerations that might make your life easier or harder.
I don't think there is a problem with adding all data in one request as long as its inherently associated with the main resource (i.e. the person in your case). If interest, fav. movies etc are resources of their own, they should also be handled as such.
I am currently designing the API for an existing PHP application, and to this end am investigating REST as a sensible architectural approach.
I believe I have a reasonable grasp of the key concepts, but I'm struggling to find anybody that has tackled object hierarchies and REST.
Here's the problem...
In the [application] business object hierarchy we have:
Users
L which have one-to-many Channel objects
L which have one-to-many Member objects
In the application itself we use a lazy load approach to populate the User object with arrays of these objects as required. I believe in OO terms this is object aggregation, but I have seen various naming inconsistencies and do not care to start a war about the precise naming convention </flame war>.
For now, consider I have some loosely coupled objects that I may / may not populate depending on application need.
From a REST perspective, I am trying to ascertain what the approach should be. Here is my current thinking (considering GET only for the time being):
Option 1 - fully populate the objects:
GET api.example.com/user/{user_id}
Read the User object (resource) and return the User object with all possible Channel and Member objects pre-loaded and encoded (JSON or XML).
PROS: reduces the number of objects, no traversal of object hierarchies required
CONS: objects must be fully populated (expensive)
Option 2 - populate the primary object and include links to the other object resources:
GET api.example.com/user/{user_id}
Read the User object (resource) and return the User object User data populated and two lists.
Each list references the appropriate (sub) resource i.e.
api.example.com/channel/{channel_id}
api.example.com/member/{member_id}
I think this is close to (or exactly) the implications of hypermedia - the client can get the other resources if it wants (as long as I tag them sensibly).
PROS: client can choose to load the subordinates or otherwise, better separation of the objects as REST resources
CONS: further trip required to get the secondary resources
Option 3 - enable recursive retrieves
GET api.example.com/user/{user_id}
Read the User object and include links to lists of the sub-objects i.e.
api.example.com/user/{user_id}/channels
api.example.com/user/{user_id}/members
the /channels call would return a list of channel resources in the form (as above):
api.example.com/channel/{channel_id}
PROS: primary resources expose where to go to get the subordinates but not what they are (more RESTful?), no requirement to get the subordinates up front, the subordinate list generators (/channels and /members) provide interfaces (method like) making the response more service like.
CONS: three calls now required to fully populate the object
Option 4 - (re)consider the object design for REST
I am re-using the [existing] application object hierarchy and trying to apply it to REST - or perhaps more directly, provide an API interface to it.
Perhaps the REST object hierarchy should be different, or perhaps the new RESTful thinking is exposing limitations of the existing object design.
Any thoughts on the above welcomed.
There's no reason not to combine these.
api.example.com/user/{user_id} – return a user representation
api.example.com/channel/{channel_id} – return a channel representation
api.example.com/user/{user_id}/channels – return a list of channel representations
api.example.com/user/{user_id}/channel_list – return a list of channel ids (or links to their full representations, using the above links)
When in doubt, think about how you would display the data to a human user without "API" concerns: a user wants both index pages ({user_id}/channel_list) and full views ({user_id}/channels).
Once you have that, just support JSON instead of (or in addition to) HTML as the representation format, and you have REST.
The best advice I can give is to try and avoid thinking about your REST api as exposing your objects. The resources you create should support the use cases you need. If necessary you might create resources for all three options:
api.example.com/completeuser/{id}
api.example.com/linkeduser/{id}
api.example.com/lightweightuser/{id}
Obviously my names are a bit goofy, but it really doesn't matter what you call them. The idea is that you use the REST api to present data in the most logical way for the particular usage scenario. If there are multiple scenarios, create multiple resources, if necessary. I like to think of my resources more like UI models rather than business entities.
I would recommend Restful Obects which is standards for exposing domain model's restful
The idea of Restful Objects is to provide a standard, generic RESTful interface for domain object models, exposing representations of their structure using JSON and enabling interactions with domain object instances using HTTP GET, POST, PUT and DELETE.
According to the standard, the URIs will be like:
api.example.com/object/user/31
api.example.com/object/user/31/properties/username
api.example.com/object/user/31/collections/channels
api.example.com/object/user/31/collections/members
api.example.com/object/user/31/actions/someFunction
api.example.com/object/user/31/actions/someFunction/invoke
There are also other resources
api.example.com/services
api.example.com/domain-types
The specification defines a few primary representations:
object (which represents any domain object or service)
list (of links to other objects)
property
collection
action
action result (typically containing either an object or a list, or just feedback messages)
and also a small number of secondary representations such as home, and user
This is interesting as you’ll see that representations are fully self-describing, opening up the possibility of generic viewers to be implemented if required.
Alternatively, the representations can be consumed directly by a bespoke application.
Here's my conclusions from many hours searching and with input from the responders here:
Where I have an object that is effectively a multi-part object, I need to treat that as a single resource. Thus if I GET the object, all the sub-ordinates should be present. This is required in order that the resource is cacheable. If I part load the object (and provide an ETag stamp) then other requestors may receive a partial object when they expected a full one. Conclude - objects should be fully populated if they are being made available as resources.
Associated object relationships should be made available as links to other (primary) resources. In this way the objects are discoverable by traversing the API.
Also, the object hierarchy that made sense for main application site may appear not be what you need to act in RESTful manner, but is more likely revealing problems with the existing hierarchy. Having said this the API may require more specialised use cases than had been previously envisaged, and specialised resources may be required.
Hope that helps someone
I am new to the REST architecural design, however I think I have the basics of it covered.
I have a problem with returning objects from a RESTful call. If I make a request such as http://localhost/{type A}/{id} I will return an instance of A from the database with the specified id.
My question is what happens when A contains a collection of B objects? At the moment the XML I generate returns A with a collection of B objects inside of it. As you can imagine if the B type has a collection of C objects then the XML returned will end up being a quite complicated object graph.
I can't be 100% sure but this feels to be against the RESTful principles, the XML for A should return the fields etc. for A as well as a collection of URI's to the collection of B's that it owns.
Sorry if this is a bit confusing, I can try to elaborate more. This seems like a relatively basic question, however I can't decide which approach is "more" RESTful.
Cheers,
Aidos
One essential RESTful principle is that everything has a URI.
You have URI's like this.
/A/ and /A/id/ to get a list of A's and a specific A. The A response includes the ID's of B's.
/B/ and /B/id/ to get a list of B's and a specific B. The B response includes the ID's of C's.
/C/ and /C/id/ to get a list of C's and a specific C.
You can, through a series of queries, rebuild the A-B-C structure. You get the A, then get the relevant B's. When getting a B, you get the various C's that are referenced.
Edit
Nothing prevents you from returning more.
For example, you might have the following kinds of URI's.
/flat/A/id/, /flat/B/id/ and /flat/C/id/ to return "flat" (i.e., no depth) structures.
/deep/A/id/, /deep/B/id/ and /deep/C/id/ to return structures with complete depth.
The /deep/A/id/ would be the entire structure, in a big, nested XML document. Fine for clients that can handle it. /flat/A/id/ would be just the top level in a flat document. Best for clients that can't handle depth.
There's nothing saying your REST interface can't be relational.
/bookstore/{bookstoreID}
/bookstore/{bookstoreID}/books
/book/{bookID}
Basically, you have a 1:1 correspondence with your DB schema.
Except for Many-to-Many relations forming child lists. For example,
/bookstore/657/books should return a list of book IDs or URLs. Then if you want a specific book's data you can invoke the 3rd URL.
This is just off the top of my head, please debate the merits.
Make a flat Universe that you expose to the world.
Even when I use SOAP, which can easily handle hierarchical object graphs up to whatever depth, I flatten the graph and link everything with simple IDs (you could even use your database IDs, although the idea is that you want you don't want to expose your PKs to the world).
Your object universe inside your app is not necessarily the same one you expose to the world. Let A have children and let B have children, but there's no need to reflect that in the REST request URLs.
Why flatten? Because then you can do things like fetch the objects later by ID, or send them in bursts (both the same case, more or less)... And better than all that, the request URIs don't change when the object hierarchy changes (object 37252 is always the same, even when it's been reclassed).
Edit: Well, you asked for it... Here's the architecture I ended up using:
package: server - contains the superclasses that are shared between the front-end server and the back-end server
package: frontEndServer - contains a Server interface which the front-end server must adhere to. The interface is nice because if you decide to change from SOAP to a straight Web client (that uses JSON or whatever, as well), you've got the interface all laid out. It also contains all the implementations for the frontEnd classes that will be tossed to the client, and all the logic for the interaction between classes except how to talk to the client.
package: backEndServer - contains a Server interface which the back-end server will adhere to. An example of a Server implementation would be one that talks to a MySql DB or one that talks to an XML DB, but the Server interface is neutral. This package also contains all the classes that the implementations of the Server interface use to get work done, and all the logic for the backend except for persistence.
then you have implementation packages for each of these... which include stuff like how to persist for the backend and how to talk to the client for the front end. The front-end implementation package might know, for instance, that a user has logged in, whereas the frontEndServer just knows that it has to implement methods for creating users and logging in.
After beginning to write this up I realize that it would take a while more to describe everything, but here you have the gist of it.