I'm trying to construct middleware-code in d-lang that reads an array of struct DATA from a connected POSIX named pipe and identify changes by comparing with existing data, if changes occurred add the changed struct data into db.
Struct is constructed like this and all variables are likely to be changed
struct DATA
{
char[20] name;
int_16 sect;
int_16 ptyp;
ulong mode;
ulong cpmap;
}
DATA [NO_OF_STRUCTS*sizeof(struct DATA)] flowDATA;
Input from pipe works but when it comes to fetch individual latest version from the database, I'm outside of my knowledge
I.e. my problem is how to construct the query (code not for terminal) to retrieve the latest inserted data for each array indexposition from the db to compare and if changed, write/insert into db and also inform och what changed back via the pipe.
Feels like I need to add extra fields like an index_no to filter on.
The query would be like this in below pseudo code (can't get this code correct in my d-code):
db.lab.aggregate([{$match {dateadded{$in:["dateadded"]}}.{ $group:{_id:"$name"}}}]);
Select DATA from lab Where LATEST(dateAdded) and index_no == arrayIndex
Any ideas?
My testcode init the db like this
db.lab.insert([
{"name": "News 1",
"sektion":2,
"ptyp":1,
"mode":1024,
"cpmap":886,
dateAdded: new Date()
},{
"name": "Base 2",
"sektion":2,
"ptyp":1,
"mode":1024,
"cpmap":886,
dateAdded: new Date()
},{
"name": "Name 3",
"sektion":1,
"ptyp":3,
"mode":24,
"cpmap":886,
dateAdded: new Date()
},{
"name": "Name 4",
"sektion":1,
"ptyp":1,
"mode":0,
"cpmap":1024,
dateAdded: new Date()
}]};
Related
I'm using Azure Data Factory to monitor an AWS S3 bucket that will have files containing JSON objects that are written out by an AWS process. The process may combine multiple JSON objects into a single file with no CRLF or delimiters between the objects. I need Azure Data Factory to process each of these object individually to insert them into a SQL database. I'm not finding any examples of how to process this scenario. Sorry if this is rather basic in Azure Data Factory, however, I'm rather new to the product.
Here is a sample of the file format:
{
"AWSInfoField1": "Test Record 1",
"AWSInfoField2": "Just Another Field",
"Attributes": {
"Attribute1": 1,
"Attribute2": "Another Attribute"
}
}
{
"AWSInfoField1": "Test Record 2",
"AWSInfoField2": "Just Another Field In Record 2",
"Attributes": {
"Attribute1": 2,
"Attribute2": "Another Attribute In Record 2"
}
}
{
"AWSInfoField1": "Test Record 3",
"AWSInfoField2": "Just Another Field In Record 3",
"Attributes": {
"Attribute1": 3,
"Attribute2": "Another Attribute In Record 3"
}
}
I copy the data to a single file in my storage(with no CRLF or delimiters between the objects):
I tried and found that Data Factory will auto add the default delimiter ',' to the JSON data in Source. We can see it in Source data preview:
Then choose the SQL database as sink and mapping data to Sink table:
Run the pipeline and check the data in table:
I am having difficulties getting multiple datasets out of my database with RestTemplate. I have many routines that extract a single row, with a format like:
IndicatorModel indicatorModel = restTemplate.getForObject(URL + id,
IndicatorModel.class);
and they work fine. However, if I try to extract a set of data, such as:
Map<String, List<S_ServiceCoreTypeModel>> coreTypesMap =
restTemplate.getForObject(URL + id, Map.class);
this returns values in a
Map<String, LinkedHashMap<>>
format. Is there an easy way to return a List<> or Set<> in the desired format?
Fundamentally the issue is that your Java object model does not match the structure of your json document. You are attempting to deserialize a single json element into a java List. Your JSON document looks like:
{
"serviceCoreTypes":[
{
"serviceCoreType":{
"name":"ALL",
"description":"All",
"dateCreated":"2016-06-23 14:46:32.09",
"dateModified":"2016-06-23 14:46:32.09",
"deleted":false,
"id":1
}
},
{
"serviceCoreType":{
"name":"HSI",
"description":"High-speed Internet",
"dateCreated":"2016-06-23 14:47:31.317",
"dateModified":"2016-06-23 14:47:31.317",
"deleted":false,
"id":2
}
}
]
}
But you cannot turn a serviceCoreTypes into a List, you can only turn a Json Array into a List. For instance if you removed the unnecessary wrapper elements from your json and your input document looked like:
[
{
"name": "ALL",
"description": "All",
"dateCreated": "2016-06-23 14:46:32.09",
"dateModified": "2016-06-23 14:46:32.09",
"deleted": false,
"id": 1
},
{
"name": "HSI",
"description": "High-speed Internet",
"dateCreated": "2016-06-23 14:47:31.317",
"dateModified": "2016-06-23 14:47:31.317",
"deleted": false,
"id": 2
}
]
You should be able to then deserialize THAT into a List< S_ServiceCoreTypeModel>. Alternately if you cannot change the json structure, you could create a Java object model that models the json document by creating some wrapper classes. Something like:
class ServiceCoreTypes {
List<ServiceCoreType> serviceCoreTypes;
...
}
class ServiceCoreTypeWrapper {
ServiceCoreType serviceCoreType;
...
}
class ServiceCoreType {
String name;
String description;
...
}
I'm assuming you don't actually mean database, but instead a restful service as you're using RestTemplate
The problem you're facing is that you want to get a Collection back, but the getForObject method can only take in a single type parameter and cannot figure out what the type of the returned collection is.
I'd encourage you to consider using RestTemplate.exchange(...)
which should allow you request for and receive back a collection type.
I have a solution that works, for now at least. I would prefer a solution such as the one proposed by Ben, where I can get the HTTP response body as a list of items in the format I chose, but at least here I can extract each individual item from the JSON node. The code:
S_ServiceCoreTypeModel endModel;
RestTemplate restTemplate = new RestTemplate();
JsonNode node = restTemplate.getForObject(URL, JsonNode.class);
JsonNode allNodes = node.get("serviceCoreTypes");
JsonNode oneNode = allNodes.get(1);
ObjectMapper objectMapper = new ObjectMapper();
endModel = objectMapper.readValue(oneNode.toString(), S_ServiceCoreTypeModel.class);
If anyone has thoughts on how to make Ben's solution work, I would love to hear it.
How do I render embedded objects in Apigility? For example, if I have a 'user' object and it composes a 'country' object, should I be rendering the 'country' object as an embedded object? And how should I do this?
I am using the Zend\Stdlib\Hydrator\ArraySerializable. My getArrayCopy() method simply returns an array of properties that I want exposed. The array keys are the property names. The array values are the property values. In the case of user->country, the value is an object, not a scalar.
When I return the user object from UserResource->fetch(), here's how it is rendered:
{
"id": "1",
"firstName": "Joe",
"lastName": "Bloggs",
"status": "Active",
"email": "test#example.com",
"country": {
"code": "AU",
"name": "Australia"
},
"settings": "0",
"_links": {
"self": {
"href": "http://api.mydomain.local/users/1"
}
}
}
Note that 'country' is not in an _embedded field. If it is supposed to be in _embedded, I would have thought that Apigility would automatically do that (since it automatically adds the _links object).
As a related issue, how do I go about returning other rel links, such as back, forward, etc?
The easiest way to get Apigility to render embedded resources is when there is an API/resource associated to the embedded object. What I mean for your example is that you'd have an API resource that has a country entity. In that case, if your getArrayCopy returned the the CountryEntity, Apigility would render it automatically as an embedded resource.
If your getArrayCopy is returning country as an array with code and name, you'll end up with what you saw.
For the other part, the rel links for first, last, prev and next will come from the fetchAll method when you return a Paginator. Your collection extends from this already, but it needs an adapter. The code could look something like this:
public function fetchAll($params)
{
// Return a \Zend\Db\Select object that will retrieve the
// stuff you want from the database
$select = $this->service->fetchAll($params);
$entityClass = $this->getEntityClass();
$entity = new $entityClass();
$hydrator = new \Zend\Stdlib\ArraySerializable();
$prototype = new \Zend\Db\ResultSet\HydratingResultSet($hydrator, $entity);
$paginator = new \Zend\Paginator\Adapter\DbSelect($select, $this->sql, $prototype);
$collectionClass = $this->getCollectionClass();
return new $collectionClass($paginator);
}
There are other paginator adapters as well - an ArrayAdapter which will take in an array of however big and then paginate it so you only get the desired number of results. The downside to this if you use it with database results, you'll potentially be retrieving and discarding a lot of results. The DbSelect paginator will modify the $select object to add the limit and order clause automatically so you only retrieve the bits you need. There are also adapters if you're using DbTableGateway, Iterators or even callbacks. You can also implement your own of course.
Hope this helps. If you have more specific needs or clarification, please comment and I'll do my best.
I posted this example on github.
https://github.com/martins-smb/apigility-renderCollection-example
Hope this helps.
Looking at the example code
var accounts = new Backbone.Collection;
accounts.url = '/accounts';
accounts.fetch();
this works if the route returns an array
[{id:1, name:'bob'}, {id:2, name:'joe'}]
but the REST service I'm using returns an object like this
{
items: [{id:1, name:'bob'}, {id:2, name:'joe'}],
page: 1,
href: '/acounts'
}
How to I go about telling Backbone.Collection that the collection is in items?
Parse function seems appropriate.
From the documentation:
http://backbonejs.org/
"When fetching raw JSON data from an API, a Collection will automatically populate itself with data formatted as an array, while a Model will automatically populate itself with data formatted as an object:
[{"id": 1}] ..... populates a Collection with one model.
{"id": 1} ....... populates a Model with one attribute.
However, it's fairly common to encounter APIs that return data in a different format than what Backbone expects. For example, consider fetching a Collection from an API that returns the real data array wrapped in metadata:
{
"page": 1,
"limit": 10,
"total": 2,
"books": [
{"id": 1, "title": "Pride and Prejudice"},
{"id": 4, "title": "The Great Gatsby"}
]
}
In the above example data, a Collection should populate using the "books" array rather than the root object structure. This difference is easily reconciled using a parse method that returns (or transforms) the desired portion of API data:
var Books = Backbone.Collection.extend({
url: '/books',
parse: function(data) {
return data.books;
}
});
"
Hope it helps.
I have two arrays in my _User table: hasCreated and isMemberOf that hold group objects belonging to a Group table.
When I POST to a Group, it happens as expected, but when I GET User with
includeKey=isMemberOf&hasCreated
I get the following:
hasCreated: [
{
__type: Object,
className: Group,
createdAt: 2014 - 06 - 04T18: 59: 02.325Z,
createdById: abc,
…
}
{
__type: Pointer,
className: Group,
objectId: def
}
{
__type: Pointer,
className: Group,
objectId: ghi
}]
isMemberOf: [
{
__type: Object,
className: Group,
createdAt: 2014 - 06 - 04T18: 59: 02.325Z,
createdById: abc,
…
}
{
__type: Pointer,
className: Group,
objectId: def
}
{
__type: Pointer,
className: Group,
objectId: ghi
}]
So basically, I only get back one of the three groups in its entirety, and the _type fields are different as well. Here is what I have in the User table in both hasCreated and isMemberOf array fields:
[{
"__type": "Pointer",
"className": "Group",
"objectId": "abc"
}, {
"__type": "Pointer",
"className": "Group",
"objectId": "def"
}, {
"__type": "Pointer",
"className": "Group",
"objectId": "ghi"
}]
What am I doing wrong? I need to get the full objects for each of the groups. Please advise! Thanks!
The parse data that you create will provide you only pointer's of other user. So in order to get the data of other user you will have to save those pointer's or objectId that you get from query you fire. Then you could use those objectId or pointer to again fire a query to fetch data of other member's. So I said that you could create another function which accept's pointer or array(containing objectId's) as parameter and in that function you could fire query's to fetch data depending on of objectId ,and at same time as you get detail's you could populate your view with that data. – walle84 41 mins ago
But my concern is that as your list of object is long then it would take a long time and also would effect you view as it won't let you update it until it's done or might crash. So to make your view update ,you could run these query's in different thread's using dispatch method's ,just like parallel thread but not on main thread. This way user will be able to interact with view ,also do handle the view appropriate way as in not letting it to crash. For suggestion till detail are populating you could show activity indicator or progress bar.Still not much idea about your app usage or feel n look – walle84 38 mins ago