Related to this question the counterparty provider engine is somehow set up to check the group order of FIX tags and reject anything out of some expected order.
Why does expected tag group order matter? I guess it's quicker to validate tags in a given order.
How is expected tag group order set? I understand this is a random hash set, except that does not make sense, does it not depend on the order of tags in the data dictionary?
Besides rewriting a class to set group order, is there a quickfix setting to use?
To be exact with QuickFix version 2.2.0 I send the following message
8=FIX.4.4 9=173 35=R 34=2 49=CLIENT 52=20200909-18:11:10.426 56=SIMULATOR 131=EEB85F9C 146=1 55=EUR/USD 460=4 167=FOR 38=1000.0 64=SP 15=EUR 1=FOR 553=test 1300=XOFF 10=086
and receive a reject
8=FIX.4.4 9=145 35=3 34=2 49=SIMULATOR 52=20200909-18:11:10.427 56=CLIENT 45=2 58=The group 146 must set the delimiter field 460 371=55 372=R 373=15 10=224
So in the sent message the tag 460 is coming after the tag 55 and I can't get these tags the other way around. In the code I set up the repeating group g
QuickFix.FIX44.QuoteRequest.NoRelatedSymGroup g = new QuickFix.FIX44.QuoteRequest.NoRelatedSymGroup();
and add the data to the group in the order I am looking for, so like:
Product product = new Product(4);
g.Product = product;
Symbol symbol = new Symbol("EUR/USD");
g.SetField(symbol);
and so on... I am looking at the g.getFieldOrder and g.SetFields but is there another way?
In other quickfix versions like 1.6.2 the reject message is Out of order repeating group members for the same reason, as far as I can tell.
Thanks to #ChristopheJohn I got this working in QuickFixN with code:
using QuickFix;
class MyGroup : Group
{
private static int[] FIELD_ORDER = { 460, 1300, 167, 55, 15, 38, 64, 1, 553, 0 };
public MyGroup() : base(146, 460, FIELD_ORDER) { }
}
Which I called from my message construction methods with MyGroup g = new MyGroup();
Note the 0 at the end of the field order.
Related
Updated: Is it possible to window a data stream on a signals phase.
For example, there is a stream of timestamp, key, value:
[<t0, k1, 0>, <t1, k1, 98>, <t2, k1, 145>, <t4, k1, 0>, <t3, k1, 350>, <t5, k1, 40>, <t6, k1, 65>, <t7, k1, 120>, <t8, k1, 240>, <t9, k1, 352>].
The output would be two windows for key k1:
t0 - t3: [0, 98, 145, 350]
t4 - t9: [0, 40, 65, 120, 240, 352]
E.g. every time the value hits 0, start a new window for the group.
After your question edit and use case clarification I would recommend to look into custom windowing to extend the standard sessions. As a starting point I built the following example (it can be improved upon).
Through WindowFn.AssignContext we can access the element() that it's being windowed into a proto-session. If it's equal to a given stopValue the window length will be confined to the minimum instead of using gapDuration for that purpose:
#Override
public Collection<IntervalWindow> assignWindows(AssignContext c) {
Duration newGap = c.element().getValue().equals(this.stopValue) ? new Duration(1) : gapDuration;
return Arrays.asList(new IntervalWindow(c.timestamp(), newGap));
}
Then, when merging the sorted windows we'll check if they do overlap but also that the window duration is not equal to 1 ms.
Collections.sort(sortedWindows);
List<MergeCandidate> merges = new ArrayList<>();
MergeCandidate current = new MergeCandidate();
for (IntervalWindow window : sortedWindows) {
// get window duration and check if it's a stop session request
Long windowDuration = new Duration(window.start(), window.end()).getMillis();
if (current.intersects(window) && !windowDuration.equals(1L)) {
current.add(window);
} else {
merges.add(current);
current = new MergeCandidate(window);
}
}
merges.add(current);
for (MergeCandidate merge : merges) {
merge.apply(c);
}
Of course, we also can add some code so that we can provide different stopping values: a stopValue field, a withStopValue method, constructors, display data if using the Dataflow Runner, etc.
/** Value that closes the session. */
private final Integer stopValue;
/** Creates a {#code StopSessions} {#link WindowFn} with the specified gap duration. */
public static StopSessions withGapDuration(Duration gapDuration) {
return new StopSessions(gapDuration, 0);
}
/** Creates a {#code StopSessions} {#link WindowFn} with the specified stop value. */
public StopSessions withStopValue(Integer stopValue) {
return new StopSessions(gapDuration, stopValue);
}
/** Creates a {#code StopSessions} {#link WindowFn} with the specified gap duration and stop value. */
private StopSessions(Duration gapDuration, Integer stopValue) {
this.gapDuration = gapDuration;
this.stopValue = stopValue;
Now in our pipeline we can import and use the new StopSessions class with:
import org.apache.beam.sdk.transforms.windowing.StopSessions; // custom one
...
.apply("Window into StopSessions", Window.<KV<String, Integer>>into(StopSessions
.withGapDuration(Duration.standardSeconds(10))
.withStopValue(0)))
To mimic your example we create some data with:
.apply("Create data", Create.timestamped(
TimestampedValue.of(KV.of("k1", 0), new Instant()), // <t0, k1, 0>
TimestampedValue.of(KV.of("k1",98), new Instant().plus(1000)), // <t1, k1, 98>
TimestampedValue.of(KV.of("k1",145), new Instant().plus(2000)), // <t2, k1, 145>
TimestampedValue.of(KV.of("k1",0), new Instant().plus(4000)), // <t4, k1, 0>
...
With standard sessions the output would be:
user=k1, scores=[0,145,350,120,0,40,65,98,240,352], window=[2019-06-08T19:13:46.785Z..2019-06-08T19:14:05.797Z)
And with the custom one I get the following:
user=k1, scores=[350,145,98], window=[2019-06-08T21:18:51.395Z..2019-06-08T21:19:03.407Z)
user=k1, scores=[0], window=[2019-06-08T21:18:54.407Z..2019-06-08T21:18:54.408Z)
user=k1, scores=[65,240,352,120,40], window=[2019-06-08T21:18:55.407Z..2019-06-08T21:19:09.407Z)
user=k1, scores=[0], window=[2019-06-08T21:18:50.395Z..2019-06-08T21:18:50.396Z)
Changing the stopValue with .withStopValue(<int>) works as expected. The 98, 145 and 350 events are in a different session than the rest. Please note that this is not exactly like in the description as the stopValue gets assigned to a separate window instead of the new one but it can be filtered downstream and it gives you an idea on how to proceed. I would like to revisit this and also look for a Python implementation, too.
All files here.
Likely not, from your description. There are at least two problems:
PCollections in Beam are unordered and distributed:
there are no guarantees in the model that events from one group will arrive in that order;
data-driven triggers are not supported (probably for similar reasons):
https://beam.apache.org/documentation/programming-guide/#data-driven-triggers
However you can look into stateful processing and see if you can handle this manually. E.g. you accumulate all the incoming events in the state and then from time to time analyze the accumulated events and emit the results.
Or if you can extract/assign a common key in your business logic, then you might want to check if GroupByKey+ParDo or Combine would be helpful.
See:
https://beam.apache.org/blog/2017/02/13/stateful-processing.html
https://docs.google.com/document/d/1zf9TxIOsZf_fz86TGaiAQqdNI5OO7Sc6qFsxZlBAMiA/edit
https://beam.apache.org/documentation/programming-guide/#combine
https://beam.apache.org/documentation/programming-guide/#groupbykey
I'm trying to test out Firebase to allow users to post comments using push. I want to display the data I retrieve with the following;
fbl.child('sell').limit(20).on("value", function(fbdata) {
// handle data display here
}
The problem is the data is returned in order of oldest to newest - I want it in reversed order. Can Firebase do this?
Since this answer was written, Firebase has added a feature that allows ordering by any child or by value. So there are now four ways to order data: by key, by value, by priority, or by the value of any named child. See this blog post that introduces the new ordering capabilities.
The basic approaches remain the same though:
1. Add a child property with the inverted timestamp and then order on that.
2. Read the children in ascending order and then invert them on the client.
Firebase supports retrieving child nodes of a collection in two ways:
by name
by priority
What you're getting now is by name, which happens to be chronological. That's no coincidence btw: when you push an item into a collection, the name is generated to ensure the children are ordered in this way. To quote the Firebase documentation for push:
The unique name generated by push() is prefixed with a client-generated timestamp so that the resulting list will be chronologically-sorted.
The Firebase guide on ordered data has this to say on the topic:
How Data is Ordered
By default, children at a Firebase node are sorted lexicographically by name. Using push() can generate child names that naturally sort chronologically, but many applications require their data to be sorted in other ways. Firebase lets developers specify the ordering of items in a list by specifying a custom priority for each item.
The simplest way to get the behavior you want is to also specify an always-decreasing priority when you add the item:
var ref = new Firebase('https://your.firebaseio.com/sell');
var item = ref.push();
item.setWithPriority(yourObject, 0 - Date.now());
Update
You'll also have to retrieve the children differently:
fbl.child('sell').startAt().limitToLast(20).on('child_added', function(fbdata) {
console.log(fbdata.exportVal());
})
In my test using on('child_added' ensures that the last few children added are returned in reverse chronological order. Using on('value' on the other hand, returns them in the order of their name.
Be sure to read the section "Reading ordered data", which explains the usage of the child_* events to retrieve (ordered) children.
A bin to demonstrate this: http://jsbin.com/nonawe/3/watch?js,console
Since firebase 2.0.x you can use limitLast() to achieve that:
fbl.child('sell').orderByValue().limitLast(20).on("value", function(fbdataSnapshot) {
// fbdataSnapshot is returned in the ascending order
// you will still need to order these 20 items in
// in a descending order
}
Here's a link to the announcement: More querying capabilities in Firebase
To augment Frank's answer, it's also possible to grab the most recent records--even if you haven't bothered to order them using priorities--by simply using endAt().limit(x) like this demo:
var fb = new Firebase(URL);
// listen for all changes and update
fb.endAt().limit(100).on('value', update);
// print the output of our array
function update(snap) {
var list = [];
snap.forEach(function(ss) {
var data = ss.val();
data['.priority'] = ss.getPriority();
data['.name'] = ss.name();
list.unshift(data);
});
// print/process the results...
}
Note that this is quite performant even up to perhaps a thousand records (assuming the payloads are small). For more robust usages, Frank's answer is authoritative and much more scalable.
This brute force can also be optimized to work with bigger data or more records by doing things like monitoring child_added/child_removed/child_moved events in lieu of value, and using a debounce to apply DOM updates in bulk instead of individually.
DOM updates, naturally, are a stinker regardless of the approach, once you get into the hundreds of elements, so the debounce approach (or a React.js solution, which is essentially an uber debounce) is a great tool to have.
There is really no way but seems we have the recyclerview we can have this
query=mCommentsReference.orderByChild("date_added");
query.keepSynced(true);
// Initialize Views
mRecyclerView = (RecyclerView) view.findViewById(R.id.recyclerView);
mManager = new LinearLayoutManager(getContext());
// mManager.setReverseLayout(false);
mManager.setReverseLayout(true);
mManager.setStackFromEnd(true);
mRecyclerView.setHasFixedSize(true);
mRecyclerView.setLayoutManager(mManager);
I have a date variable (long) and wanted to keep the newest items on top of the list. So what I did was:
Add a new long field 'dateInverse'
Add a new method called 'getDateInverse', which just returns: Long.MAX_VALUE - date;
Create my query with: .orderByChild("dateInverse")
Presto! :p
You are searching limitTolast(Int x) .This will give you the last "x" higher elements of your database (they are in ascending order) but they are the "x" higher elements
if you got in your database {10,300,150,240,2,24,220}
this method:
myFirebaseRef.orderByChild("highScore").limitToLast(4)
will retrive you : {150,220,240,300}
In Android there is a way to actually reverse the data in an Arraylist of objects through the Adapter. In my case I could not use the LayoutManager to reverse the results in descending order since I was using a horizontal Recyclerview to display the data. Setting the following parameters to the recyclerview messed up my UI experience:
llManager.setReverseLayout(true);
llManager.setStackFromEnd(true);
The only working way I found around this was through the BindViewHolder method of the RecyclerView adapter:
#Override
public void onBindViewHolder(final RecyclerView.ViewHolder holder, int position) {
final SuperPost superPost = superList.get(getItemCount() - position - 1);
}
Hope this answer will help all the devs out there who are struggling with this issue in Firebase.
Firebase: How to display a thread of items in reverse order with a limit for each request and an indicator for a "load more" button.
This will get the last 10 items of the list
FBRef.child("childName")
.limitToLast(loadMoreLimit) // loadMoreLimit = 10 for example
This will get the last 10 items. Grab the id of the last record in the list and save for the load more functionality. Next, convert the collection of objects into and an array and do a list.reverse().
LOAD MORE Functionality: The next call will do two things, it will get the next sequence of list items based on the reference id from the first request and give you an indicator if you need to display the "load more" button.
this.FBRef
.child("childName")
.endAt(null, lastThreadId) // Get this from the previous step
.limitToLast(loadMoreLimit+2)
You will need to strip the first and last item of this object collection. The first item is the reference to get this list. The last item is an indicator for the show more button.
I have a bunch of other logic that will keep everything clean. You will need to add this code only for the load more functionality.
list = snapObjectAsArray; // The list is an array from snapObject
lastItemId = key; // get the first key of the list
if (list.length < loadMoreLimit+1) {
lastItemId = false;
}
if (list.length > loadMoreLimit+1) {
list.pop();
}
if (list.length > loadMoreLimit) {
list.shift();
}
// Return the list.reverse() and lastItemId
// If lastItemId is an ID, it will be used for the next reference and a flag to show the "load more" button.
}
I'm using ReactFire for easy Firebase integration.
Basically, it helps me storing the datas into the component state, as an array. Then, all I have to use is the reverse() function (read more)
Here is how I achieve this :
import React, { Component, PropTypes } from 'react';
import ReactMixin from 'react-mixin';
import ReactFireMixin from 'reactfire';
import Firebase from '../../../utils/firebaseUtils'; // Firebase.initializeApp(config);
#ReactMixin.decorate(ReactFireMixin)
export default class Add extends Component {
constructor(args) {
super(args);
this.state = {
articles: []
};
}
componentWillMount() {
let ref = Firebase.database().ref('articles').orderByChild('insertDate').limitToLast(10);
this.bindAsArray(ref, 'articles'); // bind retrieved data to this.state.articles
}
render() {
return (
<div>
{
this.state.articles.reverse().map(function(article) {
return <div>{article.title}</div>
})
}
</div>
);
}
}
There is a better way. You should order by negative server timestamp. How to get negative server timestamp even offline? There is an hidden field which helps. Related snippet from documentation:
var offsetRef = new Firebase("https://<YOUR-FIREBASE-APP>.firebaseio.com/.info/serverTimeOffset");
offsetRef.on("value", function(snap) {
var offset = snap.val();
var estimatedServerTimeMs = new Date().getTime() + offset;
});
To add to Dave Vávra's answer, I use a negative timestamp as my sort_key like so
Setting
const timestamp = new Date().getTime();
const data = {
name: 'John Doe',
city: 'New York',
sort_key: timestamp * -1 // Gets the negative value of the timestamp
}
Getting
const ref = firebase.database().ref('business-images').child(id);
const query = ref.orderByChild('sort_key');
return $firebaseArray(query); // AngularFire function
This fetches all objects from newest to oldest. You can also $indexOn the sortKey to make it run even faster
I had this problem too, I found a very simple solution to this that doesn't involved manipulating the data in anyway. If you are rending the result to the DOM, in a list of some sort. You can use flexbox and setup a class to reverse the elements in their container.
.reverse {
display: flex;
flex-direction: column-reverse;
}
myarray.reverse(); or this.myitems = items.map(item => item).reverse();
I did this by prepend.
query.orderByChild('sell').limitToLast(4).on("value", function(snapshot){
snapshot.forEach(function (childSnapshot) {
// PREPEND
});
});
Someone has pointed out that there are 2 ways to do this:
Manipulate the data client-side
Make a query that will order the data
The easiest way that I have found to do this is to use option 1, but through a LinkedList. I just append each of the objects to the front of the stack. It is flexible enough to still allow the list to be used in a ListView or RecyclerView. This way even though they come in order oldest to newest, you can still view, or retrieve, newest to oldest.
You can add a column named orderColumn where you save time as
Long refrenceTime = "large future time";
Long currentTime = "currentTime";
Long order = refrenceTime - currentTime;
now save Long order in column named orderColumn and when you retrieve data
as orderBy(orderColumn) you will get what you need.
just use reverse() on the array , suppose if you are storing the values to an array items[] then do a this.items.reverse()
ref.subscribe(snapshots => {
this.loading.dismiss();
this.items = [];
snapshots.forEach(snapshot => {
this.items.push(snapshot);
});
**this.items.reverse();**
},
For me it was limitToLast that worked. I also found out that limitLast is NOT a function:)
const query = messagesRef.orderBy('createdAt', 'asc').limitToLast(25);
The above is what worked for me.
PRINT in reverse order
Let's think outside the box... If your information will be printed directly into user's screen (without any content that needs to be modified in a consecutive order, like a sum or something), simply print from bottom to top.
So, instead of inserting each new block of content to the end of the print space (A += B), add that block to the beginning (A = B+A).
If you'll include the elements as a consecutive ordered list, the DOM can put the numbers for you if you insert each element as a List Item (<li>) inside an Ordered Lists (<ol>).
This way you save space from your database, avoiding unnecesary reversed data.
I'm currently using var label = GmailApp.getUserLabelByName("Delete Me");
to identify threads in Gmail Inbox that I want delete if they are of a certain age. The method I am using depends on a filter being set for messages as "Delete Me" and I'm wondering if I could instead list the senders I'd like to delete from in my script directly.
Thanks,
Although, I agree with what Henrique said, however, if you MUST do the filtering and deleting of threads from your script, one way of doing it is to use the search query function and deleting all the threads that appear as a result of that query.
For searching by user:
var threads = GmailApp.search('from:username#mail.com', 10, 10);
For searching by date:
var threads = GmailApp.search('before:04/05/2015', 10, 10);
Would then pass threads through my delete function.
EDIT:
If you want the date to be calculated based on the current date, you could do something like this:
var d = new Date();
d.setDate(d.getDate() - 20);
var searchTxt = "before:"+d;
var threads = GmailApp.search(searchTxt, 10, 10);
The bizarre properties in the .NET SDK continue to baffle me. How do I read the UnitPrice from an invoice line?
If I do this:
sild = (SalesItemLineDetail)line.AnyIntuitObject;
ln = new QBInvoiceLine(); // My internal line item class
ln.Description = line.Description;
ln.ItemRef = new QBRef() { Id = sild.ItemRef.Value, Name = sild.ItemRef.name };
if (sild.QtySpecified)
ln.Quantity = sild.Qty;
else
ln.Quantity = 0;
if (sild.ItemElementName == ItemChoiceType.UnitPrice)
ln.Rate = (decimal)sild.AnyIntuitObject; // Exception thrown here
The last line throws an invalid cast exception, even though the debugger shows that the value is 20. I've tried other types but get the same exception no matter what I do. So I finally punted and am calculating the rate like so:
ln.Rate = line.Amount / ln.Quantity;
(With proper rounding and checking for divide by zero, of course)
While we're on the subject... I noticed that in many cases ItemElementName == ItemChoiceType.PriceLevelRef. What's up with that? As far as I know, QBO doesn't support price levels, and I certainly wasn't using a price level with this invoice or customer. In this case I was also able to get what I needed from the Amount property.
Try this-
SalesItemLineDetail a1 = (SalesItemLineDetail)invoice11.Line[0].AnyIntuitObject;
object unitprice = a1.AnyIntuitObject;
decimal quantity = a1.Qty;
PriceLevelRef as an 'entity' is not supported. This means CRUD operations are not supported on this entity.
The service might however be returning readonly values in the transactions sometimes, but since this not mentioned in the docs, please consider it as unsupported.
Check that both request/response are in either json or xml format-
You can use the following code to set that-
ServiceContext context = new ServiceContext(appToken, realmId, intuitServiceType, reqvalidator);
context.IppConfiguration.Message.Request.SerializationFormat = Intuit.Ipp.Core.Configuration.SerializationFormat.Json;
context.IppConfiguration.Message.Response.SerializationFormat = Intuit.Ipp.Core.Configuration.SerializationFormat.Json;
Also, in QBO UI, check if Company->sales settings has Track Quantity and Price/rate turned on.
I used the following code to retrieve all the journal articles based on a specific category:
long catId = category.getCategoryId();
AssetEntryQuery assetEntryQuery = new AssetEntryQuery();
long[] anyCatIds = {catId};
assetEntryQuery.setAnyCategoryIds(anyCatIds);
// Line A
List<AssetEntry> assetEntryList = AssetEntryLocalServiceUtil.getEntries(assetEntryQuery);
for(AssetEntry ae : assetEntryList)
{
// Line B
JournalArticleResource journalArticleResourceObj = JournalArticleResourceLocalServiceUtil.getJournalArticleResource(ae.getClassPK());
JournalArticle journalArticleObj = JournalArticleLocalServiceUtil.getArticle(themeDisplay.getParentGroupId(), journalArticleResourceObj.getArticleId());
journalArticleList.add(journalArticleObj.getArticleId());
}
Howver, since the AssetEntry will fetch all the entries, including Blogs, and so on, the above code will throw an exception at Line B if I add a blog entry that uses the same category, since the blog entry does not have JournalArticleResource.
So, I was wondering if it is possible to filter only JournalArticle type will be fetch at Line A, then I don't have to worry about Line B anymore.
I have tried but no luck so far.
Does any one have some ideas?
Use assetEntryQuery.setClassName(JournalArticle.class.getName());