I have an issue with my fix engine which I can't quite get to the bottom of.
I ran the following test:
I sent multiple requests to the fix server(for example-new order single), let's say 20 at a time.
Then when I receive the execution reports back, the Quickfix engine takes a long time to process them, it processes a few than halts/stops for a 150 ms or so and continues(so it takes about 1 ms per message, but with long pauses). I used a packet sniffer to check the network stream and it show no such delays, all messages arrive in 2-3 ms span.
I removed all my logic, and set the session setting on bare minimum to avoid any extra work, and memory allocations.
I ran this test many times, and it shows the same results each time.
I use the engine log file timestamps to read message time.
Heres the .cfg file
[DEFAULT]
ConnectionType=initiator
ReconnectInterval=2
FileStorePath=store
FileLogPath=Log
StartTime=06:00:00
EndTime=03:00:00
UseDataDictionary=N
ValidateLengthAndChecksum=N
ValidateFieldsOutOfOrder=N
ValidateFieldsHaveValues=N
ValidateUserDefinedFields=N
SocketTcpNoDelay=Y
DataDictionary=FixSpec/FIX44.xml
SocketConnectHost=x.x.x.x
SocketConnectPort=xxxxx
LogoutTimeout=5
ResetOnLogon=Y
UseMassCancel=N
UseMassStatus=N
ResetOnDisconnect=Y
HeartBtInt=30
[SESSION]
BeginString=FIX.4.4
SenderCompID=xxxxx
TargetCompID=xxx
here is my FromApp:
public void FromApp(Message message, SessionID sessionId)
{
try
{
Crack(message, sessionId);
}
catch (UnsupportedMessageType ex)
{
UnSupportedMessage?.Invoke(this, message);
}
}
And My ExecutionReport handler
public void OnMessage(ExecutionReport m, SessionID s)
{
}
I removed all logic from my Application to avoid unnecessary noise.
Example log output:
20160811-07:10:44.541 : 8=FIX.4.49=24135=856=xxxxx49=XXX34=452=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-355=AAPL54=138=159=540=244=50.00000017=8916296071=xxxxx22=248=202180515=USD20=039=0150=037=89162960860=20160811-07:10:44.57210=129
20160811-07:10:44.545 : 8=FIX.4.49=24135=856=xxxxx49=XXX34=552=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-455=AAPL54=138=159=540=244=50.00000017=8916296091=xxxxx22=248=202180515=USD20=039=0150=037=89162961060=20160811-07:10:44.57410=128
20160811-07:10:44.546 : 8=FIX.4.49=24135=856=xxxxx49=XXX34=652=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-555=AAPL54=138=159=540=244=50.00000017=8916296111=xxxxx22=248=202180515=USD20=039=0150=037=89162961260=20160811-07:10:44.57710=128
20160811-07:10:44.549 : 8=FIX.4.49=24135=856=xxxxx49=XXX34=752=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-655=AAPL54=138=159=540=244=50.00000017=8916296131=xxxxx22=248=202180515=USD20=039=0150=037=89162961460=20160811-07:10:44.58010=128
20160811-07:10:44.551 : 8=FIX.4.49=24135=856=xxxxx49=XXX34=852=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-755=AAPL54=138=159=540=244=50.00000017=8916296151=xxxxx22=248=202180515=USD20=039=0150=037=89162961660=20160811-07:10:44.58310=137
20160811-07:10:44.554 : 8=FIX.4.49=24135=856=xxxxx49=XXX34=952=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-855=AAPL54=138=159=540=244=50.00000017=8916296171=xxxxx22=248=202180515=USD20=039=0150=037=89162961860=20160811-07:10:44.58510=145
20160811-07:10:44.557 : 8=FIX.4.49=24235=856=xxxxx49=XXX34=1052=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-955=AAPL54=138=159=540=244=50.00000017=8916296191=xxxxx22=248=202180515=USD20=039=0150=037=89162962060=20160811-07:10:44.58810=185
20160811-07:10:44.560 : 8=FIX.4.49=24335=856=xxxxx49=XXX34=1152=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-1055=AAPL54=138=159=540=244=50.00000017=8916296211=xxxxx22=248=202180515=USD20=039=0150=037=89162962260=20160811-07:10:44.59110=216
20160811-07:10:44.562 : 8=FIX.4.49=24335=856=xxxxx49=XXX34=1252=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-1155=AAPL54=138=159=540=244=50.00000017=8916296231=xxxxx22=248=202180515=USD20=039=0150=037=89162962460=20160811-07:10:44.59410=225
20160811-07:10:44.565 : 8=FIX.4.49=24335=856=xxxxx49=XXX34=1352=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-1255=AAPL54=138=159=540=244=50.00000017=8916296251=xxxxx22=248=202180515=USD20=039=0150=037=89162962660=20160811-07:10:44.59610=233
20160811-07:10:44.693 : 8=FIX.4.49=24335=856=xxxxx49=XXX34=1452=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-1355=AAPL54=138=159=540=244=50.00000017=8916296271=xxxxx22=248=202180515=USD20=039=0150=037=89162962860=20160811-07:10:44.59910=242
20160811-07:10:44.693 : 8=FIX.4.49=24335=856=xxxxx49=XXX34=1552=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-1455=AAPL54=138=159=540=244=50.00000017=8916296291=xxxxx22=248=202180515=USD20=039=0150=037=89162963060=20160811-07:10:44.60210=224
20160811-07:10:44.693 : 8=FIX.4.49=24335=856=xxxxx49=XXX34=1652=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-1555=AAPL54=138=159=540=244=50.00000017=8916296311=xxxxx22=248=202180515=USD20=039=0150=037=89162963260=20160811-07:10:44.60510=224
20160811-07:10:44.699 : 8=FIX.4.49=24335=856=xxxxx49=XXX34=1752=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-1655=AAPL54=138=159=540=244=50.00000017=8916296331=xxxxx22=248=202180515=USD20=039=0150=037=89162963460=20160811-07:10:44.60910=234
20160811-07:10:44.699 : 8=FIX.4.49=24335=856=xxxxx49=XXX34=1852=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-1755=AAPL54=138=159=540=244=50.00000017=8916296351=xxxxx22=248=202180515=USD20=039=0150=037=89162963660=20160811-07:10:44.61310=235
20160811-07:10:44.699 : 8=FIX.4.49=24335=856=xxxxx49=XXX34=1952=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-1855=AAPL54=138=159=540=244=50.00000017=8916296371=xxxxx22=248=202180515=USD20=039=0150=037=89162963860=20160811-07:10:44.61710=245
20160811-07:10:44.699 : 8=FIX.4.49=24335=856=xxxxx49=XXX34=2052=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-1955=AAPL54=138=159=540=244=50.00000017=8916296391=xxxxx22=248=202180515=USD20=039=0150=037=89162964060=20160811-07:10:44.62110=228
20160811-07:10:44.699 : 8=FIX.4.49=24335=856=xxxxx49=XXX34=2152=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-2055=AAPL54=138=159=540=244=50.00000017=8916296411=xxxxx22=248=202180515=USD20=039=0150=037=89162964260=20160811-07:10:44.71410=219
20160811-07:10:44.699 : 8=FIX.4.49=24335=856=xxxxx49=XXX34=2252=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-2155=AAPL54=138=159=540=244=50.00000017=8916296431=xxxxx22=248=202180515=USD20=039=0150=037=89162964460=20160811-07:10:44.71610=227
20160811-07:10:44.700 : 8=FIX.4.49=24335=856=xxxxx49=XXX34=2352=20160811-07:10:4432=031=0.000000151=114=06=0.00000011=rA0Ny-2255=AAPL54=138=159=540=244=50.00000017=8916296451=xxxxx22=248=202180515=USD20=039=0150=037=89162964660=20160811-07:10:44.72010=228
Any ideas?
Is this normal behavior?
Where would you look for solutions?
Any thoughts on what is going on under the hood there?
Related
I'm writing a SinkConnector in Kafka Connect and hitting an issue. This connector has a configuration as such :
{
"connector.class" : "a.b.ExampleFileSinkConnector",
"tasks.max" : '1',
"topics" : "mytopic",
"maxFileSize" : "50"
}
I define the connector's config like this :
#Override public ConfigDef config()
{
ConfigDef result = new ConfigDef();
result.define("maxFileSize", Type.STRING, "10", Importance.HIGH, "size of file");
return result;
}
In the connector, I start the tasks as such :
#Override public List<Map<String, String>> taskConfigs(int maxTasks) {
List<Map<String, String>> result = new ArrayList<Map<String,String>>();
for (int i = 0; i < maxTasks; i++) {
Map<String, String> taskConfig = new HashMap<>();
taskConfig.put("connectorName", connectorName);
taskConfig.put("taskNumber", Integer.toString(i));
taskConfig.put("maxFileSize", maxFileSize);
result.add(taskConfig);
}
return result;
}
and all goes well.
However, when starting the Task (in taskConfigs()), if I add this :
taskConfig.put("epoch", "123");
this breaks the whole infrastructure : all connectors are stopped and restarted in an endless loop.
There is no exception or error whatsoever in the connect log file that can help.
The only way to make it work is to add "epoch" in the connector config, which I don't want to do since it is an internal parameter that the connector has to send to the task. It is not intended to be exposed to the connector's users.
Another point I noticed is that it is not possible to update the value of any connector config parameter, apart to set it to the default value. Changing a parameter and sending it to the task produces the same behavior.
I would really appreciate any help on this issue.
EDIT : here is the code of SinkTask::start()
#Override public void start(Map<String, String> taskConfig) {
try {
connectorName = taskConfig.get("connectorName");
log.info("{} -- Task.start()", connectorName);
fileNamePattern = taskConfig.get("fileNamePattern");
rootDir = taskConfig.get("rootDir");
fileExtension = taskConfig.get("fileExtension");
maxFileSize = SimpleFileSinkConnector.parseIntegerConfig(taskConfig.get("maxFileSize"));
maxTimeMinutes = SimpleFileSinkConnector.parseIntegerConfig(taskConfig.get("maxTimeMinutes"));
maxNumRecords = SimpleFileSinkConnector.parseIntegerConfig(taskConfig.get("maxNumRecords"));
taskNumber = SimpleFileSinkConnector.parseIntegerConfig(taskConfig.get("taskNumber"));
epochStart = SimpleFileSinkConnector.parseLongConfig(taskConfig.get("epochStart"));
log.info("{} -- fileNamePattern: {}, rootDir: {}, fileExtension: {}, maxFileSize: {}, maxTimeMinutes: {}, maxNumRecords: {}, taskNumber: {}, epochStart : {}",
connectorName, fileNamePattern, rootDir, fileExtension, maxFileSize, maxTimeMinutes, maxNumRecords, taskNumber, epochStart);
if (taskNumber == 0) {
checkTempFilesForPromotion();
}
computeInitialFilename();
log.info("{} -- Task.start() END", connectorName);
} catch (Exception e) {
log.info("{} -- Task.start() EXCEPTION : {}", connectorName, e.getLocalizedMessage());
}
}
We found the root cause of the issue. The Kafka Connect Framework is actually behaving as designed - the problem has to do with how we are trying to use the taskConfigs configuration framework.
The Problem
In our design, the FileSinkConnector sets an epoch in its start() lifecycle method, and this epoch is passed down to its tasks by way of the taskConfigs() lifecycle method. So each time the Connector's start() lifecycle method runs, different configuration is generated for the tasks - which is the problem.
Generating different configuration each time is a no-no. It turns out that the Connect Framework detects differences in configuration and will restart/rebalance upon detection - stopping and restarting the connector/task. That restart will call the stop() and start() methods of the connector ... which will (of course) produces yet another configuration change (because of the new epoch), and the vicious cycle is on!
This was an interesting and unexpected issue ... due to a behavior in Connect that we had no appreciation for. This is the first time we tried to generate task configuration that was not a simple function of the connector configuration.
Note that this behavior in Connect is intentional and addresses real issues of dynamically-changing configuration - like a JDBC Sink Connector that spontaneously updates its configuration when it detects a new database table it wants to sink.
Thanks to those who helped us !
I'm having problems with EarlGrey's synchronization feature. I'm trying to create a simple test example for my app that basically checks if the login button shows up.
func testExample() {
let success = GREYCondition(name: "Wait for action view controller", block: { () -> Bool in
if let rootVC = UIApplication.shared.keyWindow?.rootViewController, rootVC is ValuePropsViewController {
return true
}
else {
return false
}
}).wait(withTimeout: 10)
GREYAssertTrue(success, reason: "Action view controller should appear within 5 seconds")
EarlGrey.select(elementWithMatcher: grey_accessibilityID("loginButton")).assert(grey_sufficientlyVisible())
}
However, it always times out and I get the following error message. I check the screenshots and the correct screen is showing at failure.
Exception Name: AssertionFailedException
Exception Reason: Timed out while waiting to perform assertion.
Exception with Assertion: {
"Assertion Criteria" : "assertWithMatcher: matcherForSufficientlyVisible(>=0.750000)",
"Element Matcher" : "(respondsToSelector(accessibilityIdentifier) && accessibilityID('loginButton'))"
}
Exception Details: Error Trace: [
{
"Description" : "Failed to execute block because idling resources below are busy.",
"Description Glossary" : {
"animateWithDuration:delay:options:animations:completion:" : "<NSObject: 0x610000203a30> caused the App to be in busy state for 6.8 seconds.",
"performSelector #selector(tick) on VAAppDelegate" : "Delayed performSelector caused the App to be in busy state for 2 seconds."
},
"Domain" : "com.google.earlgrey.GREYUIThreadExecutorErrorDomain",
"Code" : "0",
"File Name" : "GREYUIThreadExecutor.m",
"Function Name" : "-[GREYUIThreadExecutor executeSyncWithTimeout:block:error:]",
"Line" : "235",
"TestCase Class" : "EarlGreyVidaTests.MyFirstEarlGreyTest",
"TestCase Method" : "testExample"
},
{
"Description" : "Failed to execute assertion within 30 seconds.",
"Domain" : "com.google.earlgrey.ElementInteractionErrorDomain",
"Code" : "4",
"File Name" : "GREYElementInteraction.m",
"Function Name" : "-[GREYElementInteraction assert:error:]",
"Line" : "418",
"TestCase Class" : "EarlGreyVidaTests.MyFirstEarlGreyTest",
"TestCase Method" : "testExample"
}
]
It works if I turn off synchronization:
GREYConfiguration.sharedInstance().setValue(false, forConfigKey: kGREYConfigKeySynchronizationEnabled)
Is there something I'm doing wrong? All my tests seem to work well with synchronization off. Should I keep it off?
You shouldn't turn the synchronization off - that's what makes EarlGrey EarlGrey, and makes your tests stable and useful.
However, it looks like you have some animations or delayed performSelector that EarlGrey is tracking to make it always think your app is busy. one thing you can tweak is, if you look at the header, GREYConfiguration.h, you can find various config variables you may change according to your app's behaviour.
Disclaimer: I am part of EarlGrey's team.
I'm having a problem I don't seem to find a solution for, and it looks really odd given i've tried everything I could with the official Spring Data Couchbase documentation.
Basically all i'm trying to do is a simple count() method.
My Repository :
public interface ICourrierRepository extends CrudRepository<Courrier, String> {
List<Courrier> findByCategorie(String categorie);
Long countByCategorie(String categorie);
#View(designDocument = "_design/courrier", viewName = "courrierBase")
long count();
}
The view is setup like this : http://img15.hostingpics.net/pics/169793Capture.png
And the view map is like this :
function (doc, meta) {
if (doc._class == "com.model.Courrier") {
emit(meta.id, null);
}
}
Worst thing is it actually works when i set a "reduce" to "_count" in the CouchBase GUI, but when I launch it from my client, I always get the same message, and the return is 0 :
[cb-computations-2] INFO c.c.c.java.view.ViewRetryHandler - Received a View HTTP response code (400) I did not expect, not retrying.
Thanks for any help...
I actually found the problem... it comes from this line :
#View(designDocument = "_design/courrier", viewName = "courrierBase")
which should be
#View(designDocument = "courrier", viewName = "courrierBase")
Also, the view should be set to reduce : _count.
Hope this helps future users !
I am building an IoT application. I am using PubNub to communicate between the harware and the user.
Now I need to store all the messages and data coming from the hardware and from the user in a central server. We want to do a bit of machine learning.
Is there a way to do this other than having the server subscribe to all the output channels (There will be a LOT of them)?
I was hoping for some kind of once-a-day data dump involving the storage and playback module in PubNub
Thanks in advance
PubNub to Server Data Transfer
Yes you can perform once-a-day data dumps involving the storage and playback feature.
But first check this out! You can subscribe to Wildcard Channels like a.* and a.b.* to receive all messages in the hierarchy below. That way you can receive messages on all channels if you prefix each channel with a root channel like: root.chan_1 and root.chan_2. Now you can subscribe to root.* and receive all messages in the root.
To enable once-a-day data dumps involving Storage and Playback, first enable Storage and Playback on your account. PubNub will store all your messages on disk over multiple data centers for reliability and read latency performance boost. Lastly you can use the History API on your server to fetch all data stored as far back as forever as long as you know the channels to fetch.
Here is a JavaScript function that will fetch all messages from a channel.
Get All Messages Usage
get_all_history({
limit : 1000,
channel : "my_channel_here",
error : function(e) { },
callback : function(messages) {
console.log(messages);
}
});
Get All Messages Code
function get_all_history(args) {
var channel = args['channel']
, callback = args['callback']
, limit = +args['limit'] || 5000
, start = 0
, count = 100
, history = []
, params = {
channel : channel,
count : count,
callback : function(messages) {
var msgs = messages[0];
start = messages[1];
params.start = start;
PUBNUB.each( msgs.reverse(), function(m) {history.push(m)} );
callback(history);
if (history.length >= limit) return;
if (msgs.length < count) return;
count = 100;
add_messages();
},
error : function(e) {
log( message_out, [ 'HISTORY ERROR', e ], '#FF2262' );
}
};
add_messages();
function add_messages() { pubnub.history(params) }
}
We have the same application built on the Managed SIP Application API that we wish to run twice on the same Front End server. They're using the same name, the same MSPL script and the same code except their configuration (.config) is slightly different.
What happens is the first instance we start works fine and is receiving SIP messages, but as soon as the second instance is started the first instance is no longer receiving any SIP messages and now only the second instance is receiving SIP messages. We would like for both of our instances to handle all the SIP messages which we've been unsuccessful in so far.
The MSPL script we're using to dispatch the SIP messages is:
<?xml version="1.0"?>
<r:applicationManifest
r:appUri="http://example.com/ProgramName"
xmlns:r="http://schemas.microsoft.com/lcs/2006/05">
<!-- Handle incoming requests to Lync server -->
<r:requestFilter methodNames="ALL"
strictRoute="true"
registrarGenerated="true"
domainSupported="true"/>
<!-- Handle outgoing requests to Lync server-->
<r:responseFilter reasonCodes="ALL"/>
<r:splScript>
<![CDATA[
if (sipRequest)
{
//Ignoring benotify events.
if(sipRequest.Method =="BENOTIFY")
{
return;
}
Dispatch("OnRequest");
}
else
{
Dispatch("OnResponse");
}
]]>
</r:splScript>
</r:applicationManifest>
which are handled by the following OnRequest and OnResponse messages:
public void OnRequest(object sender, RequestReceivedEventArgs evt)
{
// Do some stuff with evt.Message
evt.ServerTransaction.EnableForking = false;
evt.ServerTransaction.CreateBranch().SendRequest(message.Message as Request);
}
public void OnResponse(object sender, ResponseReceivedEventArgs evt)
{
// Do some stuff with evt.Message
evt.ClientTransaction.ServerTransaction.SendResponse(message.Message as Response);
}
Here's how these programs are registered as server applications (output from Get-CsServerApplication):
Identity : Service:Registrar:FQDN/ExampleProgramName
Priority : 12
Uri : http://example.com/ProgramName
Name : ExampleProgramName
Enabled : True
Critical : False
ScriptName :
Script :
Identity : Service:Registrar:FQDN/ExampleProgramName
Priority : 13
Uri : http://example.com/ProgramName
Name : ExampleProgramName
Enabled : True
Critical : False
ScriptName :
Script :
We've also tried registering the same application under different names. For example, we renamed one of the two instances to ExampleProgramName2 in the MSPL script as well as in New-CsServerApplication which did not work either.
In particular, we are unsure about the requestFilter in the MSPL script and the particular SendResponse/SendRequest method calls as well as the EnableForking property.
Register the server application with different names:
New-CsServerApplication -Identity FQDN/ExampleProgramName1 -Uri http://example.com/ProgramName1 -Critical $False
New-CsServerApplication -Identity FQDN/ExampleProgramName2 -Uri http://example.com/ProgramName2 -Critical $False
As well as give each program their unique name in MSPL matching the value given in the Uri parameter of New-CsServerApplication:
<r:applicationManifest
r:appUri="http://example.com/ProgramName1"
<r:applicationManifest
r:appUri="http://example.com/ProgramName2"
This should be enough.