Detect and prevent processing of sequential duplicate Tradingview webhook json - variable-assignment

I am not very well acquainted with python, but have by much trial and error, and building upon the work of others, developed a simple working and reliable interface between Tradingview (TV) and Interactive Brokers (IB).
I have situation where occasional apparent repainting of TV charts results in duplicate orders being submitted. (Yes, I have researched methods whereby such repainting can be reduced or eliminated, but the effect of these methods greatly reduces the proven profitability of my trading algo.) I am attempting to modify my code to detect and prevent processing of duplicate orders received in sequence, as I never intend to have more than one order working for any given security and any given time. My modification attempt (see below) is quite rudimentary--simply detect whether the just-received webhook json is essentially the same as that received just previously, and ignore if so.
This is where, if you wish, you can throw stones and tell me to go read about 'ver' and 'let' and 'global'. I've tried--for hours, and just don't "get it". I'm in my 70s and come from the era of Fortran, Cobol, and Basic, when variables were simply declared (if necessary), initialized, and used. Could someone kindly help me simply to get some starting values into the two variables, xticker and xorder, in the code below? Or better yet, show me a better way to accomplish my objective?
Many thanks for any help offered!
[Edit] - Repainting noted above may possibly have resulted from my setting "calc_on_order_fills=true" and/or "process_orders_on_close="true" while also setting "calc_on_every_tick=true". I have changed the first two of these settings to "false", but have no way to ascertain if doing so gets rid of the TV alert/webhook duplicates. But irrespective of whether these script modifications are beneficial, I would still like to add a "detect and ignore duplicates" feature to my TV to IB bridge as described.
#Create root
#app.route('/', methods=['GET','POST'])
async def root(request):
return response.text('online')
#Listen for signals and execute orders
#app.route('/webhook', methods=['GET','POST'])
async def webhook(request):
if request.method == 'POST':
await checkIfReconnect()
#Parse alert data
alert = request.json
order = MarketOrder(alert['order_action'],alert['order_contracts'],account='aannnnnnn')
ticker = alert['ticker']
# Attempt to detect and prevent processing of duplicate webhook json
# Fails since xticker and xorder are undefined and have no value.
# Need to initialize xticker='dummy' and xorder='dummy' at start
# Where/how to place these two initializing statements?
if not ticker == xticker and xorder == order:
#Do stuff (process and submit valid order)
#then ...
xticker = ticker
xorder = order
return HTTPResponse("ok", 200)
return HTTPResponse("", 405)
I have tried setting xticker and xorder directly using =, var, and let, and have tried but failed to understand how local and global variable assignment works within Python. My next effort, if no assistance is offered, will be to make use of read & write operations to an external file in order to draw in initial values and then to keep track of the last valid json data-- Crude but perhaps workable-- and there must be a great many simpler and better methods, all of which are beyond my ken.

Solved my own problem using global variables after finding some examples online. Solution works as intended, with sequential duplicate orders being ignored. Relevant code extract follows:
#Create root
#app.route('/', methods=['GET','POST'])
async def root(request):
return response.text('online')
#Listen for signals and execute orders
#app.route('/webhook', methods=['GET','POST'])
async def webhook(request):
if request.method == 'POST':
await checkIfReconnect()
#Parse alert data
alert = request.json
order = MarketOrder(alert['order_action'],alert['order_contracts'],account='aannnnnnn')
ticker = alert['ticker']
action = alert['order_action']
source = alert['source']
if not (ticker == xticker and xaction == action and xsource == source):
def inner():
global xticker
xticker = ticker
global xaction
xaction = action
global xsource
xsource = source
inner()
#Do stuff (prepare and submit order to broker)
return HTTPResponse("ok", 200)
return HTTPResponse("", 405)
#Reconnect if needed
async def checkIfReconnect():
if not app.ib.isConnected() or not app.ib.client.isConnected():
app.ib.disconnect()
app.ib = IB()
app.ib.connect('127.0.0.1',7497,clientId=int(int((time.time()*1000)-1667740000000)/1000000))
#Run app
if __name__ == '__main__':
#Connect to IB
app.ib = IB()
xticker = "ticker"
xaction = "action"
xsource = "source"
app.ib.connectAsync('127.0.0.1',7497,clientId=int(int((time.time()*1000)-1667740000000)/1000000))
app.run(port=5000)

Related

SNMPTraps: pysnmp and "disableAuthorization"

I have written an SNMP trap receiver but currently I have to hardcode all the SNMPv2 community strings to be able to receive the traps.
How do I emulate the 'disableAuthorization' functionality from snmptrapd in pysnmp?
I have tried to not set the community string: config.addV1System(snmpEngine, 'my-area') but this errors about a missing param. I have also tried an empty string: config.addV1System(snmpEngine, 'my-area', '') but this stops all traps being processed.
What is the best way to allow receiving all traps through pysnmp regardless of the community string they were sent with? I haven't found anything in the pysnmp docs that could help me
I had made progress on setting up an observer for V1/2 (V3 to be added later) that picked up on notifications with an unknown community string and then called addV1System on the fly to dynamically add it in, like so:
When setting up the transportDispatcher:
snmpEngine.observer.registerObserver(_handle_unauthenticated_snmptrap,
"rfc2576.prepareDataElements:sm-failure", "rfc3412.prepareDataElements:sm-failure")
And then:
def _handle_unauthenticated_snmptrap(snmpEngine, execpoint, variables, cbCtx):
if variables["securityLevel"] in [ 1, 2 ] and variables["statusInformation"]["errorIndication"] == errind.unknownCommunityName:
new_comm_string = "%s" % variables["statusInformation"].get("communityName", "")
config.addV1System(my_snmpEngine, 'my-area', new_comm_string)
return
else:
msg = "%s" % variables["statusInformation"]
print(f"Trap: { msg }")
However, this will always throw away the first trap received while adding any new community string (and then there is the problem whereby when the daemon is restarted the updated list of community strings is lost).
In looking to improve this I then found hidden away in the docs this little gem:
https://pysnmp.readthedocs.io/en/latest/examples/v3arch/asyncore/manager/ntfrcv/advanced-topics.html#serve-snmp-community-names-defined-by-regexp
This example receives a notification and rewrites the community string into 'public' so all traps will be correctly received.
In essence, when setting up the transportDispatcher:
my_snmpEngine.observer.registerObserver(_trap_observer,
'rfc2576.processIncomingMsg:writable', cbCtx='public')
And then:
def _trap_observer(snmpEngine, execpoint, variables, community_string):
variables['communityName'] = variables['communityName'].clone(community_string)

Powershell Index math

I have a couple scenarios where a series of arbitrary string values have a specific priority sequence. For example:
forgo > log > assert
exception > failure > error > warning > alert
And, I need to evaluate an arbitrary number of scenarios that resolve to one of those values and maintain a running status. So, using the simpler example, if every evaluation is assert then the final running status would be assert. But if a single evaluation is log and the rest are assert, the the final running status is log. And a single forgo means the final status is forgo, no matter what the mix of individual status results was. I want to provide a human readable running status and individual status, do the math to determine what the new running index is, then return the human readable status.
So, I have this, and it works.
$statusIndex = #('assert', 'log', 'forgo')
$runningStatus = 'forgo'
$individualStatus = 'log'
$runningStatusIndex = $statusIndex.indexof($runningStatus)
$individualStatusIndex = $statusIndex.indexof($individualStatus)
if ($individualStatusIndex -gt $runningStatusIndex) {
$runningStatusIndex = $individualStatusIndex
}
$runningStatus = $statusIndex[$runningStatusIndex]
But, this feels like something that happens often enough that there may be a more "native" way to do it. Some built in PowerShell functionality that handles the same thing more elegantly and in less code.
Is my intuition correct, and there is a native way? Or, perhaps a more elegant approach than what I have here?
Put your terms in an array, in precedence order:
$statusIndex = #('forgo', 'log', 'assert')
Aggregate all your status values and remove any duplicates:
$statusValues = 'assert', 'assert', 'log', 'assert'
$statusValueSet = $statusValues |Sort -Unique
Now use the .Where() extension method to select only the first matching term from the precedence list:
$overallStatus = $statusIndex.Where({$_ -in $statusValueSet}, 'First')
Value of $overallStatus should now be 'log' as expected.
3-4 lines of code instead of 9-10 :)

Play/Scala Template Block Statement HTML Output Syntax with Local Variable

Ok, I've been stuggling with this one for a while, and have spent a lot of time trying different things to do something that I have done very easily using PHP.
I am trying to iterate over a list while keeping track of a variable locally, while spitting out HTML attempting to populate a table.
Attempt #1:
#{
var curDate : Date = null
for(ind <- indicators){
if(curDate == null || !curDate.equals(ind.getFirstFound())){
curDate = ind.getFirstFound()
<tr><th colspan='5' class='day'>#(ind.getFirstFound())</th></tr>
<tr><th>Document ID</th><th>Value</th><th>Owner</th><th>Document Title / Comment</th></tr>
}
}
}
I attempt too user a scala block statement to allow me to keep curDate as a variable within the created scope. This block correctly maintains curDate state, but does not allow me to output anything to the DOM. I did not actually expect this to compile, due to my unescaped, randomly thrown in HTML, but it does. this loop simply places nothing on the DOM, although the decision structure is correctly executed on the server.
I tried escaping using #Html('...'), but that produced compile errors.
Attempt #2:
A lot of google searches led me to the "for comprehension":
#for(ind <- indicators; curDate = ind.getFirstFound()){
#if(curDate == null || !curDate.equals(ind.getFirstFound())){
#(curDate = ind.getFirstFound())
}
<tr><th colspan='5' class='day'>#(ind.getFirstFound())</th></tr>
<tr><th>Document ID</th><th>Value</th><th>Owner</th><th>Document Title / Comment</th></tr>
}
Without the if statement in this block, this is the closest I got to doing what I actually wanted, but apparently I am not allowed to reassign a non-reference type, which is why I was hoping attempt #1's reference declaration of curDate : Date = null would work. This attempt gets me the HTML on the page (again, if i remove the nested if statement) but doesn't get me the
My question is, how do i implement this intention? I am very painfully aware of my lack of Scala knowledge, which is being exacerbated by Play templating syntax. I am not sure what to do.
Thanks in advance!
Play's template language is very geared towards functional programming. It might be possible to achieve what you want to achieve using mutable state, but you'll probably be best going with the flow, and using a functional solution.
If you want to maintain state between iterations of a loop in functional programming, that can be done by doing a fold - you start with some state, and on each iteration, you get the previous state and the next element, and you then return the new state based on those two things.
So, looking at your first solution, it looks like what you're trying to do is only print an element out if it's date is different from the previous one, is that correct? Another way of putting this is you want to filter out all the elements that have a date that's the same date as the previous one. Expressing that in terms of a fold, we're going to fold the elements into a sequence (our initial state), and if the last element of the folded sequence has a different date to the current one, we add it, otherwise we ignore it.
Our fold looks like this:
indicators.foldLeft(Vector.empty[Indicator]) { (collected, next) =>
if (collected.lastOption.forall(_.getFirstFound != next.getFirstFound)) {
collected :+ next
} else {
collected
}
}
Just to explain the above, we're folding into a Vector because Vector has constant time append and last, List has n time. The forall will return true if there is no last element in collected, otherwise if there is, it will return true if the passed in lambda evaluates to true. And in Scala, == invokes .equals (after doing a null check), so you don't need to use .equals in Scala.
So, putting this in a template:
#for(ind <- indicators.foldLeft(Vector.empty[Indicator]) { (collected, next) =>
if (collected.lastOption.forall(_.getFirstFound != next.getFirstFound)) {
collected :+ next
} else {
collected
}
}){
...
}

Handle errors and control breaks with Spring Batch

I have almost completed my work with Spring Batch, it's working but then I have problems to handle the errors. I'll make a simple example:
I read one flat file, that I (later) map with 3 variables:
ID CODE NAME
AAA3333333Alex
AAA3333333Mark
BBB4444444Paul
I want the reader to read the flat file with a control break (I don't know if it's the right term in english, in italian it's something like "key break"): I read the elements with the same ID and CODE and only when the key changes return them to the reader:
while ((line = (Person) peek()) != null) { //while there are elements to read
if (line.getId().equals(prevElement.getId()) && line.getCode().equals(prevElement.getCode())
//do something
}
This works fine: when the ID or CODE changes I return the elements to the writer.
To make this work I had to set the commit-interval to 1 from the application-context. The thing is that in the worst case, if the elements are different for each line, I commit every single element and it becomes all very very slow.
So I said: let's put an outer control. Instead of returning the elements to the writer each time the key changes, I put them in a list, and then I return the list every 200 key changes (like a...handmade commit-interval):
while (controlBreakCount < 200 &&) {
while (!exit && (line = (Person) peek()) != null) { //while there are elements to read
if (line.getId().equals(prevElement.getId()) && line.getCode().equals(prevElement.getCode())
//do something
else { //if the key changes
//there is a controlBreakCount++; to increase the count
//add the elements to a list
}
}
}
return the list
and this works too (the real code has more controls, but this was to explain in a simple way).
The problem comes here: how to handle the errors with the listener in this case. With the outer while I have put (the one with the controlBreakCount), if even one of the 200 elements has an error all the elements currently in the list go to the listener and so it's very difficult to recognize the element with the error.
I guess my solution is not the best way to handle the "control break", but I can't find really much about this (and I'm not very pro with Spring Batch)...may I have some help?
Thank you very much :)
The 'do something' in your code is suspect... :/ Which operations do you do on the element just read? In reader you should only aggregate items and pass them to processor or writer.
IMHO you have to set a commit-interval greater than 1 to make process faster and try to group your data with same ID+CODE key in your own custom reader and THEN pass to writer in this way:
class PersonList {
String id;
String code;
List<Person> persons = new ArrayList<Person>();
}
when you find a keybreak (ID+CODE differs from previous) you have to create a new PersonList object and until ID+CODE are the same of previous line add to current PersonList.person list.
If your problem is about manage single line error with my solution you you can manage in your reader and, if you want, skip items with same ID+CODE in your reader as well.
Your reader must change its signature to MyPersonItemReader<PersonList> and your writer will write PersonList objects, but you have under your control the object (with ID+CODE).
I hope I had understood correctly your problem.

How do I Benchmark RESTful Service with Variable Parameters?

I'm currently working on benchmarking a RESTful service I've made, and part of that is making sure it runs in a reasonable amount of times for a large array of parameters. For example, let's say I have RESTful API of the form some_site.com/item?item_id=y. In that case to be sure my service is working as fast as I'd like it to work, I'd want to try out many values for y one by one, preferably coming from some text file. I can't figure out any way of doing this in ab or httperf. I'm open to using a different benchmarking program if I have, but would prefer something simple and light. What I want to do seems like something pretty standard, so I'm guessing there must already be a program that let's me do it, but an hour or so of googling hasn't gotten me an answer. Ideas?
Answer: Jmeter (which is apparently awesome). This faq explains how to do it. Hopefully this helps someone else, as it took me like a day of searching to figure this out.
I have just had some good experience with using JavaScript (via BSF/Rhino) in JMeter.
I have put one thread group in my test plan and stick a 'Simple Controller' with two elements under it - 'HTTP Request' sampler and 'BSF PreProcessor'.
Set BSF language to 'javascript' and either type the code into the text box or point it to a file (use full path or relative to CWD of JMeter process).
/* Since `Math.random()` gives us float, we use `java.util.Random()`
* see: http://docs.oracle.com/javase/7/docs/api/java/util/Random.html */
var Random = new Packages.java.util.Random();
var min = 10-1;
var max = 2;
var maxLines = (min)+Random.nextInt(max-min);
var s = '';
for (var d = 0; d <= maxLines; d++) {
s += d.toString()+','+Random.nextInt(1000).toString()+'\n';
}
// s => '0,312\n1,104\n2,608\n'
vars.put('PAYLOAD', s);
Now I can refer to ${PAYLOAD} in the HTTP request!
You can generate JSON, but you will need to upgrade jakarta-jmeter-2.5.1/lib/js-1.6R5.jar with the newest version of Rhino to get JSON.stringify and JSON.parse. That worked perfectly for me also, though I thought I'd put a simple example here.
You can use BSF pre-processor for URL params as well, just set another variable with vars.put('X', 'some value') and pass it as ${X} in the request parameter.
This blog post helped quite a bit, by the way.