run stanford parser interactively (using stdin and stdout) or run it as a server - subprocess

I found it inefficient to reboot the parser when new input comes, so I'd like to run the parser interactively--read the input from stdin and print result to stdout. However, the instruction given on the official website Can I have the parser run as a filter? seems not compatible with options (for example, -port).
I know that CoreNLP can be run as a server but it can not receive POS tagged text as input so I won't use it.
Here is what I'm trying:
class myThread(threading.Thread):
def __init__(self,inQueue,outQueue):
threading.Thread.__init__(self)
self.cmd=['java.exe',
'-mx4g',
'-cp','*',
'edu.stanford.nlp.parser.lexparser.LexicalizedParser',
'-model', 'edu/stanford/nlp/models/lexparser/chinesePCFG.ser.gz',
'-sentences', 'newline',
'-outputFormat', 'conll2007',
'-tokenized',
'-tagSeparator','/',
'-tokenizerFactory', 'edu.stanford.nlp.process.WhitespaceTokenizer',
'-tokenizerMethod', 'newCoreLabelTokenizerFactory',
'-encoding', 'utf8']
self.subp=subprocess.Popen(cmd,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
self.inQueue=inQueue
self.outQueue=outQueue
def run(self):
while True:
rid,sentence=self.inQueue.get()
print(u"Receive sentence %s"%sentence)
sentence=sentence.replace("\n","")
self.subp.stdin.write((sentence+u'\n').encode('utf8'))
self.subp.stdin.flush()
print("start readline")
result=self.subp.stdout.readline()
print("end readline")
print(result)
self.outQueue.put((rid,result))

I think you're confusing things a bit. Both CoreNLP and Stanford Parser have an option to run as a command-line filter, reading from stdin and writing to stdout. However, only CoreNLP separately provides a webservice implementation.
Options like port only make sense for the latter.
So, at the moment, I agree that you have a valid use case (wanting to input pre-tagged text) but at present there isn't webservice support for it. The easiest path forward would be to write a simple webservice implementation for the parser. For us, it could happen sometime, but there are a bunch of other current priorities. Anyone else is welcome to write one. :)

Related

Discord.py mod log setup

I was wondering how one could make a command that sets up a mod log for a certain server, or start logging in one that already exists. This will probably require a database or JSON, anything is fine, as long as I can get it to work. Any help would be appreciated.
The way I would do this is with a .txt file that would contain all of the logs. I would add a line every time an event within the guild occurred, you could configure this to whatever events you wanted.
import datetime
#client.event
async def on_member_join(member):
with open("logs.txt", "a") as logsFile:
logsFile.write("\n[{}] {} just joined the server".format(datetime.datetime.now(),
member.name))
If you are currently using the logging module for discord.py or any other libraries, you can also use it for logging your own messages, either using the same or a separate logger.

Ida pro gragh output batch mode

Can anyone let me know how we are going to output all the subroutine's graphs in batch mode suing IDC. i.e. I have 447 subroutine's and wanna be output them all and I would like to make sure I first retrieve all the routines address automatically, cuz by knowing the address I can simply use GenFuncCall.
P.S: Is this the only cfg that I can get from Ida Pro given a binary dis-assembled file?
I needed a CFG of my whole program,the base example I started from was:
https://code.google.com/p/idapython/source/browse/trunk/examples/ex_gdl_qflow_chart.py
It uses the flow chart class:
https://www.hex-rays.com/products/ida/support/idapython_docs/idaapi.FlowChart-class.html
also worth noting to trigger in batch mode, you'll want something like this
idal64 -A -S{yourscriptname}.py {yourbinary}
Tips:
Prototype the script in the IDAPro gui first
Opening of the graph processor can cause timing issues, its hacky, but something like delaying execution of the script seemed to help, e.g.
idaapi.autoWait()
Timer(2, idacfg).start()
where idacfg is your python function from the example
print to stdout doesn't seem to work in batch mode, so you'll want to set stdout to a file for your debugging.
Closing the GUI in batch mode is still an issue for me.
Hope that helps.
If you just want the address of all known functions in the IDB, you could use something like this using IDAPython (just an example):
def main():
for count, func_ea in enumerate(Functions()):
if func_ea == BADADDR:
break
func_name = GetFunctionName(funcea)
func_start = func_ea
print("[{:4}] name: {}; start address: {:#x}".format(count, func_name, func_start))
if __name__ == "__main__":
main()

HTML form to invoke XQuery files

We have quite a number of .xqy files in several folders.
Sometimes, I need to invoke an .xqy file (via Marklogic's CQ) to test if it's working.
But I find it rather cumbersome to have to know what parameters to pass in and specify them in the xquery in CQ.
Is there a tool out there that would generate an HTML form that presents to me the parameters of a given .XQY file and invokes it when I press a "submit" button ?
If there is none out there, would somebody here know of how to make such an HTML form ? Right now, I can't seem to find any readily-available xdmp or xquery commands to tell me if an .XQY file is invocable or what parameters it expects to be fed.
Danny
The XQuery standard doesn't support introspection, nor does MarkLogic provide any functions that help with that. The closest you can get with this is using the XQDoc documentation code that is capable of parsing the XQuery code itself and producing descriptions of all function signatures within modules.
You can find more details about it here: http://developer.marklogic.com/code/xqdoc-ws
It doesn't provide a 'Submit' button, but using the XML output of xqdoc, you could make that yourself..
Good luck!
Another option would be to convert your XQuery to XQueryX and you can then process this as XML using XQuery (or XSLT) to generate an XForm or XHTML Form.
Since Danny mentioned unit testing, I'll risk plugging my own framework for that: https://github.com/mblakele/xqut
I usually run XQUT test suites in a cq buffer, but it would be easy enough to wrap one in a simple web page - with or without a form.

How can I create RRD files in Perl?

I have a separate application printing logs in every 10 seconds. I need to create RRD files from the log files. I need some Perl code to read the log files and create the RRD only without the graphs.
I have also gone through the available Perl module in CPAN, i.e. RRD::Simple and RRD::Simple::Examples, but I still need help.
I'd start with RRD::Simple. There's some example code in the documentation. Since you don't need to create a graph, simply skip that section of the example.
Some of the examples read a single sample of data, call the update function once, and then exit. Those scripts are meant to be run periodically to collect data in real time. The example that's probably more pertinent to your needs is ApacheAccessLogActivity.pl, which reads an Apache log file, parses each line with a regular expression, does a bit of analysis to figure out what it just read, and then calls update, all in a loop. Note that that example uses the standalone functions rather than the object-oriented versions.
If you've already read the documentation for that module and need more information about how to use it, or if you've tried it and found that it has shortcomings that prevent you from using it, then please be more specific about what you need to do.
RRDTool::OO also looks promising.
I'd recommend RRDTool::OO.
Exerpt from the perldoc:
$rrd->create( ... )
Creates a new round robin database (RRD). A RRD consists of one or
more data sources and one or more archives:
$rrd->create(
step => 60,
data_source => { name => "mydatasource",
type => "GAUGE" },
archive => { rows => 5 });

How can I read the URL-Data send with POST in Perl?

I'm trying to read out the POST-Data that was sent from a form in a page to my Perl Script. I googled and found out that:
read(STDIN, $param_string, $ENV{'CONTENT_LENGTH'})
reads out the whole Data-String with and writes the whole string to $param_string in the form of
Param1=Value1&Param2=Value2&Param3=Value3
by spliting it at the right places I get the necessary Data.
But I wonder why my $param_string is empty.
When I try the whole thing with GET:
$param_string = $ENV{'QUERY_STRING'};
everything works fine. Does anybody have an idea?
There absolutely no real reason for someone at your level to want to hand parse CGI requests.
Please use CGI::Simple or CGI.pm.
CGI.pm has a lot of baggage (HTML generation, function oriented interface) which makes CGI::Simple preferable.
Using any CGI processing module on CPAN is better than trying to write CGI processing code from scratch.
See parse_query_string in CGI::Simple for a way of accessing parameters passed using the query string when processing a form that is POSTed to your script.
If you want to learn how to do it right, you can read the source code of either module. Reading through the CGI.pm CHANGES file is also instructive.
If you are able to retrieve GET-data but not able to retrieve POST-data, most likely you forgot to change form method from to be post. You can check your submit method by using this condition in if statement:
if ($ENV{'REQUEST_METHOD'} eq "POST"){
read(STDIN, $param_string, $ENV{'CONTENT_LENGTH'});
}else {
$param_string = $ENV{'QUERY_STRING'};
}
Under mod_perl 2, Apache2::Request works for me.