I have a large table that I want to partition loaded in a kdb+ server. The partitioning code is in a Q script that I need to evaluate on the server, but I don't have I/O control over it (the server is running in the background, data in memory was loaded through the C API).
So far the only solution I've found is to open a handle and convert all my statements to strings (all those \" I have to use look quite messy):
if[4 <> count .z.x; 0N!"usage: part_md.q host port db_dir date (YYYY.MM.DD)"; exit[1]]
arg_host: .z.x 0
arg_port: .z.x 1
arg_db_dir: .z.x 2
arg_date: "D"$(.z.x 3)
/get server handle
h:hopen `$(":",arg_host,":",arg_port)
set_db: "db: \":",arg_db_dir, "/mydb/\""
set_sym_path: "sym_path: \":",arg_db_dir,"\""
h set_db
h set_sym_path
/select fields to partition
h "mydb_select: select [-10] A,B,C from mydb"
/enumerate symbols
h "md_select_enum: .Q.en[`$sym_path] md_select"
h "delete md_select from `."
...
How can I evaluate my script directly without using a handle to the server on the same machine?
You could use a kdb IDE like qStudio http://www.timestored.com/qstudio to send the query by entering commands exactly as you would as if at the console.
Few other suggestions:
Read up on the function form of inter process of communication, an alternative to always sending strings. More powerful as it allows using objects.
Avoid underscores in variable names as they are also operators in kdb. (See here for q language guidelines: http://www.timestored.com/kdb-guides/q-coding-standards )
Checkout the many variations of the sv function, particularly (` sv `:file,`name) it allows quickly forming file paths in a platform independent way.
Regards,
Ryan
Another option is studioForKdb+ though it does not offer as many features as qstudio http://code.kx.com/wiki/StudioForKdb%2B
Related
I'm trying to get a "retro-computing" class open and would like to give people the opportunity to finish projects at home (without carrying a 3kb monstrosity out of 1980 with them) I've heard that repl.it has every programming language, does it have QuickBasic and how do I use it online? Thanks for the help in advance!
You can do it (hint: search for QBasic; it shares syntax with QuickBASIC), but you should be aware that it has some limitations as it's running on an incomplete JavaScript implementation. For completeness, I'll reproduce the info from the original blog post:
What works
Only text mode is supported. The most common commands (enough to run
nibbles) are implemented. These include:
Subs and functions
Arrays
User types
Shared variables
Loops
Input from screen
What doesn't work
Graphics modes are not supported
No statements are allowed on the same line as IF/THEN
Line numbers are not supported
Only the built-in functions used by NIBBLES.BAS are implemented
All subroutines and functions must be declared using DECLARE
This is far from being done. In the comments, AC0KG points out that
P=1-1 doesn't work.
In short, it would need another 50 or 100 hours of work and there is
no reason to do this.
One caveat that I haven't been able to determine is a statement like INPUT or LINE INPUT... They just don't seem to work for me on repl.it, and I don't know where else one might find qb.js hosted.
My recommendation: FreeBASIC
I would recommend FreeBASIC instead, if possible. It's essentially a modern reimplementation coded in C++ (last I knew) with additional functionality.
Old DOS stuff like the DEF SEG statement and VARSEG function are no longer applicable since it is a modern BASIC implementation operating on a 32-bit flat address space rather than 16-bit segmented memory. I'm not sure what the difference between the old SADD function and the new StrPtr function is, if there is any, but the idea is the same: return the address of the bytes that make up a string.
You could also disable some stuff and maintain QB compatibility using #lang "qb" as the first line of a program as there will be noticeable differences when using the default "fb" dialect, or you could embrace the new features and avoid the "qb" dialect, focusing primarily on the programming concepts instead; the choice is yours. Regardless of the dialect you choose, the basic stuff should work just fine:
DECLARE SUB collatz ()
DIM SHARED n AS INTEGER
INPUT "Enter a value for n: ", n
PRINT n
DO WHILE n <> 4
collatz
PRINT n
LOOP
PRINT 2
PRINT 1
SUB collatz
IF n MOD 2 = 1 THEN
n = 3 * n + 1
ELSE
n = n \ 2
END IF
END SUB
A word about QB64
One might argue that there is a much more compatible transpiler known as QB64 (except for some things like DEF FN...), but I cannot recommend it if you want a tool for students to use. It's a large download for Windows users, and its syntax checking can be a bit poor at times, to the point that you might see the QB code compile only to see a cryptic message like "C++ compilation failed! See internals\temp\compile.txt for details". Simply put, it's usable and highly compatible, but it needs some work, like the qb.js script that repl.it uses.
An alternative: DOSBox and autorun
You could also find a way to run an actual copy of QB 4.5 in something like DOSBox and simply modify the autorun information in the default DOSBox.conf (or whatever it's called) to automatically launch QB. Then just repackage it with the modified DOSBox.conf in a nice installer for easy distribution (NSIS, Inno Setup, etc.) This will provide the most retro experience beyond something like a FreeDOS virtual machine as you'll be dealing with the 16-bit segmented memory, VGA, etc.—all emulated of course.
I want to have two MATLAB windows open on the same computer. The desired scenario is as follows: MATLAB window 1 is continuously running a script that has nothing to do with MATLAB window 2. At the same time, MATLAB window 2 is running a script that continuously checks for a certain condition, and if it is met, then it will terminate the script running on MATLAB window 1, and then terminate its own script as well. I want to have two MATLAB windows instead of one since I believe it will be more time efficient for what I am trying to do. I found an interesting "KeyInject" program at http://au.mathworks.com/matlabcentral/fileexchange/40001-keyinject , but I was wondering if there is a simpler way already built into MATLAB.
Do you want simple, or a flexible, infinitely expandable version 1.0? Simple would be to trigger System A via a file created by System B.
Simple would have System B create a file, then System A would check for the file with the command
if exist ( fileName, 'file' )
then do your shutdown commands. On startup, System A would delete the file with
delete ( fileName );
The second option is to use the udp command. UDP allows any data to be sent between processes, whether on the same computer or over a network. (See https://www.mathworks.com/help/instrument/udp.html for more info).
I see several ways:
Restructure to avoid this XY problem
Use (mat) files (as Hoki suggested), possibly using the parallel computing toolbox to keep everything in one MATLAB session.
Write some MEX functions that communicate with each other via a global pipe.
Write an Auto(Hot)key script.
Option 2 is probably easiest. Take a look at events and listeners if you write in OOP, otherwise, you'd have to poll inside a loop
Option 3 is harder and way more time consuming to implement, but allows for much faster detection of the condition, and much faster data transfer between the sessions. Use only if speed is essential...but I guess that doesn't apply :)
Option 4: the AutoHotkey solution is probably the most Horrible Thing® you could do on an already Horrible Construction®, but oh what fun!! In both MATLAB sessions, you create a (hidden) figure with the name Window1 or Window2, respectively. These window names are something that AutoHotkey can easily track. If the conditions are met, you update the corresponding window name, triggering the remainder of the AutoHotkey script: press a button in the other window! If you need to transfer data between the windows: you can create basic edit boxes in both GUIs, and copy-paste the data between them. If you're on Linux: you can use Autokey for the same purpose, but by then you're basically writing Python code doing the heavy lifting, so just use Python.
Or, you know, use KeyInject. Less fun.
I'm gradually working my way up the perl learning curve (with thanks to contributors to this REALLY helpful site), but am struggling with how to approach this particular issue.
I'm building a perl utility which utilises three (c++) third party programmes. Normally these are run: A $file_list | B -args | C $file_out
where process A reads multiple files, process B modifies each individual file and process C collects all input files in the pipe and produces a single output file, with a null input file signifying the end of the input stream.
The input files are large(ish) at around 100Mb and around 10 in number. The processes are CPU intensive and the whole process need to be applied to thousands of groups of files each day, so the simple solution of reading and writing intermediate files to disk is simply too inefficient. In addition, the process above is only part of a processing sequence, where the input files are already in memory and the output file also needs to be in memory for further processing.
There are a number of solutions to this already well documented and I have a prototype version utilising IPC::Open3(). So far, so good. :)
However - when piping each file to process A through process B I need to modify the arguments in process B for each input file without interrupting the forward flow to process C. This is where I come unstuck and am looking for some suggestions.
As further background:
Running in Ubuntu 16.04 LTS (currently within Virtual box)and perl v5.22.1
The programme will run on (and within) a single machine by one user (me !), i.e. no external network communication or multi user or public requirement - so simplicity of programming is preferred over strong security.
Since the process must run repeatedly without interruption, robust/reliable I/O handling is required.
I have access to the source code of each process, so that could be modified (although I'd prefer not to).
My apologies for the lack of "code to date", but I thought the question is more one of "How do I approach this?" rather than "How do I get my code to work?".
Any pointers or help would be very much appreciated.
You need a fourth program (call it D) that determines what the arguments to B should be and executes B with those arguments and with D's stdin and stdout connected to B's stdin and stdout. You can then replace B with D in your pipeline.
What language you use for D is up to you.
If you're looking to feed output from different programs into the pipes, I'd suggest what you want to look at is ... well, pipe.
This lets you set up a pipe - that works much like the ones you get from IPC::Open3 but have a bit more control over what you read/write into it.
I know how to copy records from one pf to another by reading one file in dow loop and writing into another file like below. Files are PF1 and PF2 having record format rec1 and rec2 respectively where each file have only one field named fld1 and #fld1 respectively-
READ PF1
DOW not %eof(PF1) and not %error
eval fld1 = #fld1
write Rec2
READ PF1
ENDDO
As the comments in Buck's answer mention, your team mate is alluding to using the RPG cycle to process the file. The cycle is basically an implicit read loop of files declared as 'P'rimary.
http://www-01.ibm.com/support/knowledgecenter/ssw_ibm_i_71/rzasc/sc09250726.htm%23wq121
Originally, even RPG IV programs included code to used as part of the cycle, such as automatically opening files, even if you didn't actually declare any input primary files. Now however, you can create "Linear Main" programs using the MAIN() h-spec and your program will be cycle free.
Using the cycle is frowned upon in modern RPG. Primarily because the implicit nature of what's going on makes it tricky to understand non-trivial code. Additionally, cycle code doesn't perform any better than non-cycle code; it's just less to write. The I/Os being done remain exactly the same.
Finally, again as mentioned in the comments. If you want to optimize performace, use SQL. The set based nature of SQL beats RPG's one row at a time. I haven't benchmarked it recently, but way back on v5r2 or so, copying 100 or more rows was faster with SQL than RPG.
For reference only, FWiW; i.e. not recommendations, just examples of what can be done, esp. in cases alluded but for which no specifics were given:
My team mate told me that he can write code for this problem only in 4 lines including declaration of both files in F-spec. He will also not use read, move or dow loop. I don't know how can he do this. That's why I am eager to know this.
The following source is an example Cycle-program; my FLD1 of REC1 had a 10-byte field but I described my output for 20-bytes, so to avoid failed compile per sev-20 RNF7501 "Length of data structure in Result-Field does not equal the record length of Factor 2.", I specified GENLVL(20) on the CRTBNDRPG:
FPF1 IP E DISK rename(rec1:rcd1)
FPF2 O F 20 DISK
DINOUT E DS EXTNAME(PF1)
C WRITE PF2 INOUT
I don't want to use CL program. I just want to do it with a single program either in RPG3 or RPG4
A similar RPG Cycle-program could perform effectively the same thing, similarly copying the data from PF1 to PF2 despite different column name and [thus inherently also] the different record format, using the CL command without a CL program and almost as few lines. The following example depends on the must-always-be-one-row table called QSQPTABL in QSYS2 that would typically be in the system Library List, and the second argument could reflect the actual length of the command string, but just as easily codes the max prototyped length per the Const definition assuring the blank-padding up to that length without actually having to count the [~53] bytes of the concatenated string expression:
FQSQPTABL IP E DISK rename(qsqptabl:qsqptable)
DQcmdExc PR ExtPgm('QSYS/QCMDEXC')
D 200A const
D 15P05 const
c callp QcmdExc('cpyf pf1 pf2 mbropt(*add)'
c +' fmtopt(*nochk) crtfile(*no)':200)
Whereas both of the above sources are probably an enigma to anyone unfamiliar with the Cycle, the overall effects of the latter are quite likely to be inferred correctly [¿perhaps more appropriately described as guessed correctly?], by just about anyone with an understanding of the CL command string, despite their lack of understanding of the Cycle.
And of course, as was also noted, with the SQL the program is probably arguably even easier\simpler; possibly even more readable to the uninitiated [although the WITH NONE clause, shown as WITH NC, added just in case the COMMIT(*NONE) was overlooked on the compile request, probably is not easily intuited]:
C/Exec SQL
C+ insert into pf2 select * from pf1 WITH NC
C/End-Exec
C SETON LR
P.S. The source-code from the OP was originally [at least was, prior to my comment added here] incorrectly coded with eval fld1 = #fld1 when surely what was intended was eval #fld1 = fld1 according to the setup\given.
If you need to use RPG, use embedded SQL. Look up INSERT INTO.
If you aren't limited to RPG, consider CPYF... MBROPT(*ADD).
What business problem are you trying to solve by doing it another way?
Does a command line tool ship with Informix 11.5 similar to SQLCMD for SQL Server?
If yes, how do I connect to a remote server and perform regular SELECT/INSERT/UPDATE queries using it?
As Michal Niklas says, the standard tool provided with IBM Informix Dynamic Server (colloquially IDS or even just Informix) is DB-Access. However, it is distributed only with IDS itself, not with the Informix Client SDK (CSDK) or Informix Connect (I-Connect) products.
If you want to access IDS from a machine that does not have IDS installed, you need either CSDK or I-Connect on the machine, and some other software - perhaps the original (pre-Microsoft by a decade and more) version of SQLCMD. This is what I use - and have used in various versions over the last (cough, splutter, ouch) twenty-two years or so; I wrote it because I didn't like the command line behaviour of a program called isql (part of the product Informix SQL), which was the pre-cursor to DB-Access. (Lot's of history - not too important to you.)
Usage - SQLCMD has more options than you know what to do with. The basics are simple, though:
sqlcmd -d dbname#dbserver -e 'select * from table' -x -f file.sql
This connects to a database called 'dbname' at the database server known as 'dbserver' as specified in the sqlhosts file (normally $INFORMIXDIR/etc/sqlhosts). The '-e' indicates an SQL expression - a select statement; the results will be printed to standard output in a strict format (Informix UNLOAD format), one logical line per record. The '-x' turns on trace mode; the '-f' option means read the named file for further commands. The '.sql' extension is not mandatory (beware: DB-Access requires the '.sql' extension and will add it for you). (Arguments not prefixed by either '-e' or '-f' are interpreted heuristically; if it contains spaces, it is SQL; if it does not, it is a filename.) The '-H' option prints column headings (labels) before a result set; the '-T' option prints the column types (after the headings, before the results). The '-B' option runs in benchmark mode; it turns on trace, prints the statement, the time when the statement started, and times how long it took. (Knowing when the statement started is helpful if the SQL takes many minutes to run - as it can in benchmarking scenarios). There are controls over the output format (including CSV and even variant of XML - but not an XML using namespaces) and date format, and so on. There are 'built-in' commands to redirect input and output and errors; most command line options can also be used in the interpeter, etc. SQLCMD also provides a history mechanism; it saves SQL statements and you can view, edit or rerun them. In conjunction with output redirection, you can save off a list of statements executed, etc.
The only gotcha with SQLCMD is that it is not currently ported to Windows. It did work on Windows once upon about 6 or 7 years ago. Since then, Microsoft's compilers have gotten antsy about non-MS API functions, insisting that even if I ask for them by name (by requesting POSIX functionality), the functions must be prefixed by an underscore, and by deprecating a bunch of functions that can be used safely if you pay attention to what you are doing (but, regrettably, can be abused by those who are not paying attention, and there are more inattentive than attentive coders around, it seems) - I mean functions like 'strcpy()' which can be used perfectly safely if you know the size of the source and destination strings before you call it. It is on my list of things to do - it just hasn't been done yet because it isn't my itch.
There is also another Open Source tool called SQSL that you can consider. It has some advantages over SQLCMD (conditional logic, etc) but I think SQLCMD has some advantages over SQSL.
You could also consider whether Perl + DBI + DBD::Informix + dbish would work for you.
Try DB-Access
...
DB–Access provides a user interface for entering, executing, and debugging Structured Query Language (SQL) statements and Stored Procedure Language (SPL) routines...