What is the fastest way (at run time) to have an Intersystems Cache database stored procedure return nothing but a BLOB? - intersystems-cache

This question is specifically about Intersystems-Cache databases.
I'm currently using $$$ResultSet("MySQLQueryText") to select the BLOB from a table, but this is probably writing the BLOB to the table, then reading out from the table, instead of writing directly to the output BLOB.
The .INT code compiles into code that creates a %Library.ProcedureContext object, then calls NewResultSet() on that object. However, the source for NewResultSet has a comment: "Used internally only, do not call directly".
Is there a supported way to efficiently create a result set that is nothing but a single record with a single, BLOB column? Ideally I'd like something like a stream object and write to that directly, and have that go straight to the ODBC (or other) driver without copying the stream. If there is a supported solution using another object that isn't exactly a stream that would also be great.

#psr - Based on the discussion in the comments, I believe that you should be able to use code something like the following:
/// Method that will take in various arguments and return a BLOB as an output argument
ClassMethod GetBLOB(
arg1 As %String,
arg2 As %String,
...
Output blob As %Stream.TmpBinary) [ SqlProc ]
{
// Do work to produce your BLOB
Set blob = yourBLOB
Quit
}
Actual support for the BLOB may depend on your client software and whether you are using ODBC or JDBC, but anything reasonably recent should not pose any problems.
You would invoke this stored procedure using syntax like:
CALL Your_Schema.YourClass_GetBLOB('arg1','arg2',?)
The actual method for retrieving the BLOB will then depend on your client software and access method. You can also control the stored procedure name (i.e. the piece after the schema) by adding SqlName = MyGetBLOB next to the SqlProc keyword.

Related

How do I "capture" and store the complex output of a Flutter function as code so I can write unit tests for it?

First of all, apologies if this is a stupid question. I'm new to unit tests so I'm struggling a bit here.
I'm working on an app that queries an API, receives a JSON response and then processes that response to produce a series of complex data structures. Many of these data structures are daily time series, which means each of my functions produces a list (List<Datapoint>) containing hundreds of datapoint objects.
What I'm trying to test is that, for a given API response, each function produces the output it should.
For the input of each test I have already grabbed a sample, real JSON response from the API, and I've stored it inside a test_data folder within my root test folder.
However, for the expect part... how can I obtain a sample output from my function and store it somewhere in my test_data folder?
It would be straightforward if the output of my function were a string, but in this case we're talking about a list with hundreds of custom objects containing different values inside them. The only way to create those objects is through the function itself.
I tried running the debugger to check the value of the output at runtime, which I can do... but that doesn't help me copy it or store it anywhere as code.
Should I try to print the full contents of the output to a string at runtime and store that string? I don't think this would work, as all I see in the console are a bunch of Instance Of when I do functionOutput.toString()... I would probably need to recursively print each of the variables inside those objects.
Please tell me I'm being stupid and there's a simpler way to do this :)

datajoint-matlab implementation of filepath#data datatype

For datajoint-matlab, #327 seems to indicate that File external storage (#143, PR #197) should be implemented in the current version. I can create a table with the datatype filepath#data, after defining the store 'data', but I get an error on insert.
Error using dj.Relvar/insert/makePlaceholder (line 244)
The field `fref` with datatype `filepath#data` is not yet supported.
Error in dj.Relvar/insert (line 334)
[v, placeholder] = makePlaceholder(i,
tuple.(header.attributes(i).name));
Is this still not implemented, or is the error-checking here just stopping me from using it? Happy to provide more details on the test if needed.
Maintainer for DataJoint here. Looks like there is a bit of confusion so let's see if I can help to bring some clarity. Hope to use this discussion as a resource to improve the documentation.
DataJoint provides a few DataJoint-only datatypes. Of these types, we identify the ones associated with external storage by embedding an # symbol. We classify each part of the type as <datatype>#<store>. Essentially with these types, information (i.e. datatype) is stored remotely in an object store (i.e. a store) with proper reference links within the relational database and client configuration for access.
For datatype, there are currently 3 options:
blob: Equivalent to a blob type but for external stores. Currently, this type is supported both in datajoint-python and most recently in datajoint-matlab.
attach: A special type that captures file content as binary information but does not preserve any path information. Currently, this type is supported only in datajoint-python. Documentation is available for this type in File Attachment Datatype section.
filepath: A special type that captures file content as binary information and includes path (along with file name) detail. Currently, this type is in preview in datajoint-python requiring it to be enabled. Documentation is available for this type in Filepath Datatype section. See note in documentation to enable it.
For store, there is an External Store section in the documentation. Multiple stores can be configured as a mapping located under the stores key in dj.config. For MATLAB, see help('dj.config') for examples and in Python, refer to the docs for attach and filepath above.
Stores currently support 2 protocols:
s3: Stores objects in an S3 bucket. Currently, this store is supported both in datajoint-python and most recently in datajoint-matlab.
file: Stores files in a directory accessible via the filesystem on the client. Currently, this store is supported both in datajoint-python and most recently in datajoint-matlab.
Issue #143 and PR's #197, #327 that you've mentioned refer to efforts in implementing file, s3 stores for the blob datatype in DataJoint MATLAB. The error you've experienced is expected and a simple placeholder until we support the other 2 datatypes in DataJoint MATLAB.

How can you call programs from a stored procedure?

I have followed other examples on stack exchange to call a built-in program from a stored procedure, but continue to get an error.
Working off this example (Looking for a working example of any OS/400 API wrapped in an external SQL stored procedure wrapped in a user defined SQL function) I built the following to attempt to create a wrapper command to allow me to change object security (my issue is that objects I create, regardless of library) are not always accessible to others in my some function, I must manually set to a common security group.
CREATE OR REPLACE PROCEDURE XX.TST( IN XOBJ CHAR(32), IN XOBJTYPE CHAR(10), IN XNEWOWN CHAR(10))
LANGUAGE CL
SPECIFIC XX.TST
NOT DETERMINISTIC
NO SQL
CALLED ON NULL INPUT
EXTERNAL NAME 'QSYS/CHGOBJOWN'
PARAMETER STYLE GENERAL;
CALL XX . TST('XX/TBL1','*FILE','GRPFRIENDS');
I get the following error:
External program CHGOBJOWN in QSYS not found
But have confirmed that going to the CL of the terminal emulator and typing QSYS/CHGOBJOWN takes me into the parameter input screen
You are trying to define a command as a program, and that just won't work. A command object (*CMD) and a program object (*PGM) are two different things, and cannot be invoked the same way. All is not lost though. There is a DB2 service that allows you to execute commands. You just have to build the proper command string.
Instead of defining a stored procedure, you can call the existing DB2 service like this:
call qsys2.qcmdexec('CHGOBJOWN OBJ(XX/TBL1) OBJTYPE(*FILE) NEWOWN(GRPFRIENDS)');
There are a whole list of services. Documentation can be found here.

Azure Data Factory - 'variables' is not a recognized function

We are using an Azure SQL Database sink in a Copy Activity.
The requirement is for us to execute a stored procedure here via the "Pre-Copy Script" property of the Sink. We are using Dynamic Content, passing in a "ProcessName" parameter. As you can see, we have a ProcessName variable, and it is used in a call to #concat() to build the stored procedure string for this Sink property.
However, any time we use the variables collection in Dynamic Content, we get this warning as shown in the image. The warning states:
'variables' is not a recognized function
Is there a way to avoid having this "Warning" in the UI? It works fine, but it looks terrible. It appears everywhere we use variables, not just in this case.
Try using pipeline parameters instead of variables and calling it as explained here: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions
You are most likely going to have to change your logic to do this. But if it works fine as it is, I wouldn't mind changing logic or code just to stop seeing the warning in the UI.
Hope this helped!

loading parameter files for data different sets

I need to analyse several sets of data which are associated with different parameter sets (one single set of parameters for each set of data). I'm currently struggling to find a good way to store these parameters such that they are readily available when analysing a specific dataset.
The first thing I tried was saving them in a script file parameters.m in the data directory and load them with run([path_to_data,'/parameters.m']). I understand, however, that this is not good coding practice and it also gave me scoping problems (I think), as changes in parameters.m were not always reflected in my workspace variables. (Workspace variables were only changed after Clear all and rerunning the code.)
A clean solution would be to define a function parameters() in each data directory, but then again I would need to add the directory to the search path. Also I fear I might run into namespace collisions if I don't give the functions unique names. Using unique names is not very practical on the other hand...
Is there a better solution?
So define a struct or cell array called parameters and store it in the data directory it belongs in. I don't know what your parameters look like, but ours might look like this:
parameters.relative_tolerance = 10e-6
parameters.absolute_tolerance = 10e-6
parameters.solver_type = 3
.
.
.
and I can write
save('parameter_file', 'parameters')
or even
save('parameter_file', '-struct', 'parameters', *fieldnames*)
The online help reveals how to use -struct to store fields from a structure as individual variables should that be useful to you.
Once you've got the parameters saved you can load them with the load command.
To sum up: create a variable (most likely a struct or cell array) called parameters and save it in the data directory for the experiment it refers to. You then have all the usual Matlab tools for reading, writing and investigating the parameters as well as the data. I don't see a need for a solution more complicated than this (though your parameters may be complicated themselves).