I have a basic mapping:
one relational source and one flat file source
a joiner transformation
an expression transformation
finally the target table
I know it is possible to somehow query the metadata, but I do not know how or what tool to use.
How can I get a list of all starting source columns(output ports at source) linked to their final target column(input ports at target)?
Having the intermediate steps (example, source to joiner, joiner to expression, expression to target), would be very helpful as well, but source to target is the most important.
Querying metadata is not very convenient. You can try to export your mapping to XML and use http://powercenter-xmlanalyzer.appspot.com/ to do the source-to-target dependency analysis for you.
Related
There must be a part in the query planner of Postgres that is responsible for identifying which index to use based on various information (relation, column name, operator class/family, statistics, etc.).
I know that the source code of Postgres is available online but I would like a direct link to the part that performs the access path selection. The codebase is big and I can't find the relevant part.
The possible index access paths are found in the function create_index_paths in src/backend/optimizer/path/indxpath.c.
How to get column name and data type returned by a custom query in postgres? We have inbuilt functions for table/views but not for custom queries. For more clarification I would say that I need a postgres function which will take sql string as parameter and will return colnames and their datatype.
I don't think there's any built-in SQL function which does this for you.
If you want to do this purely at the SQL level, the simplest and cheapest way is probably to CREATE TEMP VIEW AS (<your_query>), dig the column definitions out of the catalog tables, and drop the view when you're done. However, this can have a non-trivial overhead depending on how often you do it (as it needs to write view definitions to the catalogs), can't be run in a read-only transaction, and can't be done on a standby server.
The ideal solution, if it fits your use case, is to build a prepared query on the client side, and make use of the metadata returned by the server (in the form of a RowDescription message passed as part of the query protocol). Unfortunately, this depends very much on which client library you're using, and how much of this information it chooses to expose. For example, libpq will give you access to everything, whereas the JDBC driver limits you to the public methods on its ResultSetMetadata object (though you could probably pull more information from its private fields via reflection, if you're determined enough).
If you want a read-only, low-overhead, client-independent solution, then you could also write a server-side C function to prepare and describe the query via SPI. Writing and building C functions comes with a bit of a learning curve, but you can find numerous examples on PGXN, or within Postgres' own contrib modules.
I am making two separate SimBiology Models with the same compartments but different species. Transport between the compartments are different for each species. I want to combine the two models so I can add an interaction term between the two species and use the simulations to get concentration profiles of both the species. Is there a function to do that in SimBiology? I have not been able to find one. If not, what would be the best approach to code a function to do so?
The most relevant function is copyobj. You can also use the SimBiology Desktop's graphical interface to copy and paste one model into another. I also have a prototype of a function for combining two models that I would be happy to share. If you're interested in that, please contact me via my MathWorks community profile.
Your best bet would be to write a script based on copyobj function provided by Matlab, if you want to copy programmatically. You need to be little cautious here as this function throws an error, IF the object copied and the target model have name conflicts. For example, if you want to copy a compartment by name 'C1' from a source model to a target model that already contains a compartment by the same name ('C1'), then the program throws an error due to name conflicts. In such a case you have to programmatically rename the source 'C1' compartment before you copy it to the target model. The rename command provided by Matlab might be useful.
Even after overcoming the errors due to name conflicts, there is another problem with the above method --- the layout/diagram of the models will not be preserved. If you want to preserve the model layout (that is the diagram of the model), you can use GUI based simbiology model builder. First you manually select the model/submodel that you want to copy, click 'Ctrl+C' (like you would do for copying a text) and paste (Ctrl+V) it in your target model. The name conflicts will be automatically handled by the matlab. That is the source 'C1' compartment will be automatically renamed by matlab (to probably 'C2'), and pasted in the target model.
You may look at this function, that probably helps you to build your custom model merger programmatically. It is an old function that works with older versions of Matlab, and may not be relevant now. But it may provide some guidance on building your own script.
I'm using scio from spotify for my Dataflow jobs.
In last scio version, new bigtable java api is used (com.google.bigtable.v2)
Now scio bigtable entry point required "RowFilter" to filter instead of Hbase "Scan". Is there a simple way to transform "Scan" to "RowFilter" ? I looked for adapters in source code but I'm not sure how to use it.
I don't find documentation to easy migrate from hbase api to "new" api.
A simple scan I used in my code that I need to transform:
val scan = new Scan()
scan.setRowPrefixFilter("helloworld".getBytes)
scan.addColumn("family".getBytes, "qualifier".getBytes)
scan.setMaxVersions()
In theory, you can add the bigtable-hbase dependency to the project and call com.google.cloud.bigtable.hbase.adapters.Adapters.SCAN_ADAPTER.adapt(scan) to convert the Scan to a RowFilter, or more specifically a [ReadRowsRequest][3] which contains a [RowFilter][4]. (The links are to the protobuf definition of those objects which contain the variables and extensive comments).
That said, the bigtable-hbase dependency adds quite a few transitive dependencies. I would use the bigtable-hbase SCAN_ADAPTER in a standalone project, and then print the RowFilter to see how it's constructed.
In the specific case that you mention, the RowFilter is quite simple, but there may be additional complications. You have three parts to your scan, so I'll give a breakdown of how to achieve them:
scan.setRowPrefixFilter("helloworld".getBytes). This translates to a start key and end key on BigtableIO. "helloworld" is the start key, and you can calculate the end key with RowKeyUtil. calculateTheClosestNextRowKeyForPrefix. The default BigtableIO does not expose set start key and set end key, so the scio version will have to change to make those setters public.
scan.addColumn("family".getBytes, "qualifier".getBytes) translates to two RowFilters added to a RowFilter with a Chain (mostly analogous to an AND). The first RowFilter will have familyNameRegexFilter set, and the second RowFilter will have columnNameRegexFilter
scan.setMaxVersions() converts to a RowFilter with cellsPerColumnLimitFilter set. It would need to be added to a the chain from #2. Warning: If you use a timestampRangeFilter or value filter of a RowFilter to limit the range of the columns, make sure to put the cellsPerColumnLimitFilter at the end of the chain.
I'm currently doing a paper on PostgreSQL and I can't find anywhere (including their documentation) which is the intermediate language used for query processing and optimization.
There is no intermediate "language" as such.
The SQL is parsed into a parse tree of Node*. This is then passed through the query rewriter, then transformed into a plan tree by the planner/optimizer. You can view these trees using the (documented) options debug_print_parse, debug_print_rewritten and debug_print_plan.
See the source code - src/backend/parser/, src/backend/rewrite and src/backend/optimizer/ in particular, along with src/include/nodes/nodes.h, plannodes.h, parsenodes.h, etc. Note that there are README files in both the optimizer and parser source directories.