I am implementing a java client for postgres and added postgis support.
I am trying to figure out the wire protocol types that are returned for geometric data types.
It looks like different postgis versions has different type number returned.
In some version it was 18011 and another has 17995 for line string. You can see related test I did here:
// previous
PostgreSQLColumnDecoderRegistry.Instance.registerDecoder(18011, JtsColumnDecoder())
// current
PostgreSQLColumnDecoderRegistry.Instance.registerDecoder(17995, JtsColumnDecoder())
IIUC the types can be seen on the db itself here on pg_type table: https://www.postgresql.org/docs/current/catalog-pg-type.html
Do you know where I can find docs / code about the type numbering rules?
The object IDs of the PostGIS data types are dynamically allocated when you create the extension. Consequently, they will normally be different in different PostgreSQL databases. One of the simplest queries to find the object ID of geometry is
SELECT 'geometry'::regtype::oid;
Related
Since TYPO3 uses doctrine it is possible to use tables from multiple databases in one instance (with some restrictions like no joins).
But what is possible at all?
At the moment I need two external tables for an extension and instead of using them directly I import them to work locally as usual. But the importing has some draw backs.
Draw backs I can accept:
the data is not live (changes to the external tables are imported later)
the data is read only (changes are done externally anyway)
For importing I use ext:external_import but there are some problems as not all data can be imported in a single run, and then there are errors (e.g. there are reports about duplicate keys, alas there are no duplicate keys in the external tables)
On the other hand I doubt I can use the external tables directly as they have not the usual TYPO3 structure (fields: 'uid', 'pid', 'tstamp', ...). (Maybe they can be mapped in a view?) (of course in the tables I import the data into these fields exist)
Also external changes may be unnoticed and cached content does not reflect current data. In my case that would be a minor problem, as we currently already have no 'live' data, but this needs to be cleaned regularly for cache and for the search index (solr).
What are possible solutions? ? (do they depend on the TYPO3 version?)
What are your experiences?
EDIT:
While trying to realize it considering the given answers more doubts appear:
the tables are readonly (as they are changed from outside):
How do I declare it to TYPO3?
the tables does not follow the usual name rules, especially one table is named sys_category which in this way conflicts with the TYPO3 table sys_category.
Can I build a mapping inside of TYPO3?
Can I build a view from TYPO3 for renaming tables and fields?
like:
CREATE View tx_myext_category
SELECT id as uid, name as title, ...
FROM databasename.sys_category;
Yes, you can fetch data directly from other databases/tables. Of course it highly depends on the usecases and the data you get:
It works fine to read/write data by using the queryBuilder and all the APIs you know from https://docs.typo3.org/m/typo3/reference-coreapi/master/en-us/ApiOverview/Database/Index.html like ConnectionPool, QueryBuilder
If you want to show the data in the formengine, e.g. list module, you will need to have the minimum columns like uid, pid and a valid TCA as well.
From my experience, the mapping mechanism only works if the external table has a almost similar structure as TYPO3 tables. You need at least a uid field on the external side. This cannot be mapped! A missing pid field could be managed with on the TYPO3 side, also crdate or tstamp if needed. Just fill the local data array with the values TYPO3 needs.
Problems arise if you have relations to deal with. Many external systems have other ways to handle relations. You could run into many problems if you try to rely only on the mapping mechanism.
Other problems are fields with date format. Most external tables in the MS world use another format as the unixtime.
If you run into problems with the mapping mechanism you can switch to the TYPO3 queryBuilder. This is a powerful fallback. I experienced problems only with a special type of JOIN statements.
But with the TYPO3 queryBuilder you are on your own. You place instances of the queryBuilder code in the repository and add your model code as usual: thus you can continue to work with Fluid in the frontend as you are used to.
ANSWER TO EDIT:
With the TYPO3 queryBuilder readonly tables aren't a problem. Just don't implement the setter classes in your models.
With TYPO3 queryBuilder you can call any external table with any name. You have full control over the output data in your repository because the mapping is handled inside of it.
As far as I know, there is no way to create SQL views in TYPO3 up to v9, neither with the DBAL mapping mechanism nor with the TYPO3. queryBuilder.
I'm coming from Neo4j and evaluating OrientDB, and I had a quick question about classes - do they have to exist prior to inserting a record? That is, in Neo4j there's the 'MERGE' command, which will update or create a node if it doesn't exist. Classes seem roughly equivalent to Neo4j's labels, and if a label doesn't exist when performing a MERGE, it will be created. Is there similar functionality in OrientDB? Currently when I try to insert a record whose class doesn't exist, OrientDB throws an exception, "Class SOME_CLASS not found in database".
I've been reading through the docs trying to get a handle on the various data models available, but I can't find anything explicit on this issue. One thing I tried was to add a cluster, and then insert a record with a class that does not exist. This worked, and in OrientDB Studio, under 'DB', I see the cluster with number of records equal to '1'; however, the class of that new record does not appear under 'Schema'.
If dynamic class creation of this sort isn't possible, is it acceptable to check if a class exists in the schema, and if not, create it, and then proceed with creating the record? A different question is, if it's acceptable, is it good to do this, or should I always define the schema manually?
If you use one of the CREATE commands, then the object is placed in a default class; for example:
CREATE VERTEX
Created vertex 'V#9:0 v1' in 0.047000 sec(s).
(In this case, the class is V.)
And of course if you use the INSERT INTO ... form, then you have to specify a class.
So perhaps your first question boils down to whether it is possible to change the class of an OrientDB vertex or edge.
It is possible to change the parent class of a vertex -- see
http://orientdb.com/docs/2.1/SQL-Move-Vertex.html -- but there are important caveats.
To test whether a class exists programmatically, see e.g. this SO entry:
Check if class exists or not in orientdb
This gives a Java example, but a similar approach is possible using other supported languages.
As to the wisdom of changing the class of an entity dynamically -- perhaps the safe answer is that if you can achieve whatever you want using a property label, then use labels.
We would like to create an OData REST API. Our data model is such that each customer has their own database. All database objects have the same definition across all customer databases, with the exception of a single table.
The customer specific table we will call Contact. When a customer adds a column the system creates a column with a standardised name with a definition translated from options selected by the user in the UI. The user only refers to the column data by a field name they have specified to enable the user to be able to generate friendly queries.
It seems to me that the following approaches could be used to enable OData for the model described:
1) Create an OData open type to cater for the dynamic properties. This has the disadvantage of user requests for a customer not providing an indication of the dynamic properties that can be queried against. Even though they will be known for the user (via token authentication). Also, because dynamic properties are a dictionary, some data pivoting and inefficient query writing would be required. Not sure how to implement the IQueryable handling of query options for the dynamic properties to enable our own custom field querying.
2) Create a POCO class with e.g. 50 properties; CustomField1, CustomField2... Then somehow control which fields are exposed for use in OData calls. We would then include a separate API call to expose the custom field mapping. E.g. custom field friendly name of MobileNumber = CustomField12.
3) At runtime, check to see if column definitions of table changed since last check. If have, generate class specific to customer using CodeDom and register it with OData. Aiming for a unique URL for each customer. E.g. http://domain.name/{customer guid}/odata
I think the ideal for us is option 2. However, the fact the CustomField1 could be an underlying SQL data type of nvarchar, int, decimal, datetime, etc, there are added complications.
Has anyone a working example of how to achieve what has been described, satisfactorily?
Thanks in advance for any help.
Rik
We have run into a similar situation but with our entire dataset being unknown until runtime. Using the ODataConventionModelBuilder and EdmModel classes, you can add properties dynamically to the model at runtime.
I'm not sure whether you will have to manually add all of the properties for this object type even though only some of them are unknown or whether you can add your main object and then add your dynamic ones afterwards, but I guess either would be workable.
If you can get hold of which type of user it is on the server, you could then add only the properties that you are interested in (like option 3 but not having to CodeDom).
There is an example of this kind of untyped OData server in the OData samples here that should get you started: https://github.com/OData/ODataSamples/tree/master/WebApi/v4/ODataUntypedSample
The research we carried out actually posed Option 1 as the most suitable approach for some operations. i.e. Create an SQL view that unpivots the data in a table to a key/value pair of column name/column value for each column in the table. This was suitable for queries returning small datasets. This was far less effort than Option 3 and less confusing for the user than Option 2. The unpivot query converted the field values to nvarchar (string) values and thus meant that filtering in the UI by column value data types was not simple to achieve. (If we decide to implement this ability, I believe this can be achieved by creating a custom attribute that derives from EnablQueryAttribute, marking the controller action with it and manipulate the IQueryable before execution).
However, we wanted to expose a /Contacts/Export endpoint that when called would output the columns from a table with a fixed schema joined on a table with a client specific schema and output to a CSV file. All the while utilising the OData supported filter syntax. One of our customer databases has more than 12 million rows of data and is made up of approximately 30 columns.
To achieve this it looks like our best bet would have been to work with the Microsoft.OData.Core.UriParser.UriQueryExpressionParser class, unfortunately Microsoft in their wisdom have declared this as internal, as well as many of it's dependants.
Walking an abstract syntax tree built from OData supported query options and applying our own visitor to each node to build some dynamic Linq query/SQL seems like a possible solution.
For the time-being we will simply implement a cut-down set of supported $filter criteria without the support for grouping parenthesis.
I have an existing database with a set of stored procedures. I am redesigning the application layer without making changes to database objects. One of the difficulties I am faced with is there are many stored procedures which are similar to each other in that they query the same tables but return different combinations of columns.
I am neither able to return partially filled entities nor I could find a way to return anonymous types from stored procedures using Entity Framework 4.1 (and SQL Server 2008 R2). This is forcing me to define too many complex types one per stored procedure although underlying table structure is the same.
My questions are :
Please suggest solutions for minimising number of entities / complex types which can be implemented using EF and without making changes to database objects.
Also, is it possible to return partially filled entities? This would enable me to reuse entities. I am not planning to use object tracking features.
How can we return anonymous types from stored procedures outputs? This is also good enough for me since in most cases I am going to return the data to client in JSON format
Thanks
You won't be able to return anonymous types - those are typically restricted to the scope of the method they're defined inside of.
Returning partially filled types isn't possible per se - but you could do this:
in your database layer, call the stored procedure in question; it will return a specific complex type to match its "signature"
using something like AutoMapper, you could easily copy those returned fields in the complex type into an entity of your system, thus getting a "partially filled" entity
return that entity from your database layer to the application to be used
You won't be able to avoid having lots of complex types just for the sake of getting the return values from the stored procedures - but you'll keep those locked away inside your database layer, they won't "leak out" into your entire application.
Just use System.Data.DataTable like good old days.
why synonyms are used?,advantages of syNONYMS IN SQL?
They're just shorthand names for objects inside a database. For example, you can create a synonym called Products if you have a namespace'd table in a database called ProductionControl.Inventory.Products. They're also handy for controlling named access to other databases in stored procedures. If you have SPs that refer to tables in other databases, creating a synonym and using that instead gives you more control in case the target of the synonym ever changes. This is useful in scenarios where you have SPs that refer to a development database, but when you deploy to production the name is different. So you'd just update the synonym and you'd be OK.
From MSDN Understanding Synonyms
A synonym is a database object that
serves the following purposes:
Provides an alternative name for another database object, referred to
as the base object, that can exist on
a local or remote server.
Provides a layer of abstraction that protects a client application
from changes made to the name or
location of the base object.
In some enterprise systems, you may have to deal with remote objects over which you have no control. For example, a database that is maintained by another department or team.
Synonyms can help you decouple the name and location of the underlying object from your SQL code. That way you can code against a synonym table even if the table you want is moved to a new server/database or renamed.
For example, I could write a query like this:
insert into MyTable
(...)
select ...
from remoteServer.remoteDatabase.dbo.Employee
but then if the server, or database, schema, or table changes it would impact my code. Instead I can create a synonym for the remote server and use the synonym instead:
insert into MyTable
(...)
select ...
from EmployeeSynonym
If the underlying object changes location or name, I only need to update my synonym to point to the new object.
http://www.mssqltips.com/sqlservertip/1820/use-synonyms-to-abstract-the-location-of-sql-server-database-objects/