I am currently evaluating jooq. I am very new to it. I want to use jooq to access a SQL Server 2008 R2 database. When I generate the classes, the codegen tool goes and generates code for objects from all the databases on that server which is not what I want.
What settings do I use in the configuration file to restrict the tool to only one database? I checked the documentation and is not so obvious. Well, for restricting tables I guess I could use the <includes></includes> and <excludes></excludes>.
Could I use the same tags to restrict the tool to objects only in certain databases by using fully qualified table names with three parts [database].[schema].[table]?
Some other comments:
It would be nice if the tool allowed specifying inclusion/exclusion rules per object type i.e. table, view, procedures , functions etc.
place classes for tables/views/procedures in separate packages, if possible, instead of lumping them all together.
One of databases had a schema <Domain>\<user name> and the code generated was invalid. Just make sure \ are generated to \\ in strings.
Thanks
Answers for your current questions / comments:
Could I use the same tags to restrict the tool to objects only in certain databases by using fully qualified table names with three parts [database].[schema].[table]?
Yes, the <includes/> and <excludes/> elements match both qualified and unqualified names, so you can for instance exclude database.schema.table (leave out the brackets).
It would be nice if the tool allowed specifying inclusion/exclusion rules per object type i.e. table, view, procedures , functions etc.
Indeed, that's on the roadmap: #5263
place classes for tables/views/procedures in separate packages, if possible, instead of lumping them all together.
You can override the current behaviour with a generator strategy:
Programmatic implementation (more powerful)
Declarative implementation (more simple)
One of databases had a schema \ and the code generated was invalid. Just make sure \ are generated to \ in strings.
Yes, the <includes/> and <excludes/> elements (as many other elements in the code generator configuration) take Java regular expressions with all the associated semantics.
A better solution to your actual problem:
In jOOQ 3.9, you can use the catalog / schema mapping feature in your code generator. There are different ways to configure it:
Only one input catalog (standard name for what SQL Server calls database)
This is the simplest configuration to get you started. It will generate everything inside of a single database:
<configuration>
<generator>
<database>
<inputCatalog>database</inputCatalog>
...
You can further reduce the generated output to only a single schema inside that database as such:
<configuration>
<generator>
<database>
<inputCatalog>database</inputCatalog>
<inputSchema>schema</inputSchema>
...
Multiple input catalogs
In more sophisticated setups (or as your project grows), a more appropriate approach would be to list all catalogs and schemas explicitly:
<configuration>
<generator>
<database>
<catalogs>
<!-- This configuration generates everything inside of that catalog -->
<catalog>
<inputCatalog>database1</inputCatalog>
</catalog>
<!-- This configuration generates only some schemas inside of the catalog -->
<catalog>
<inputCatalog>database2</inputCatalog>
<schemata>
<schema>
<inputSchema>schema1</inputSchema>
</schema>
<schema>
<inputSchema>schema2</inputSchema>
</schema>
</schemata>
</catalog>
...
Background:
By default, the jOOQ code generator always generates everything it can see. There are two ways to explicitly restrict this:
The code generator's catalog / schema mapping feature
The standard <includes/> and <excludes/> regular expressions (which apply to all objects)
With jOOQ 3.8, code generation support for multiple catalogs (in SQL Server: databases) was added, without adding support for restricting them through the catalog mapping feature. This was rectified in jOOQ 3.9, when catalog mapping was added (#4794).
Prior to #4794, the only workaround was to use <excludes/> to exclude all content from the unwanted catalogs, which resulted in the jOOQ 3.8 code generator to produce only empty catalogs.
See also this discussion on the jOOQ User Group
Related
Since TYPO3 uses doctrine it is possible to use tables from multiple databases in one instance (with some restrictions like no joins).
But what is possible at all?
At the moment I need two external tables for an extension and instead of using them directly I import them to work locally as usual. But the importing has some draw backs.
Draw backs I can accept:
the data is not live (changes to the external tables are imported later)
the data is read only (changes are done externally anyway)
For importing I use ext:external_import but there are some problems as not all data can be imported in a single run, and then there are errors (e.g. there are reports about duplicate keys, alas there are no duplicate keys in the external tables)
On the other hand I doubt I can use the external tables directly as they have not the usual TYPO3 structure (fields: 'uid', 'pid', 'tstamp', ...). (Maybe they can be mapped in a view?) (of course in the tables I import the data into these fields exist)
Also external changes may be unnoticed and cached content does not reflect current data. In my case that would be a minor problem, as we currently already have no 'live' data, but this needs to be cleaned regularly for cache and for the search index (solr).
What are possible solutions? ? (do they depend on the TYPO3 version?)
What are your experiences?
EDIT:
While trying to realize it considering the given answers more doubts appear:
the tables are readonly (as they are changed from outside):
How do I declare it to TYPO3?
the tables does not follow the usual name rules, especially one table is named sys_category which in this way conflicts with the TYPO3 table sys_category.
Can I build a mapping inside of TYPO3?
Can I build a view from TYPO3 for renaming tables and fields?
like:
CREATE View tx_myext_category
SELECT id as uid, name as title, ...
FROM databasename.sys_category;
Yes, you can fetch data directly from other databases/tables. Of course it highly depends on the usecases and the data you get:
It works fine to read/write data by using the queryBuilder and all the APIs you know from https://docs.typo3.org/m/typo3/reference-coreapi/master/en-us/ApiOverview/Database/Index.html like ConnectionPool, QueryBuilder
If you want to show the data in the formengine, e.g. list module, you will need to have the minimum columns like uid, pid and a valid TCA as well.
From my experience, the mapping mechanism only works if the external table has a almost similar structure as TYPO3 tables. You need at least a uid field on the external side. This cannot be mapped! A missing pid field could be managed with on the TYPO3 side, also crdate or tstamp if needed. Just fill the local data array with the values TYPO3 needs.
Problems arise if you have relations to deal with. Many external systems have other ways to handle relations. You could run into many problems if you try to rely only on the mapping mechanism.
Other problems are fields with date format. Most external tables in the MS world use another format as the unixtime.
If you run into problems with the mapping mechanism you can switch to the TYPO3 queryBuilder. This is a powerful fallback. I experienced problems only with a special type of JOIN statements.
But with the TYPO3 queryBuilder you are on your own. You place instances of the queryBuilder code in the repository and add your model code as usual: thus you can continue to work with Fluid in the frontend as you are used to.
ANSWER TO EDIT:
With the TYPO3 queryBuilder readonly tables aren't a problem. Just don't implement the setter classes in your models.
With TYPO3 queryBuilder you can call any external table with any name. You have full control over the output data in your repository because the mapping is handled inside of it.
As far as I know, there is no way to create SQL views in TYPO3 up to v9, neither with the DBAL mapping mechanism nor with the TYPO3. queryBuilder.
I'm working with a legacy database that I can't easily create an entity model over because it uses extension tables with what is effectively composite keys and EF only supports single column keys for mapping one entity to multiple tables.
So, what I've decided to do is create updatable views (with INSTEAD OF triggers to handle CRUD operations) over the top of the legacy tables (which cannot be touched) and then have my entity model (either using EF or DevExpress XPO) built on top of the database views. This will also allow me to do stuff like easily add sub-queries in the select clause to retrieve child counts on parent records when retrieving a list of parent records in a single query.
However, I don't particularly want to manually write the SQL for all the views and triggers so I thought I'd use data model defined in the .EDMX file and t4 templates to help me generate the bulk of the T-SQL needed to create the views and the triggers. I thought there would be some template that I could use as the basis for doing this, but seems that's not so easy to find.
Can someone please suggest a t4 template that I could use as the basis where mappings are being retrieved from the .EDMX. Alternatively can anyone advise how to use the StorageMappingItemCollection to retrieve the mapping information from the EDMX file. I know a few people have said that apparently you can't use it or that they just use Linq to Xml, but I would have thought it should certainly be possible to use the StorageMappingItemCollection class as a strongly typed class to access this data.
Any examples of how I could use StorageMappingItemCollection to access mapping info would be very helpful. Thanks.
See http://brewdawg.github.io/Tiraggo.Edmx/ you can install it via NuGet within Visual Studio and it serves up all of the metadata from your EDMX files that Microsoft hides from you, very simple, works great.
I'm using Entity Framework 5 on ASP MVC 4 web site I'm developing.
Because I am using shared hosting which charge for the number of databases I use I would like to run a test site near my production site.
I have two problems:
1) I use Code First and Database Migration. The migration classes seem to embed the schema dbo inside the name of the tables.
How can I change the schema according to the test/production flag
2) How can I change the schema from which EF select data?
Thank you,
Ido.
Both migration and EF take schema from mapping so if you want to change the schema you must update your mapping to use:
modelBuilder.Entity<MyEntity>().ToTable("MyTable", "MySchema");
and control the value of MySchema from configuration but this is really bad idea. One day you forget to change the value and break your production. Use local database for development and test.
As already said: use identical databases (structurally) for development, test and production.
The goal of schemas is to group database objects, like we do with namespaces in e.g. C#, or to simplify permissions for groups of database objects. Not to identify database stages. By using them for the latter you also make it much harder, if not impossible, to use schema appropriately. See for instance this MSDN white paper.
It is much easier to use some database name conventions to indicate their purpose.
Is there a way to compose queries from 2 different entity models if the models are hitting the same underlying database.
The scenario I have is this:
I have a framework that uses EF for data access.(EDM 1)
I have a client application that uses services of the framework and also uses EF for it's own data access.(EDM2)
There are situations where I need to compose queries and join on entities that span the 2 EDMs.
Is there a way to do this without getting the data in memory from the first EDM and then apply additional predicates/joins in memory from the entities of the 2nd EDM?
I hope I'm articulating this the right way
EDIT
#Ladislav Mrnka:
The first EDM is the data access layer for a reusable framework.
It doesn't make sense to couple the EF generated entities from this EDM with
those of the consuming client
It defeats reusabilty of the API if I did this and I'd have to carry around additonal bloat
(EF metadata and DB tables of the client) everytime I wanted to redeploy the framework.
Also this would make managing the model in the designer unwieldy.
I'm currently using what you mention n item 7 as the solutuon and the performance is abysmal
due to the fact that I have to end up returning more data(i.e. entities) than needed from framework using
EDM1 and then filter out the ones not needed based on predicates/conditions based on value of
properties from entities in the second EDM. End result is a huge performance degradation and an unhappy DBA.
For this reason I ended up pushing the logic needed to retrieve the entities
to a SPROC in which I can access the tables that both EDMs use and apply the predicates needed
and have the entire query run in the DB as opposed to bringing the data in memory
and then filter out unnecessary ones.Downside is that I can't use LINQ
Item 8 that you mention sounds interesting but from what it sounds like I doubt that
you get strong typing at design time, or do you?
Can you upload your code sample someplace so that I can try it out?
Important edit
There is no build in support for achieving this with two ObjectContext types. Your query must always be executed against single ObjectContext.
Probably the best way to go: This was interesting enough for me to try it myself. I started with very simple idea. Two EDMX files (used with POCO T4 generators), each containing single entity. I take metadata description from second connection string and added it to first connection string. I used ObjectContext and ObjectSet directly. By doing this I was able to query and modify both entities from single ObjectContext instance. I also tryed to create query joining entities from both models and it worked.
This obviously works only if both EDMX map to the same database (same db connection string).
The important part is connections string:
<configuration>
<connectionStrings>
<add name="TestEntities" connectionString="metadata=res://*/FirstModel.csdl|res://*/FirstModel.ssdl|res://*/FirstModel.msl|res://*/SecondModel.csdl|res://*/SecondModel.ssdl|res://*/SecondModel.msl;provider=System.Data.SqlClient;provider connection string="Data Source=.;Initial Catalog=Test;Integrated Security=True;MultipleActiveResultSets=True"" providerName="System.Data.EntityClient" />
</connectionStrings>
</configuration>
This connection string contains metadata from two models - FirstModel.edmx and SecondModel.edmx.
Another problem is to force EF to use mapping from both these files. Each EDMX file must define unique container for SSDL and CSDL. ObjectContext offers property called DefaultContainerName. This property can be set directly or through some constructor overloads. Once you set this property you bind your ObjectContext instance to single EDMX - for this scenario you must not set this property. Omitting DefaultContainerName can have some consequences because some features and declarations can stop working (you will get runtime errors). You should not have problems with POCO unless you want to use some advanced features. You will most probably have proplems if you are using Entity objects (heavy EF entities. All methods using entity sets defined as strings are dependent on container. Due to this I suggest using such configuration only when necessary - for cross models queries.
Last problem is generating entities and "strongly typed" derived ObjectContext. The way to go is modify T4 template so that one template reads data from multiple EDMX files and generates context end entities for all of them - I already doing this in my project and it works. Default T4 implementation doesn't follow needed approach described in previous paragraph. Derived ObjectContext from default T4 implementation is dependent on single EDMX and entity container.
This part has been written before previous edit.
I'm leaving the rest of information just because some of them can be useful in other scenarios including work with multiple databases.
ORM like entity framework operates on top of mapping between object world and database world. In EF the object world is described by CSDL, database world is described as SSDL and mapping between them is described as MSL (all are just XML with well known schema). At design time these descriptions are part of model stored in EDMX file. During compilation these descriptions are extracted from EDMX and by default included as resource files to compiled assembly.
When you create instance of ObjectContext it receives connections string which contains reference to CSDL, SSDL and MSL resource files. SSDL or MSL do not specify include element to add information from other files. CSDL offers Using element which will allow you reusing existing mapping but this feature is not supported by designer. ConnectionString is used to initialize EntityConnection instance which is in turn used to initialize ObjectContext's MetadataWorkspace (runtime mapping information). Also ObjectContext doesn't provide any functionality of nesting multiple contexts into hiearchy. Connection string can't contain reference to multiple instances of these files. Edit: It can. I just tested it. See the initial paragraphs.
When you run Linq or ESQL query on the instance of ObjectContext it usese MSL to map your entities or POCO classes (defined by CSDL) into DB query (defined by SSDL description of database tables). If it doesn't have this information it will not work (and it can't have that information if it is stored in separate EDMX).
So how to solve this problem? There are several ways:
Always consider: Merge your mapping into one file (if multiple files are used for same database). That is supposed way to use EF and as you mentioned you are querying same DB so two EF models are not needed.
Duplicate entity description in second model. If you use EF4 and POCO you can map same descriptions from multiple models into one POCO class definition. I don't like this solution but sometimes it can help.
Define DB View or Stored procedure containing your query (or core of your query) and map it in one model to new entity.
Use DefiningQuery in one model (you will probably need 3rd one if you use Update from database feature) and map it to new entity. DefiningQuery is custom SQL query defined in SSDL instead of table or view description.
Use Function with custom CommandText specifying DB query. It is similar to using DefiningQuery and it has the same limitation. You must manually (in EDMX) map the result of the function into new complex type (another difference to DefiningQuery which is mapped to new entity).
Define new type for result of the query (properties of the type must have same names as returned columns in query) and use ObjectContext's ExecuteStoreQuery (only in EF4).
Divide query into two parts each executed separately on its own context and use linq-to-objects to get result. I don't like this solution.
This one is only high level idea - I didn't try it and I don't know if it works. As described above runtime mapping is dependent on the content of MetadataWorkspace instance which is filled from EntityConnection. EntityConnection also provides constructor which receives MetadataWorkspace instance directly. So generally if it would be possible to fill MetadataWorkspace from multiple EDMX you will not need multiple ObjectContext instances but your mapping would be still separated into two EDMXs. This would hopefully allow you writing custom Linq queries on top of two mapping files). Edit: It should be possible because it is exactly what EF is doing if you define multiple mappings in connection string.
Use CSDL Using feature for breaking the model into multiple reused parts.
I have classes for entities like Customer, InternalCustomer, ExternalCustomer (with the appropriate inheritance) generated from an xml schema. I would like to use JPA (suggest specific implementation in your answer if relevant) to persist objects from these classes but I can't annotate them since they are generated and when I change the schema and regenerate, the annotations will be wiped. Can this be done without using annotations or even a persistence.xml file?
Also is there a tool in which I can provide the classes (or schema) as input and have it give me the SQL statements to create the DB (or even create it for me?). It would seem like that since I have a schema all the information it needs about creating the DB should be in there. I am not talking about creating indexes, or any tuning of the db but just creating the right tables etc.
thanks in advance
You can certainly use JDO in such a situation, dynamically generating the classes, the metadata, any byte-code enhancement, and then runtime persistence, making use of the class loader where your classes have been generated in and enhanced. As per
http://www.jpox.org/servlet/wiki/pages/viewpage.action?pageId=6619188
JPA doesn't have such a metadata API unfortunately.
--Andy (DataNucleus)