I'm building a PHP application and I'm using Doctrine 2.5 to map my objects into the database. My database choice is PostgreSQL 10.
Instead of using the default schema (public), I've tagged all my classes with #ORM\Table(schema="my-schema") so it uses this schema instead of the default one. This works as intended: all the tables are created on that schema.
But I've found that tables for ManyToMany relations are not being created in my schema; instead they're created on the default one
(I suppose this is occurring because they're not backed up with a class on my model).
Then my question is if it's possible to configure somewhere the default schema I want Doctrine to use (this would also allow me to get rid of all the #ORM\Table(schema="my-schema") annotations). Maybe that is a bug on Doctrine, I don't know. Any kind of workaround to prevent this also would be appreciated.
Related
Since TYPO3 uses doctrine it is possible to use tables from multiple databases in one instance (with some restrictions like no joins).
But what is possible at all?
At the moment I need two external tables for an extension and instead of using them directly I import them to work locally as usual. But the importing has some draw backs.
Draw backs I can accept:
the data is not live (changes to the external tables are imported later)
the data is read only (changes are done externally anyway)
For importing I use ext:external_import but there are some problems as not all data can be imported in a single run, and then there are errors (e.g. there are reports about duplicate keys, alas there are no duplicate keys in the external tables)
On the other hand I doubt I can use the external tables directly as they have not the usual TYPO3 structure (fields: 'uid', 'pid', 'tstamp', ...). (Maybe they can be mapped in a view?) (of course in the tables I import the data into these fields exist)
Also external changes may be unnoticed and cached content does not reflect current data. In my case that would be a minor problem, as we currently already have no 'live' data, but this needs to be cleaned regularly for cache and for the search index (solr).
What are possible solutions? ? (do they depend on the TYPO3 version?)
What are your experiences?
EDIT:
While trying to realize it considering the given answers more doubts appear:
the tables are readonly (as they are changed from outside):
How do I declare it to TYPO3?
the tables does not follow the usual name rules, especially one table is named sys_category which in this way conflicts with the TYPO3 table sys_category.
Can I build a mapping inside of TYPO3?
Can I build a view from TYPO3 for renaming tables and fields?
like:
CREATE View tx_myext_category
SELECT id as uid, name as title, ...
FROM databasename.sys_category;
Yes, you can fetch data directly from other databases/tables. Of course it highly depends on the usecases and the data you get:
It works fine to read/write data by using the queryBuilder and all the APIs you know from https://docs.typo3.org/m/typo3/reference-coreapi/master/en-us/ApiOverview/Database/Index.html like ConnectionPool, QueryBuilder
If you want to show the data in the formengine, e.g. list module, you will need to have the minimum columns like uid, pid and a valid TCA as well.
From my experience, the mapping mechanism only works if the external table has a almost similar structure as TYPO3 tables. You need at least a uid field on the external side. This cannot be mapped! A missing pid field could be managed with on the TYPO3 side, also crdate or tstamp if needed. Just fill the local data array with the values TYPO3 needs.
Problems arise if you have relations to deal with. Many external systems have other ways to handle relations. You could run into many problems if you try to rely only on the mapping mechanism.
Other problems are fields with date format. Most external tables in the MS world use another format as the unixtime.
If you run into problems with the mapping mechanism you can switch to the TYPO3 queryBuilder. This is a powerful fallback. I experienced problems only with a special type of JOIN statements.
But with the TYPO3 queryBuilder you are on your own. You place instances of the queryBuilder code in the repository and add your model code as usual: thus you can continue to work with Fluid in the frontend as you are used to.
ANSWER TO EDIT:
With the TYPO3 queryBuilder readonly tables aren't a problem. Just don't implement the setter classes in your models.
With TYPO3 queryBuilder you can call any external table with any name. You have full control over the output data in your repository because the mapping is handled inside of it.
As far as I know, there is no way to create SQL views in TYPO3 up to v9, neither with the DBAL mapping mechanism nor with the TYPO3. queryBuilder.
We are migrating our Software from our own and legacy PHP framework to Symfony.
Our application has different Schema for every user. So beside public Schema, we have about 2.000 user Schemas.
Let's say that we have a system similar to the marketplace, and every user can customize some tables inside their own Schema, but let's not focus on that.
We are currently searching for the best solutions for two problems:
When a new user is registered, our Symfony should run migrations only for that newly created user Schema.
When we create migration we need that that migration affects ALL existing user Schemas.
Currently, in our old system, we do these kinds of migration with a bunch of PHP logic, and I would like to avoid customizing DoctrineMigrationsBundle or have hundreds of lines of PHP code in a single migration.
So I would like to know is there any easier way to handle all user Schemas in Doctrine/Symfony, so when I run for example:
php bin/console doctrine:migrations:migrate
all our existing user Schemas are updated.
Our user Schema names are like app234234, app453453 and except those Schemas we have only default public Schema
We are using:
Symfony 4.3
PostgreSQL 9.1
PHP 7.3
1 - For every schema you have, you can define a schema filter. Let's say you prefix user table with schema name app234234_ . To prevent migrations from removing your other tables, you can use doctrine dbal schema_filter property in your config.yml/doctrine.yml or wherever you define new users's schema
## doctrine.yml
doctrine:
dbal:
connections:
app234234:
schema_filter: "/^app234234_/" ## more likely a %schema_name% parameter
According to symfony doc: This ignores the tables on the DBAL level and they will be ignored by the diff command.
Note that if you have multiple connections configured then the schema_filter configuration will need to be placed per-connection
2- Regarding your second point, i don't know if this is something you can/should handle with symfony.
I have been using Database First Entity Framework (EDMX) and SQL Server Data Tools Database Projects in combination very successfully - change the schema in the database and 'Update Model from Database' to get them into the EDMX. I see though that Entity Framework 7 will be dropping the EDMX format and I am looking for a new process that will allow me to use Code First in Combination with Database Projects.
Lots of my existing development and deployment processes rely on having a database project that contains the schema. This goes in source control is deployed along with the code and is used to update the production database complete with data migration using pre and post deployment scripts. I would be reluctant to drop it.
I would be keen to split one big EDMX into many smaller models as part of this work. This will mean multiple Code First models referencing the same database.
Assuming that I have an existing database and a database project to go with it - I am thinking that I would start by using the following wizard to create an initial set of entity and context classes - I would do this for each of the models.
Add | New Item... | Visual C# Items | Data | ADO.NET Entity Data Model | Code first from database
My problem is - where do I go from there? How do I handle schema changes? As long as I can get the database schema updated, I can use a schema compare operation to get the changes into the project.
These are the options that I am considering.
Make changes in the database and use the wizard from above to regenerate. I guess that I would need to keep any modifications to the entity and/or context classes in partial classes so that they do not get overwritten. Automating this with a list of tables etc to include would be handy. Powershell or T4 Templates maybe? SqlSharpener (suggested by Keith in comments) looks like it might help here. I would also look at disabling all but the checks for database existence and schema compatibility here, as suggested by Steve Green in the comments.
Make changes in code and use migrations to get these changes applied to the database. From what I understand, not having models map cleanly to database schemas (mine don't) might pose problems. I also see some complaints on the net that migrations do not cover all database object types - this was also my experience when I played around with Code First a while back - unique constraints I think were not covered. Has this improved in Entity Framework 7?
Make changes in the database and then use migrations as a kind of comparison between code and the database. See what the differences are and adjust the code to suit. Keep going until there are no differences.
Make changes manually in both code and the database. Obviously, this is not very appealing.
Which of these would be best? Is there anything that I would need to know before trying to implement it? Are there any other, better options?
So the path that we ended up taking was to create some T4 templates that generate both a DbContext and our entities. We provide the entity T4 a list of tables from which to generate entities and have a syntax to indicate that the entity based on one table should inherit from the entity based on another. Custom code goes in partial classes. So our solution looks most like my option 1 from above.
Also, we started out generating fluent configuration in OnModelCreating in the DbContext but have swapped to using attributes on the Entities (where attributes exist - HasPrecision was one that we had to use fluent configuration for). We found that it is more concise and easier to locate the configuration for a property when it is right there decorating that property.
We're using a very basic JPA implementation that should create tables consistently from our models.
I believe we're using EclipseLink or TopLink (whichever one is default with the latest Netbeans/Glassfish). The problem is, the tables are created with inconsistent capitaliztion and with the columns out of order. For me, It creates the "User" table as "user", and for other members of my team it creates "USER".
I've tried using the #Table annotation (#Table(name="USer")), but it doesn't work.
How do we get EclipseLink to generate consistent table names? Frankly this seems like a rather amateurish mistake for a framework like this.
Sub-question : the reason this is a problem is because EclipseLink by default has no default way of managing schema/data migrations, as far as I know of. The way we're handling it is by writing a bunch of INSERT INTO's to bootstrap the objects we need in our database, and drop-and-recreating the tables every time the schema changes. I know this is not the best practice for propagating schema changes -- does anyone know how this is typically handled in a standard JPA implementation?
Thanks.
By default EclipseLink uses all upper case for the table name, the class User would be USER.
If you specify an #Table annotation with name="USer", then the table will be created as "USer".
Perhaps you are using your own scripts to create the tables, or you database is changing the case based on the OS or its own settings. What database are you using?
If you enable logging in EclipseLink, it will show the exact DDL that it is executing (if it is executing DDL).
In EclipseLink 2.4 there is also a "create-or-extend-tables" DDL generation option to alter existing tables.
We never found any good answer for this. Luckily, we found a workaround for the ways we were using to update the table, which didn't care about capitalization.
I'm using Entity Framework 5 on ASP MVC 4 web site I'm developing.
Because I am using shared hosting which charge for the number of databases I use I would like to run a test site near my production site.
I have two problems:
1) I use Code First and Database Migration. The migration classes seem to embed the schema dbo inside the name of the tables.
How can I change the schema according to the test/production flag
2) How can I change the schema from which EF select data?
Thank you,
Ido.
Both migration and EF take schema from mapping so if you want to change the schema you must update your mapping to use:
modelBuilder.Entity<MyEntity>().ToTable("MyTable", "MySchema");
and control the value of MySchema from configuration but this is really bad idea. One day you forget to change the value and break your production. Use local database for development and test.
As already said: use identical databases (structurally) for development, test and production.
The goal of schemas is to group database objects, like we do with namespaces in e.g. C#, or to simplify permissions for groups of database objects. Not to identify database stages. By using them for the latter you also make it much harder, if not impossible, to use schema appropriately. See for instance this MSDN white paper.
It is much easier to use some database name conventions to indicate their purpose.