what possible ways to include external tables in TYPO3 - typo3

Since TYPO3 uses doctrine it is possible to use tables from multiple databases in one instance (with some restrictions like no joins).
But what is possible at all?
At the moment I need two external tables for an extension and instead of using them directly I import them to work locally as usual. But the importing has some draw backs.
Draw backs I can accept:
the data is not live (changes to the external tables are imported later)
the data is read only (changes are done externally anyway)
For importing I use ext:external_import but there are some problems as not all data can be imported in a single run, and then there are errors (e.g. there are reports about duplicate keys, alas there are no duplicate keys in the external tables)
On the other hand I doubt I can use the external tables directly as they have not the usual TYPO3 structure (fields: 'uid', 'pid', 'tstamp', ...). (Maybe they can be mapped in a view?) (of course in the tables I import the data into these fields exist)
Also external changes may be unnoticed and cached content does not reflect current data. In my case that would be a minor problem, as we currently already have no 'live' data, but this needs to be cleaned regularly for cache and for the search index (solr).
What are possible solutions? ? (do they depend on the TYPO3 version?)
What are your experiences?
EDIT:
While trying to realize it considering the given answers more doubts appear:
the tables are readonly (as they are changed from outside):
How do I declare it to TYPO3?
the tables does not follow the usual name rules, especially one table is named sys_category which in this way conflicts with the TYPO3 table sys_category.
Can I build a mapping inside of TYPO3?
Can I build a view from TYPO3 for renaming tables and fields?
like:
CREATE View tx_myext_category
SELECT id as uid, name as title, ...
FROM databasename.sys_category;

Yes, you can fetch data directly from other databases/tables. Of course it highly depends on the usecases and the data you get:
It works fine to read/write data by using the queryBuilder and all the APIs you know from https://docs.typo3.org/m/typo3/reference-coreapi/master/en-us/ApiOverview/Database/Index.html like ConnectionPool, QueryBuilder
If you want to show the data in the formengine, e.g. list module, you will need to have the minimum columns like uid, pid and a valid TCA as well.

From my experience, the mapping mechanism only works if the external table has a almost similar structure as TYPO3 tables. You need at least a uid field on the external side. This cannot be mapped! A missing pid field could be managed with on the TYPO3 side, also crdate or tstamp if needed. Just fill the local data array with the values TYPO3 needs.
Problems arise if you have relations to deal with. Many external systems have other ways to handle relations. You could run into many problems if you try to rely only on the mapping mechanism.
Other problems are fields with date format. Most external tables in the MS world use another format as the unixtime.
If you run into problems with the mapping mechanism you can switch to the TYPO3 queryBuilder. This is a powerful fallback. I experienced problems only with a special type of JOIN statements.
But with the TYPO3 queryBuilder you are on your own. You place instances of the queryBuilder code in the repository and add your model code as usual: thus you can continue to work with Fluid in the frontend as you are used to.
ANSWER TO EDIT:
With the TYPO3 queryBuilder readonly tables aren't a problem. Just don't implement the setter classes in your models.
With TYPO3 queryBuilder you can call any external table with any name. You have full control over the output data in your repository because the mapping is handled inside of it.
As far as I know, there is no way to create SQL views in TYPO3 up to v9, neither with the DBAL mapping mechanism nor with the TYPO3. queryBuilder.

Related

Is there Database diagram of Typo3?

I have to migrate my custom system to typo3 version 6.2 but I did not find any database diagram(design/schema) of typo3. Would be great if someone can help. Thank you.
In a TYPO3 installation without extensions, there are not many tables that will interest you. Interesting tables are
pages and pages_language_overlay: Pages and their translations.
tt_content: Content, translations are in the same table.
sys_language: Define available languages. For pages with only a default language it can stay empty.
sys_domain: Define Domains under which a site is available. For single domain installations it can be left empty.
sys_file_reference: References to files, which are stored in table sys_files. The table sys_files will be automatically filled when you put files into the fileadmin folder.
fe_users and fe_groups: Frontend users and groups.
The other tables are either caching tables (names start with cf_) or are mostly used internally (most tables having names starting with sys_).
In some of the tables, especially in pages and tt_content, there are deprecated fields.
The usage of some fields depends on the actual TYPO3 setup. For example, the text of a content element from tt_content could be stored in the field bodytext, or in the field pi_flexform. So the question how to import your data depends on your setup.
For further information have a look at the official docs: https://docs.typo3.org/typo3cms/CoreApiReference/latest/ApiOverview/Database/DatabaseStructure/Index.html
I don't have a diagram either, but this note is too long for a comment:
Most M:N relationships (e.g. users & groups) are handled with foreign key attributes stored as comma-separated entries inside table cells, inside rows of the parent table. Of course, this nonatomic way of storing data violates first-normal form of database-table design.
However, some newer extensions do rely on bridge tables. In typo3, this is called an M:M relation, and some tables have a naming corresponding convention, e.g. tt_news_cat_mm, linking news-messages with news-categories.

How can I create an EF Code First data model for AdventureWorks2014?

The fact that EF does not handle the hierarchyid MSSQL data type is widely known. The problem using the AdventureWorks sample database to generate an EF data model caused by a table using hierarchyid is also quite well known and widely reported. A good proposed workaround, for read only scenarios, is to use a view in place of the 'offending' Production.Document table, and cast that column to a more acceptable type, like 'nvarchar'.
However, it seems the only avenue open for creating a Code First data model is to brutally remove the table altogether, as the EF Power Tools code generator skips the pleasantries allowing a user to select which tables to include in the model. So here, even using a view won't work, because the view depends on its base table.
Does anyone know a workaround for this that doesn't involve modifying the T4 templates by hard-coding to skip this table?
Yes, install VS 2013 Update 4, which include the latest EF Tools, and use the Code First Model from Database feature, it allows you to select which tables to generate.

Generating SQL Server Views from EDMX using T4 templates

I'm working with a legacy database that I can't easily create an entity model over because it uses extension tables with what is effectively composite keys and EF only supports single column keys for mapping one entity to multiple tables.
So, what I've decided to do is create updatable views (with INSTEAD OF triggers to handle CRUD operations) over the top of the legacy tables (which cannot be touched) and then have my entity model (either using EF or DevExpress XPO) built on top of the database views. This will also allow me to do stuff like easily add sub-queries in the select clause to retrieve child counts on parent records when retrieving a list of parent records in a single query.
However, I don't particularly want to manually write the SQL for all the views and triggers so I thought I'd use data model defined in the .EDMX file and t4 templates to help me generate the bulk of the T-SQL needed to create the views and the triggers. I thought there would be some template that I could use as the basis for doing this, but seems that's not so easy to find.
Can someone please suggest a t4 template that I could use as the basis where mappings are being retrieved from the .EDMX. Alternatively can anyone advise how to use the StorageMappingItemCollection to retrieve the mapping information from the EDMX file. I know a few people have said that apparently you can't use it or that they just use Linq to Xml, but I would have thought it should certainly be possible to use the StorageMappingItemCollection class as a strongly typed class to access this data.
Any examples of how I could use StorageMappingItemCollection to access mapping info would be very helpful. Thanks.
See http://brewdawg.github.io/Tiraggo.Edmx/ you can install it via NuGet within Visual Studio and it serves up all of the metadata from your EDMX files that Microsoft hides from you, very simple, works great.

TYPO3 - How to retrieve/update own table in extension?

I would like to create an extension like tt_news , so i need to connect with my own tables .
So how to write data fetching and insertion to the custom table
In general, creating the extension from the scratch doesn't make a sense especially when you're learning.
Old school
For 'old school' extension it's the best way to install Extension Kickstarter.
It will help you to create tables, all required structure etc. Also will allow you to extend existing tables (ie. you can use it to modify tt_news tables and add custom filds without touching the tt_news' sources)
You should choose this way especially when you want to impact with some well-known extension written the same way.
All methods for working with DB can be found in the API
MVC - Extbase
If you're more familiar with MVC it would be better to use Extension Builder
It's the funnier way and allows to create extension faster, however it's less documented and more abstract.
It has also built-in modeler for creating your DB tables (Models) and creates set of default actions for the listing, displaying, modifying and removing records from your table. (with the bit of experience simplified version of the tt_news ca be created in few hours)
I generally prefer extbase and fluid for my new exts, esspecialy as it's some kind of preparation to work in the future with flow3, but you need consider which points are more important to you.

Entity Framework model-first design not won't let you edit the table mappings?

If we've been using an Entity Framework 4 model for some time, and we eventually want to switch the underlying database to a different vendor's product (say, from SQL Server to MySQL), is it simple to adjust the table and column mappings in the entity model without needing to update any of the entity class code?
We're trying to design code that is as database agnostic as possible, so I'd like to know in advance how much trouble we're in for if we ever switch our databases around. Ideally, we'd like to not have to touch our applications that use our entity classes. I can't seem to find any way in the entity designer or XML editor to adjust the underlying database column names without it giving me an error.
(I can, however, edit the entity's property names in the designer while leaving the database column names alone, but that's the opposite of what I need.)
Thanks!
EDMX is not database agnostic. SSDL part of EDMX is tightly coupled with database server (in case of MSSQL even with its version). You need separate SSDL for each supported database server.
I don't understand how changing column names relates to database agnostic model. Reverse is true! If you need your database to have different column names for different server products you need separate mapping for each of them!
Changing column names when using model first is possible only if you modify T4 template used for generating database creation SQL script. But every time you create that script designer will delete whole your storage description (SSDL) and mapping (MSL) and replace them with a new one.
The easiest way to have database agnostic code is using code first but even then you can have problems with some type and feature inconsistency among servers.
If you want database agnostic ORM you should probably check NHibernate.