Trouble with Multi-Tenant Schema Generator Example - codefluent

We are attempting to use CFE to generate one schema for each tenant as outlined in the CodeFluent blog post (http://blog.codefluententities.com/2014/12/04/multi-tenant-using-multiple-schema/). In this scenario, we are expecting that each schema generated should be identical and we are using the ICodeFluentPersistence Hook system to identify the company for a user and then properly set the schema to be used. All of that works fine, but when we run the code to generate the multiple schemas (https://github.com/SoftFluent/CodeFluent-Entities/tree/master/Extensions/SoftFluent.MultiTenantGenerator), it is removing the constraints. I then tried to see if there was an issue with my configuration, but running the sample program from GitHub produces the same results. After running the sample program, the Primary key was not present in the contoso schema, even though is was properly defined in the dbo schema (and in the model).
Has anyone used the CFE Multi-Schema generator or have any insight into what the issue may be?

Thanks for your response, but I am not sure that I agree. The whole reason (at least of me) to use the Multi-Tenant generator is to create as many database schemas as needed (one per client) from a single CFE model. The idea that you would lose the constraints in all but one of them didn't feel right so I did a bit more investigation and found the following in "Microsoft SQL Server 2012 Internals" by Kalen Delaney and Craig Freeman (through Google Books):
And in fact was able to do a quick test to prove this out by creating two identical tables with identical PK names:
So it would appear to me that CFE should be able to create the two identical databases from the same model and seems to point to a deficiency in the SQLServer diff engine.

The multi-schema generator loads the model and change it dynamically to modify the schema of the entities. Then it call the standard code production process with only the database producers (SQL Server, Oracle, etc.).
So if you want to generate 2 differents schema (dbo and contoso) against an empty database, the process is the following:
Generate the database for the dbo schema from a blank database
Generate the database for the contoso schema from the previously generated database
Before creating a constraint, the SQL Server diff engine drops the constraint with the same name. In fact SQL Server does not allow 2 constraints to have the same name (I can't find a page on MSDN with more details about that). So in your case the existing PK is dropped when you generate the contoso schema because the name of the PK is the same as the one that exists in the dbo schema. Maybe this can be improved, but the diffs engine tries to generate a code that works for SQL Server 2000 to SQL Server 2016.
Workarounds
You can generate each schema in a different database, so the diffs engine will generate the code you expect. Then you can run the generated scripts on the production database. Not the easiest way but it should work.
You can use the patch producer to replace the name of the schema in the file. For SQL files you should use the SqlServerPatchProducer as explain in the KnowledgeBase:
namespace Sample
{
public class SqlServerPatchProducer : SqlServerProducer
{
public SqlServerPatchProducer()
{
}
protected override void RunProceduresScript()
{
string path = GetPath(Project.DefaultNamespace + "_procedures.sql");
ProduceFrom(path, "before");
SearchAndReplaceProducer.ProducePatches(Project, null, this, null, ProductionFlags, Element);
Utilities.RunFileScript(path, Database, OutputEncoding);
ProduceFrom(path, "after");
}
}
}

Related

TypeORM: Dynamically set database schema for EntityManager (or repositories) at runtime?

Situation:
For our SaaS API we use schema-based multitenancy, which means every customer (~tenant) has its own separate schema within the same (postgres) database, without interfering with other customers. Each schema consists of the same underlying entity-model.
Everytime a new customer is registered to the system, a new isolated schema is automatically created within the db. This means, the schema is created at runtime and not known in advance. The customer's schema is named according to the customer's domain.
For every request that arrives at our API, we extract the user's tenancy-affiliation from the JWT and determine which db-schema to use to perform the requested db-operations for this tenant.
Problem
After having established a connection to a (postgres) database via TypeORM (e.g. using createConnection), our only chance to set the schema for a db-operation is to resort to the createQueryBuilder:
const orders = await this.entityManager
.createQueryBuilder()
.select()
.from(`${tenantId}.orders`, 'order') // <--- setting schema-prefix here
.where("order.priority = 4")
.getMany();
This means, we are forced to use the QueryBuilder as it does not seem to be possible to set the schema when working with the EntityManager API (or the Repository API).
However, we want/need to use these APIs, because they are much simpler to write, require less code and are also less error-prone, since they do not rely on writing queries "manually" employing a string-based syntax.
Question
In case of TypeORM, is it possible to somehow set the db-schema when working with the EntityManager or repositories?
Something like this?
// set schema when instantiating manager
const manager = connection.createEntityManager({ schema: tenantDomain });
// should find all matching "order" entities within schema
const orders = manager.find(Order, { priority: 4 })
// should find a matching "item" entity within schema using same manager
const item = manager.findOne(Item, { id: 321 })
Notes:
The db-schema needs to be set in a request-scoped way to avoid setting the schema for other requests, which may belong to other customers. Setting the schema for the whole connection is not an option.
We are aware that one could create a whole new connection and set the schema for this connection, but we want to reuse the existing connection. So simply creating a new connection to set the schema is not an option.
To answer my own question:
At the moment there is no way to instantiate TypeORM repositories with different schemas at runtime without creating new connections.
So the only two options that a developer is left with for schema-based multi tenancy are:
Setting up new connections to connect with different schemas within the same db at runtime. E.g. see NestJS Request Scoped Multitenancy for Multiple Databases. However, one should definitely strive for reusing connections and and be aware of connection limits.
Abandoning the idea of working with the RepositoryApi and reverting to using createQueryBuilder (or executing SQL queries via query()).
For further research, here are some TypeORM GitHub issues that track the idea of changing the schema for a existing connections or repositories at runtime (similar to what is requested in the OP):
Multi-tenant architecture using schema. #4786 proposes something like this.photoRepository.useSchema('customer1').find()
Handling of database schemas #3067 proposes something like getConnection().changeDefaultSchema('myschema')
Run-time change of schema #4473
Add an ability to set postgresql schema per call #2439
P.S. If TypeORM decides to support the idea discussed in the OP, I will try to update this answer.
Here is a global overview of the issues with schema-based multitenancy along with a complete walkthrough a Github repo for it.
Most of the time, you may want to use Postgres Row Security Policy instead. It gives most of the benefits of schema-based multitenancy (especially on developer experience), without the issues related to the multiplication of connections.
Since commenting does not work for me, here a hint from the documentation of NestJS:
https://docs.nestjs.com/techniques/database#async-configuration
I am not using NestJS but reading the docs at the moment to decide, if it's a fitting framework for us. We have an app where only some modules have multi tenancy with schema per tenant, so using TypeOrmModule.forRootAsync(dynamicCreatedDbConfig) might be an option for me too.
This may help you if you have an interceptor or middleware, which prepares the dynamicCreatedDbConfig data before...

Oracle Data Modeler generated column name from datamodel too long

I'm trying to create a datamodel in the Oracle Data Modeler module that is available in Oracle SQL Developer. I would like to maintain my data dictionary from this model. To do this (I think) I need a generated DDL file for which the attributes are not longer than 30 characters.
I have just discovered Oracle SQL Developer and am completely new to creating these kind of models. What I have done so far. I have created a logical model and have engineered it to a relational model. From the relational model I can then generate DDL scripts that I can run on the database to make the changes I want.
When doing this I run into a problem. When engineering the logical model to a relational model I can see that the foreign keys I have made become more than 30 characters. This is because it seems to generate the name as (see picture)
From searching it seems you should be able to fix this with naming standard templates. I have looked for this menu option but can't find it. I have found the name abbreviations functionality for which you can upload .csv files but I think this is for something different.
Rightclicking on the logical model in the datamodeler browser view gives me the opportunity to Apply naming standards, but this gives me a message that I should turn off the keep as the name of the originating attribute option (see picture). I have looked for this but can't find this option.
My version of Oracle SQL Developer is 4.1.3.20, Build MAIN-20.78.
Please let me know if my story is not clear. Thanks.
Generated column name
Applying name standards message
You will find the option under Tools/Preferences/Data Modeler/Model/Logical

Change Schema of Entity Framework

I'm using Entity Framework 5 on ASP MVC 4 web site I'm developing.
Because I am using shared hosting which charge for the number of databases I use I would like to run a test site near my production site.
I have two problems:
1) I use Code First and Database Migration. The migration classes seem to embed the schema dbo inside the name of the tables.
How can I change the schema according to the test/production flag
2) How can I change the schema from which EF select data?
Thank you,
Ido.
Both migration and EF take schema from mapping so if you want to change the schema you must update your mapping to use:
modelBuilder.Entity<MyEntity>().ToTable("MyTable", "MySchema");
and control the value of MySchema from configuration but this is really bad idea. One day you forget to change the value and break your production. Use local database for development and test.
As already said: use identical databases (structurally) for development, test and production.
The goal of schemas is to group database objects, like we do with namespaces in e.g. C#, or to simplify permissions for groups of database objects. Not to identify database stages. By using them for the latter you also make it much harder, if not impossible, to use schema appropriately. See for instance this MSDN white paper.
It is much easier to use some database name conventions to indicate their purpose.

Synchronizing data with identity columns between SQL Express Servers using Entity Framework

I have designed a database whose MDF file will be copied to remote offices, so basically I will have different databases wth the same scheme. However, some tables from these databases will have to contain the same data. First I was happy because I knew it was easy to sync them using RowVersion columns in each table, but then I remembered that primary key columns in these tables (columns named "ID") are also identity columns. So I have no idea on how to synchronize them in way that they are identical. With same IDs and everything. Also I am doing this through Entity Framework, which sits between the SQL Server 2008 R2 Express and .NET Framework 4 WCF Service. Any clues?
Note that this is a one-way sync, remote offices need to replicate these tables from the main database but they are not able to modify them and write changes back.
The original thread was started here: http://social.msdn.microsoft.com/Forums/en-US/adodotnetentityframework/thread/e5f89bac-959c-490a-befc-a80d5aa9a9a5/ but I haven't come to a solution yet. If you take a look at the thread I linked to, you will see that the proposed solution was to attach records from the main DB context to the client DB context and call "ApplyCurrentValues" method to update the client DB. However I have come to conclusion that it would not work at all due to these reasons:
Different EntityKey values between data from two contexts. You can't attach a record to a context if that record's EntityKey doesn't correspond with the context. To get past this issue I had to convert the object from mainDB to the object from clientDB using AutoMapper and set the EntityKey manually prior to attaching the record to clientDB context.
If you want to add a new record (if one exists in mainDB but not in clientDB) you can't use Attach. If the record you are trying to attach doesn't exist in the store, EF will throw the exception back at you.
If you want to add a new record, you must use AddObject, but that implies the EntityKey is generated automatically and you will not have control over the identity column. If you try to set EntityKey manually prior to adding a new record, EF will throw an exception at you.
So, the question is, how can I replicate data from the main DB to the client DB using EntityFramework?
We have recently implemented this solution, however our database was simple enough and we had one meta server (we call it meta as it is our server which holds only identities) and we have data servers. We have three data servers doing three way sync. Now originally we only had three servers, but inserting new IDs were problem and we didnt want to use GUID as well because it is not human readable.
So we introduced concept of IdentityServer (we called it MetaServer), which hosts a simple Web Service and simple database, database consists of Tables with only Identities, Hash and LastUpdate, Hash and LastUpdate are used to validate synchronization.
For example, following two tables are there on Meta Server,
Tickets
TicketID (Primary Key,Identity)
LastUpdate (DateTime)
Hash (Hash of Ticket)
Tasks
TaskID (Primary Key, Identity)
LastUpdate (DateTime)
Hash (Hash of Task)
Now Data Servers will contain Tickets as follow,
Tickets
TicketID (Primary Key)
Subject
Message
...
...
Tasks
TaskID (Primary Key)
Subject
Message
...
...
And our Save method on ObjectContext looks like following,
Task task = new Task();
task.TaskID = MetaService.GetNewTaskID();
...
...
// following is save method, checking insert or update, as it is used in
//synchronization, thats why i wrote it like this
void SaveTask(Task task){
Task copy = ObjectContext.Tasks.FirstOrDefault(x=>x.TaskID==task.TaskID);
if(copy==null){
copy = new Task();
ObjectContext.Tasks.AddObject(copy);
}
CloneData(task,copy);
ObjectContext.SaveChanges();
}
To Perform syncing, I would suggest, add a table like this which will Save every Change in Meta Server (Master Server)
Changes
ChangeID
ChangeType = Insert,Update,Delete
ChangeTable
ChangeKey
ChangeTime
Which then every data server can read from Meta Server and update changes...

Entity framework 4 and multiple database

Something changes or it still not support this?
For example join database1.dbo.Users and database2.dbo.Addresses
I actually did find a way to make an EF model span multiple databases if your database supports Synonyms. Basically you setup Synonyms to Database2 on Database1, create separate edmx models for each, then merge the XML
I posted the exact steps to make an edmx file span multiple databases here if you're interested, along with a script to do the merge for you whenever something changes.
I think what ais asked is if you can join tables from different databases, not different providers, resulting in one entity mapped to two or more tables or views from different databases.
If you think about it, when you create a EDM model with Visual Studio it ask you to give an existing database, and when finished creating the model, it generates an EF connection string, that internally address to the given underlying database connection string.
E.g: metadata=res:///EFTestModel.csdl|res:///EFTestModel.ssdl|res:///EFTestModel.msl;provider=System.Data.SqlClient;provider connection string="Data Source=.\;Initial Catalog=EFTest;Integrated Security=True;MultipleActiveResultSets=True"*
So each model matches only a database, only a connection string.
EF4 still does not support creating one conceptual model which works with N storage models. At least this is not supported with any built-in provider. Perhaps in the future this could be done through a new provider that combines the support of many storages (from the same providers or different).
I havent done enough research on it, but perhaps Windows Server AppFabric (Codename Velocity) could be the bridge to go through this gap.
Note: I have tried even editing manually the xml for the EDM (edmx) to insert a second element inside the <edmx:StorageModels> tag but it does not match the EDM XML Schema so VS warns about it:
Error 10021: Duplicated Schema element encountered.
Rafa Ortega
MAP2010
See answer to similar question:
Entity Framework - Inserting entity with multiple models and databases