The idea of the SaaS tool is to have dynamic tables with dynamic custom fields and values of different types, we were thinking to use "force.com/salesforce.com" example but is seems to be too complicated to maintain moving forward, also making some reports to create with a huge abstraction level, so we came up with simple idea but we have to be sure that this is kinda good approach.
This is the architecture we have today (in few steps).
Each tenant has it own separate database on the cluster (Postgres 12).
TABLE table, used to keep all of those tables as reference, this entity has ManyToOne relation to META table and OneToMany relation with DATA table.
META table is used for metadata configuration, has OneToMany relation with FIELDS (which has name of the fields as well as the type of field e.g. TEXT/INTEGER/BOOLEAN/DATETIME etc. and attribute value - as string, only as reference).
DATA table has ManyToOne relation to TABLES and 50 character varying columns with names like: attribute1...50 which are NULL-able.
Example flow today:
When user wants to open a TABLE DATA e.g. "CARS", we load the META table with all the FIELDS (to get fields for this query). User specified that he want to query against: Brand, Class, Year, Price columns.
We are checking by the logic, the reference for Brand, Class, Year and Price in META>FIELDS table, so we know that Brand = attribute2, Class = attribute 5, Year = attribute6 and Price = attribute7.
We parse his request into a query e.g.: SELECT [attr...2,5,6,7] FROM DATA and then show the results to user, if user decide to do some filters on it, based on this data e.g. Year > 2017 AND Class = 'A' we use CAST() functionality of SQL for example SELECT CAST(attribute6 AS int) AND attribute5 FROM DATA WHERE CAST(attribute6 AS int) > 2017 AND attribute5 = 'A';, so then we can actually support most principles of SQL.
However moving forward we are scared a bit:
Manage such a environment for more tenants while we are going to have more tables (e.g. 50 per customer, with roughly 1-5 mil per TABLE (5mil is maximum which we allow, for bigger data we have BigQuery) which is giving us 50-250 mil rows in single table DATA_X) which might affect performance of the queries, especially when we gave possibilities to manage simple WHERE statements (less,equal,null etc.) using some abstraction language e.g. GET CARS [BRAND,CLASS,PRICE...] FILTER [EQ(CLASS,A),MT(YEAR,2017)] developed to be similar to JQL (Jira Query Language).
Transactions lock, as we allow to batch upload CSV into the DATA_X so once they want to load e.g. 1GB of the data, it kinda locks the table for other systems to access the DATA table.
Keeping multiple NULL columns which can affect space a bit (for now we are not that scared as while TABLE creation, customer can decide how many columns he wants, so based on that we are assigning this TABLE to one of hardcoded entities DATA_5, DATA_10, DATA_15, DATA_20, DATA_30, DATA_50, where numbers corresponds to limitations of the attribute columns, and those entities are different, we also support migration option if they decide to switch from 5 to 10 attributes etc.
We are on super early stage, so we can/should make those before we scale, as we knew that this is most likely not the best approach, but we kept it to run the project for small customers which for now is working just fine.
We were thinking also about JSONB objects but that is not the option, as we want to keep it simple for getting the data.
What do you think about this solution (fyi DATA has PRIMARY key out of 2 tables - (ID,TABLEID) and built in column CreatedAt which is used form most of the queries, so there will be maximum 3 indexes)?
If it seem bad, what would you recommend as the alternative to this solution based on the details which I shared (basically schema-less RDBMS)?
IMHO, I anticipate issues when you wanted to join tables and also using cast etc.
We had followed the approach below that will be of help to you
We have a table called as Cars and also have a couple of tables like CarsMeta, CarsExtension columns. The underlying Cars table will have all the common fields for a ll tenant's. Also, we will have the CarsMeta table point out what are the types of columns that you can have for extending the Cars entity. In the CarsExtension table, you will have columns like StringCol1...5, IntCol1....5, LongCol1...10
In this way, you can easily filter for data also like,
If you have a filter on the base table, perform the search, if results are found, match the ids to the CarsExtension table to get the list of exentended rows for this entity
In case the filter is on the extended fields, do a search on the extension table and match with that of the base entity ids.
As we will have the extension table organized like below
id - UniqueId
entityid - uniqueid (points to the primary key of the entity)
StringCol1 - string,
...
IntCol1 - int,
...
In this case, it will be easy to do a join for entity and then get the data along with the extension fields.
In case you are having the table metadata and data being inferred from separate tables, it will be a difficult task to maintain this over long period of time and also huge volume of data.
HTH
I am trying to add relationships for multiple tables but Power Bi is only allowing ONE table to be with "Both" for the "Cross Filter Direction".
If I try to select other tables to be with "Both" cross filter, then an error comes up saying "Power BI Desktop allows only one filtering path between tables in a Data Model". Is there a way to go around that?
I have three data sets ( tables ), that are joined based on two non-unique columns. First dataset (table) has site, item numbers, orders, due dates. Second dataset (table) has site, item numbers, and available quantities. Third dataset has site, item numbers, and released orders to manufacture. The three non-unique columns are sites and item numbers.
Out of these three datasets (tables), I have created two more datasets with one unique columns (sites, and item numbers) to inforce unique relationships between the three datasets.
The problem that I am facing is when I have filtering in the models it doesn't flow back and forth between the datasets. I assume fixing the filtering direction will result in fixing that.
Thanks!
I would not use the Both cross filter direction for that scenario. As you are observing it comes with a lot of restrictions, so I only use it for "Many-to-Many" scenarios - not what you described.
I think you are on the right path with your sites and item number tables. Just be careful to use the copy of the fields in those tables (e.g. item number from item numbers) in your visuals, not the same fields in the 3 "data" tables.
I'm creating a compliance mailing for my organization, the mailing will include merge fields that identify the office location, physician, and SiteId. The mailing will also include a table of information that is dependent upon the particular SiteId.
I'd like to use the import table function of MS word and set up a query that references a merged field (SiteId) so that the inserted tables populate the appropriate data for the particular site. I'm unable to do this.
How can I set up this document so that I can import only records from my source (an ms access query) that match the SiteId merge field?
Word's mail merge does not support one-to-many relationships. There are ways to coerce it, but only one of them can yield a table as a result and over the years it has become less and less reliable as Microsoft has not regarded it as important enough to maintain...
What you need to do is set up a query that provides ONLY the information you want displayed in the table, plus the key (SiteId). It's best to sort it so that all the SiteId entries list together, and are in the order the data will come through in the mail merge data source.
On the Insert tab go to Text/Quick Parts/Insert Field and select the Database field from the list in the dialog box. Click "Insert Database" and follow the instructions in the dialog box to link in the data. Be sure to set the Query Options to filter on the first SiteId from the data source. When you "Insert Data" make sure to choose the option to "Insert as a field".
This inserts a DATABASE field in the document which you can see by toggling field codes (Alt+F9). The field code can be edited and what you need to do is substitute the literal SiteId value you entered for the query with its corresponding MergeField.
When you execute the merge to a new document that should generate a table for each data record corresponding to the SiteId for the record. But, as I said, Microsoft hasn't done a great job of maintaining this, so it may require quite a bit of tweaking and experimenting.
If the results are not satisfactory then you should give up the idea of mail merge and use automation code to generate and populate the documents.
You can find more (albeit somewhat out-dated) information on this topic at http://homepage.swissonline.ch/cindymeister/mergfaq1.htm
I'm writing a budget database, and while planning out the tables went fine, forms are proving to be trickier.
I have a 'transactions' table, and two queries based off it. The tables and column names are below
'people_to_reimburse' : payee_name, total
'unwritten_checks' : payee_name, amount, description, date_incurred
I'd like to make a form where you can go through the people to reimburse, and there's a subform showing which checks are theirs. Obviously, the payee_name would be the field to link on.
However, in Access's Form Wizard, when I select these two tables, I get an error of:
You have chosen fields from record sources which the wizard can't connect. You may have chosen fields from a table and from a query based on that table.
How would I set up a form like this, if it's possible? If not, why can't I?
I have a form for entering patient data, which is bound to a number of tables, including patient, hospital, symptoms, treatment and so on. I'd asked a query previously and been advised to change the design to reduce the number of tables and therefore joins, which I've done. When I enter new data now using the form it is saved to the underlying tables correctly. However when the form is opened it is displaying only the first of 2 rows in the tables. The user will enter data in some fields but not all of them, so would the form (and the record source query under it) not be returning the next record because it does not appear in ALL the linked tables, only some? For example, all records will have a row in the Patient table, but not all will have a record in the Surgery table, yet the form needs to be able to record data for all possible fields. Any advice gratefully received!