I'm creating a web shop using .NET Core and I'm storing all of the product information in a SQL database using EF Core.
As a part of this, I would like to store the x, y, and z dimension of each product. But I'm wondering about the best way of storing three values that belong together in this way in a database using EF Core.
EF Core doesn't allow me to store them as an array or list of for example floats or integers, and I want to avoid creating a dedicated model concered only with the three dimensional size values for each product.
Any suggestions?
Store it as a Json column (Refer this answer) and if you frequently run read queries on it then setup indexes.
Related
I am developing a multi-step data pipeline that should optimize the following process:
1) Extract data from a NoSQL database (MongoDB).
2) Transform and load the data into a relational (PostgreSQL) database.
3) Build a data warehouse using the Postgres database
I have manually coded a script to handle steps 1) and 2), which is an intermediate ETL pipeline. Now my goal is to build the data warehouse using the Postgres database, but I came across with a few doubts regarding the DW design. Below is the dimensional model for the relational database:
There are 2 main tables, Occurrence and Canonical, from which inherit a set of others (drawn in red and blue, respectively). Note that there are 2 child data types, ObserverNodeOccurrence and CanonicalObserverNode, that have an extra many-to-many relationship with another table.
I made some research regarding how inheritance should be implemented in a data warehouse and figured the best practice would be to merge together the family data types (super and child tables) into a single table. Doing this would imply adding extra attributes and a lot of null values. My new dimensional model would look like the following:
Question 1: Do you think this is the best approach to address this problem? If not, what would be?
Question 2: Any software recommendations for on-premise data warehouses? (on-premise is a must since it contains sensitive data)
Usually having fewer tables to join and denormalizing data will improve query performance for data warehouse queries, so they are often considered a good thing.
This would suggest your second table design. NULL values don't occupy any space in a PostgreSQL table, so you need not worry about that.
As described here there are three options to implement inheritance in a relational database.
IMO the only practicable way to be used in data warehouse is the Table-Per-Hierarchy option, which merges all entities in one table.
The reason is not only the performance gain by saving the joins. In data warehouse often the historical view of the data is important. Think, how would you model a change in a subtype in some entity?
An important thing is to define a discriminator column which uniquely defines the source entity.
As part of my final thesis, I must transform a relational database in a graph-oriented database, specifically a PostgreSQL database into a Neo4j embedded database. Now, the way is the problem. In Rik Van Bruggen's book: Learning Neo4j, he mentions a data import process using ETL activities with Trascend and MuleSoft tools, but in their official sites, there's no documentation about how to do it, neither help documentation nor examples. Apart from these tools, what other ways can I use to transform this information without using my own code?
Some modeling advice:
A well normalized relational model, which was not yet denormalized for performance reasons can be translated into the equivalent graph model.
Graph model shapes are mostly driven by use-cases, so there will be opportunity for optimization and model evolution afterwards.
A good, normalized Entity-Relationship diagram often already represents a decent graph model.
So if you still have the orignal ER diagram available, try to use it as a guide.
Here are some tips that help you with the transformation:
Each entity table is represented by a label on nodes
Each row in a table is a node
Columns on those tables become node properties.
Remove technical primary keys, keep business primary keys
Add unique constraints for business primary keys, add indexes for frequent lookup attributes
Replace foreign keys with relationships to the other table, remove them afterwards
Remove data with default values, no need to store those
Data in tables that is denormalized and duplicated might have to be pulled out into separate nodes to get a cleaner model.
Indexed column names, might indicate an array property (like email1, email2, email3)
JOIN tables are transformed into relationships, columns on those tables become relationship properties
It is important to have an understanding of the graph model before you start to import data, then it just becomes the task of hydrating that model.
LOAD CSV might be your best option, but of course it means outputting a CSV first. Here are some great resources:
http://neo4j.com/docs/stable/query-load-csv.html
http://watch.neo4j.org/video/112447027
http://jexp.de/blog/2014/06/load-csv-into-neo4j-quickly-and-successfully/
http://jexp.de/blog/2014/10/load-cvs-with-success/
http://www.markhneedham.com/blog/2014/10/23/neo4j-cypher-avoiding-the-eager/
I've also written a ruby gem which lets you write a little ruby code to import data from various sources. It's called neo4apis. You can look at the neo4apis-twitter gem to get an idea for how it works:
https://github.com/neo4jrb/neo4apis-twitter/
https://github.com/neo4jrb/neo4apis-twitter/blob/master/lib/neo4apis/twitter.rb
I've actually been wanting to implement a neo4apis-activerecord to make it easy to import from SQL with ActiveRecord
You can not directly export data from relational and import to neo4j.
Because these are two different database structures.
Relational Database -
A relational database is a set of tables containing data fitted into predefined categories. Each table (which is sometimes called a relation) contains one or more data categories in columns. Each row contains a unique instance of data for the categories defined by the columns.
Graph-oriented database -
A graph database is essentially a collection of nodes and edges. Each node represents an entity (such as a person or business) and each edge represents a connection or relationship between two nodes.
Sollution To your Problem-
First, you need to design Neo4j Data structure. e.g What will be the nodes you required, what will be the relationships between the nodes.
After that you create Script in your application language to fetch data from relational database and insert it into neo4j.
Load CSA is a option to Import/Export (backup) functionality with graph database. you can not directly Export/Import data from Relational DB to Graph DB
I am working on creating a service layer for a large sql server database (2008 R2) that is currently the backend for a winforms POS application with strongly typed datasets.
I think WCF is the way to go, and at first glance it seemed EF 4 was a good choice but now I'm having my doubts. Here is what I have found:
The stored procedure mapping isn't that great. I have hundreds of stored procs that I want to reuse. Most of them wouldn't return an 'entity' so the stored procs would have to be mapped to a complex type. Many of the procs use dynamic sql or temp tables so EF can't figure out what complex type to crete. Many of the procs return multiple result sets. I've read that EF extensions have a way to map stored procs with multiple result sets, but only for entities, so that doesn't help me.
Large models are a problem. There doesn't seem to be a good way to handle large entity models. The workaround of creating smaller models isn't that desirable and splitting the model loses design support, am I missing something?
EF mappings only go so far. The stored procs that I want to reuse return projections or information from many tables into a result set. There doesn't seem to be a way to map these results into entities, am I wrong? I've read about combining results from 2 table into 1 entity, but that only works if the tables have the same primary key.
Are people using EF in large scale existing databases? If not what would you recommend?
I've used EF on large scale databases, but as you say, the support for SPs as you have got is not great. That's not specifically a failing of EF per-se - ORMs in general work on the same principle and have the same "limitation".
If you have lots of SPs and are mapping them to datasets, you'll have to do lots of work even without SPs in terms of no longer referencing datasets and referencing your domain model types through your system, so you'd need to have some way to map your SPs to your domain model and back anyway.
I have a doubt about Core Data migration.
Say I have an application which has some predefined values in a table A. I want to sync it with another database, with a table B in such a way that when new records are added totable B, that record should get added to my table A.
I know using Core Data migration, when I add columns to a table, I will be able to access the values previously stored in the older table before the addition of the column.
I would like to know how my table can be updated with the added records on another table.
Update:
From comment below:
The question I had in mind is this...
I want to release an update for my
app. I'm stuck on how to update the
existing Core Data database which also
stores data entered by the user. All I
need to do is update a couple of
records and preserve current user
data. How do I do this?
Core Data is not SQL. Entities are not tables. Objects are not rows. Columns are not attributes. Core Data is an object graph management system that may or may not persist the object graph and may or may not use SQL far behind the scenes to do so. Trying to think of Core Data in SQL terms will cause you to completely misunderstand Core Data and result in much grief and wasted time.
That way lies madness.
It sounds like you don't actually want to migrate as the term is used in Core Data. Migration in Core Data means moving from an earlier version of a data graph's persistent store to a newer version of the same.
E.g. In the 1.0 version you have an entity Person with the attributes firstNameand lastName. After the app has been release you wish to update to the 2.0 version and add a phoneNumber attribute to the Person entity. You would use migration to update the user's existing object graphs and persistent stores to the new object graph.
If by "table" you actually mean entities, then you can link entities together in a relationship so that they can watch each other. If by "table" you mean a data model or persistent store, then the answer is more complex. It can be done using configurations, fetched attributes, UUIDs etc but you must understand what you really need to do before you jump through all those hoops.
I have a SQLite database. Should I put the DB in a data structure with Core Data. How can I do? My problem is "z relations" between tables.
It's possible?
Core Data isn't SQL even when it employs an SQLite store. Although it is theoretically possible to convert a standard SQLite file to the schema Core Data uses, that is difficult and risky especially given that Apple doesn't document the schema and can therefore change it without warning. You really need to translate the SQL data into Core Data objects.
The best way is to write a utility app containing you Core Data model. Read in the SQL data with the standard functions and then use that data and relationships to create the appropriate managed objects and object relationships in Core Data.
Usually you have code anyway for creating managed objects, populating attributes and setting relationships. Just use that code but instead of providing the data from the UI or a feed, provide it from the data provided by SQL.
I found a solution. In the future, should I use SQLite directly, but for those who have a similar problem to mine this solution works well.
Step 1: Core Data in your table add column headed gl'ID temporary relations of the original table.
Step 2: In the data in CSV add two columns. The first column contains the value 1 and refers to P_OPT of Core Data and the second column contains the identifier of the table and retrieved P_ENT generated by reading the SQLite Core Data in the table Z_PRIMARYKEY.
Step 3: With any editor Mac transfer your data in SQLite files generated by Core Data. Remember to attach gl'ID (relations) in the temporary columns.
Step 4: Through the use of the SQL UPDATE command (works with any SQL editor on the Mac) updates all ID columns of relations in Core Data with the value Z_PK. The value retrieved by the queries and the use of temporary columns.
Sorry for the bad English. I hope not to have been convoluted with the explanation and useful to others.