Need a push in the right direction with this one.
I want to "dev0ps" the workflow of our local development with docker.
As best practice our mongodb should run in a seperate container, having a volume attached and working. (this part checks out fine)
Our developers should then keep the data in sync, so its able to be "pushed & pulled".
Can i achieve this with git? (The data folder is around 500mb, but this is just a fresh project)
Should i write a script that performs a mongodump and upload it (to git?)?
Should i consider spinning up a mongodb server of my own, where they push and pull from?
thanks in advance!
I see what you're trying to achieve, but I don't think the best approach is to keep neither the data from one local working copy in sync inside Git, as this is likely to provide inconsistencies in the Database if Developers have different copies of the data.
The best thing you could do is have a set of Fixtures (Application Data inside the MongoDB database) which is inserted and deleted via your code, that way you control what goes inside the database and what does not, That data can very well be included in the Git repository in a folder called fixtures or database.
You could also have different sets of data, one for developing, one for testing, etc. As you can see you can make it as complex or complementary as possible.
I've been using this approach for more than one year and has not presented any problems.
Let me know if you have any concerns about this approach.
Related
I'm currently building my first real project that includes Express and MongoDB. Since it's one of the first backend-heavy projects I've worked on outside of my Udemy course, I've run into a lot of questions.
My project is supposed to be a mock-online store that would display items I have created inside of my MongoDB server. The problem I'm having is that I don't know the proper way of serving those image files that should be associated with each item (such as the image of a hat, for a hat item). I could add them directly into the project's public folder, but I don't know if that would be feasible in terms of the scalability that I want this project to demonstrate. But it doesn't seem like MongoDB will let me store images within each item. How would I go about doing that?
Sorry in advance if any of this is unclear, it's my first time posting as well. I'll try and provide more information if I need too. Thanks!
If you want a scalable solution for images, you typically would use a separate service like AWS S3 or Imgix.
There are several benefits to using a 3rd party service. You don't bog down your web server with image requests, or image resizing. You get virtually unlimited space. Etc.
In your MongoDB document, you would then store a key like /item/1.jpg or whatever, rather than the image itself. Your front-end then uses the key to request the image when someone visits your website.
If you want a turn-key solution, I recommend starting with Imgix (or Cloudinary, or some similar service). It is more expensive than S3, but it is pretty cheap for a small project, and it will get you up and running a lot faster.
I am working on Tableau Server where I have say, three solution environments viz. Development Staging and Production.
I have created workbooks in Development environment and each workbook use many data sources.
For instance lets say I created the workbook Account Workbook and I have used these two data sources Dev-DataSource1 and Dev-DataSource2. I have other data sources available like Stage-DataSource1, Stage-DataSource2, Dev-DataSource3, Stage-DataSource3 etc.
The problem I am facing is while moving workbooks from Development to Staging environment, the data sources remains the same.
So continuing with the same instance if I move Account Workbook to Staging from Development environment, Account Workbook still uses the same data sources Dev-DataSource1 and Dev-DataSource2. I want it to use the Stage-DataSource1 and Stage-DataSource2 .
This might be a very trivial question or maybe a bad design, but I have created so many workbooks such that creating them again might take loads of time, so please refrain from asking me to recreate those workbooks again. If there is no other way other than that then do tell me.
Thanks in advance
Can you not just edit the data sources, for instance from Dev-DataSource1 to Stage-DataSource1?
I assume they have the exact same structures, field names, etc?
Right click on the data source and click "Edit". When in the data source page, pull in table(s) from
the new datasource and replace the one(s) that are there. If you have joins you may need to recreate them. If you have custom sql you just need to change your sql to point to the new table(s).
You will need to refresh your data sources. It should recognize the fields assuming they are exactly the same as before.
If you need to edit the connection, you can do that as well (if you need to point to a different server, etc.)
Make sure you have a backup of your workbook in case something goes wrong.
It may also be worth your time to look into the Tableau Python document API: https://github.com/tableau/document-api-python
This would allow you to do (some/all?) of what are describing programmatically.
I'm working through the issues of using an offline data sync with a mobile back end, and have worked through numerous articles/documentation regarding it's use.
The scenario that I'm looking for is a bit different that the normal demo projects. We have a Azure SQL db that we have been using for some time now, and a use case has come up to use this data in a mobile back end.
I have yet to see an article on how best practises of using an existing sql DB (EF codefirst), with offline data sync tables that need to inherit from EntityData, which adds additional fields for the sync process.
My original thought was to use the ef code first definition of the data models, and use them with Mobile App... however the requirement for EntityData doesn't work, as I'm not going to add those fields to a production system.
My question is, what is the best practise of using data from a production system, and get them syncing with mobile back ends? I'm thinking a intermidary DB is required, but that just means that there are three databases/tables to sync - which doesn't feel right at this point in time.
Anybody know of an article that starts with an Azure DB, and the process/decisions they made to allow data sync to offline devices?
This scenario is discussed in detail in my book - http://aka.ms/zumobook - chapter 3.
I have a postgresql database which I keep updated using Osmosis. Osmosis can write to two different database schemas, named Simple and Snapshot. There are not that much different from the database Geoserver uses, But I can't make Geoserver use it perfectly.
The main problem seems to be the way tags are stored in those DBs. I can add the nodes layer and display it with that default Points style, but as soon as I use a "ogc:Filter" in my style to filter the nodes by their "place" tag, the WMS is broken and does not respond (says: The requested Style can not be used with this layer. The style specifies an attribute of place and the layer is: TestDB:nodes)
Is there anyway to make GeoServer understand that one of those shemas, or make Osmosis update to the DB GeoServer knows?
This is a case for using TRIGGERs to manage the integration. The two programs use two different schemas. You can CREATE TRIGGERs in the database which ensure that data written to one application is made available to another. Another option is you can set one or both to use VIEWs populated in part by the other application. In PostgreSQL, a VIEW can have triggers attached so these are not really
This is, in any case, a potentially large project so rather than offering sample code, I will offer a general outline of what sorts of things you need to think about.
Are these generally applicable? If so do you want to start an open source integration project?
Are both of these read-only workloads? Does data ever update? In general, if you are going to use views, updates pose the most concerns, so you want to run the views on the side not doing the updates if such is the case.
What is the write model of both sides? Insert/Update? Append only? Static data? What data do you have to "replicate" between the schemas?
Once you have those answers it should be relatively straightforward to get started and ask for help (either as an open source project or here) where you get stuck.
I have a system where there are two identical databases. One is for back of house work where data is imported, edited generally worked on. Once the data in the first database is as required it is coped to the second database, which is used to drive a public facing (read only) site.
So once a month, or so I will need to push data from database to another. I'd like to drive all this with EF, is that reasonable, can EF do this kind of thing, or will I get stuck part way down the line?
It's probably doable, but frankly, EF (or any other ORM) is not really suited for this kind of task. If you do decide to implement your synchronization tool with EF, at least make sure to turn off change tracking.
I wouldn't dismiss Yuri's suggestion (simply using a scheduled backup/restore), if the databases are really identical. It's certainly the easiest to implement!
Another solution would be to use a database synchronization tool, like Sql Server Integration Services.