I am confused regarding PostgreSQL BDR and I have several questions:
Question 1: Is BDR truly multi-master for PostgreSQL?
According to the docs here, it says that:
The BDR (Bi-Directional Replication) project adds multi-master
replication to PostgreSQL 9.4
but if I read on 2ndQuadrant, I read the following:
If I read that part, they don't mention multi-master much at all; just that a "second master, working in passive", which indicates its not a real master?
Question 2: Is BDR open-source?
I read here that it is, at least that it was:
BDR is the first open source multi-master replication system for PostgreSQL
Is it still? Because when I look, I am often directed to 2ndQuadrants webpage, and that gives me the impression that its not open-source, when they say that:
How can you get Postgres-BDR?
Just fill out the contact form below and a PostgreSQL expert will be in touch shortly!
Sounds like selling to me =)
Question 3: What version is what?
I read that 2ndQuadrant released version 1.0.5 in March this year. I also read on 2ndQuadrants webpage that
In the complex environment of replication, the 3rd generation of BDR achieves...
The 3rd gen? Is version 1.0.5 that same 3rd gen, or is it something else?
Also, the same page says that:
Note for current Postgres-BDR users: BDR 1.x will reach EOL in December 2019. Our team of PostgreSQL experts can help plan and execute your upgrade with minimal impact and almost zero downtime. Contact us today and a member of our professional services team will be in touch with you as soon as possible.
So, 1.0.5 was released in March, but has EOL in December 2019? Is 2.x not open-source, so some license cost associated with it, and 1.x is EOL 2019?
BDR1 is open source. BDR2 is not. BDR3 is not yet, but should become so at some later stage.
BDR is truly multi-master. The "AlwaysOn Architecture" is a simplified model for BDR deployments that uses active/standby with fast-failover, designed to retain better compatibility with existing applications while improving HA and robustness.
So BDR can and often is deployed in fully multi-master roles, the AlwaysOn architecture just doesn't use it that way.
The BDR 1.x series for PostgreSQL 9.4 (+BDR patches) is open source. It will go EoL in December 2019. It works fine, but I don't recommend it for new deployments given the planned EoL.
The BDR 2.x series (for PostgreSQL 9.6) is not open source and is only available for 2ndQuadrant customers. However, parts of it have been submitted to PostgreSQL itself. It has been superseded by BDR 3.x.
The BDR 3.x series, which is now entering production, is not currently open source and is available only to 2ndQuadrant customers. My understanding is that it's intended for eventual open source release, but no date has been set, and I cannot speak officially for 2ndQuadrant about this. BDR3 adds a much more robust node communication model, better conflict handling, and a lot more, plus it runs on PostgreSQL 10 and 11.
I have been encouraging the relevant people to provide some updated official guidance on these matters. The latest I have for you right now is "News and Roadmap for BDR (Multi-master PostgreSQL)" on the 2ndQuadrant blog.
I received an answer from 2ndQuadrant via email, so I will post it here as it addresses the questions above:
1- "BDR is truly master-master; the shadow master is still a master.
BDR is an eventually consistent multi master solution; in eventually
consistent multi master cluster, it is possible to write on more than
one master at the same time, and conflicts might arise when the same
rows are written at the same time. Conflicts might be acceptable or
not depending on the logical model of the application. Some people do
not need to write on both nodes at the same time, and will use BDR
only to achieve faster failover, as in our BDR-AlwaysOn architecture.
Other people need to write on both nodes, and in that case we need to
assess impact and likeliness of conflicts."
2- BDR 1.x is open source (http://bdr-project.org/docs/stable/) .
Later versions including BDR3 is only available to 2ndQuadrant
Production Support customers. Happy to talk about that in more detail.
You are right, it does sound like selling, we are a business :)
3- The latest version is BDR3, this is the third generation of BDR. It
will still be live, but is only available to Support Customers.
1.x is open source, but EOL as you mentioned.
also, "is only available to Support Customers", meaning it is not a product you can buy one time. You have to pay. Every year. It is a product you rent :( .
Related
We are using a Variscite VAR-SOM-AM33 platform for our project, and software platform is based on OpenEmbedded/Yocto.
To ensure the hardware is running with the current software, the devices are connected to the internet. So far, we have been following the OE recipes and generating ipk and applying software updates via opkg.
However, the process is not satisfactory as some of the recipes are poorly written (fails to uninstall/install during the upgrade process). What robust technique/solution are available for OE/Yocto based systems?
Thanks in advance.
I'd like to add SWUpdate to the list of packages that you should consider. It was recommended in a 2016 paper by the Konsulko Group for Automotive-Grade Linux. That paper mentions a few other options, and provides an analysis of the various tools, so it's probably worth a read. Quoting from the paper:
It is our recommendation that the reference AGL software update strategy make use of SWUpdate in a dual copy configuration and integrate OSTree support. This allows recovery from a corrupt partition for the exception case, but also optimizes the common case where small, incremental updates can be quickly applied or rolled back as needed to [meet] OEM policy.
I don't completely agree with the paper. For example, they wrote off Mender.io because it lacks community support, but IMO the Automotive-Grade Linux group is influential enough to create popularity from scratch. Still, it's a good paper, and the fact that they settled on SWUpdate was interesting to me. I was already leaning toward it because the author, sbabic, is involved in U-Boot software development, and we use U-Boot to burn new images into our device.
At the moment I'm unsatisfied with all of the current options, but mostly because I want extra functionality. I'll probably settle on a custom system which incorporates one or more of the aforementioned packages. Unfortunately that's not the kind of definitive answer that SO prefers, but I hope that it was helpful.
I'm working on a metadata layer to integrate the Software Updater (swupd) from Clear Linux with the Yocto Project / OpenEmbedded Core.
swupd performs whole of OS updates, rather than package-based updates, using binary deltas to only update the files which change and to do so in an efficient manner.
I recently wrote some documentation (within the docs/Guide.md file in the meta-swupd repo) about adopting the "Clear Linux Way" to utilise meta-swupd from an OE/YP based distro. A wikified version of that guide, including a link to the layer git repository, are available on the Yocto Project wiki:
https://wiki.yoctoproject.org/wiki/Meta-swupd
I also have a sample layer on Github which demonstrates use of the layer (this is also the distro layer I test much of meta-swupd with):
https://github.com/incandescant/meta-myhouse
About mender.io, I have recently talked to them regarding their open-source update.
Currently, they already have their client side developed, and is working on the server side. They use HTTP and JSON. This is their git, it is only supporting Beaglebone and QEMU at the moment.
The way mender.io works is: they will have one persistence data and uboot. and 2 rootfs (active and backup) to update. So, when there is an update on the server, the users will be notified to pull it down. Give a mender -rootfs image update command. And if the upgrade is success, the user gives another mender -commit command. If there is no mender -commit, the rootfs will be rolled back to the previous rootfs in the next reboot. Mender currently only support update of kernel and rootfs.
The main role of mender.io is to ensure that the mass distributed image upgrade process is recoverable from errors. In the Server side, mender.io developed a management server to the mass distributed devices using UUID.
Not to advertise but please try out mender.io and give feedback so that the software could be more mature.
Mender Introduction pdf
Well, you can either use package based upgrades, like you do. In that case, you'll need to test and verify everything locally before you push any updates to the field. Obviously, you'll likely need to improve a number of recipes. (And I assume that you upstream those immprovements, right?)
The alternative is to use image-based upgrades. Either with full images, see for instance the discussion at Stackoverflow: Embedded Linux mechanism for deloying firmware updates or swupd
Note: I got distracted while writing this answer, so look at the answer from joshuagi; he explains a lot more of swupd.
I think they are two problems here. We (OpenEmbedded) do need to be careful that we do not break package based updates.
Also, there are image updates like swupd (mentioned above and swupdate, described at: https://sbabic.github.io/swupdate/swupdate.html
meta-updater provides support for OSTree-based updates to OE systems.
OSTree is interesting because it provides a half-way house between full image updates (which are large and tricky to handle correctly) and package based updates (which are tricky to make robust). It has a 'git-like' object representation of a root filesystem, and uses chroot and hard links to atomically switch between file system images.
(Disclosure: I'm a contributor to meta-updater)
Posts here were done years ago. Now also RAUC seems to be promising alternative to mender.io
Several days ago, Mirth Connect version 3.3.0 was released. Noting the great new features, we decided to upgrade immediately (just days after the initial release). We followed these Upgrade Guide instructions during the upgrade. However, the specifics of upgrading from 3.2 to 3.3 are missing from these Upgrade Guide, so we did not suspect much to change on the way Mirth should be implemented...
During this process, we ran into a handful of issues that caused our production channels to go down for several hours (†).
It would have been really nice to have specific information for this upgrade. Some issues that would have been really useful to know beforehand (just examples, no need to actually answer):
Are you changing the default toString() method for objects/arrays to return JSON representations?
Does this upgrade include a db migration, meaning we can't revert to
previous version once upgraded?
Because code templates are now children of "libraries", will we need to access the code template through the library, or will we be able to call it directly (as it was in 3.2)?
A solid documentation like this would have allowed us to understand the full gravity of what needs to be accounted for when upgrading. Typically, Mirth has some documentation for each minor release. But even then, the documentation is very terse. Would it be possible for the Mirth team to start being very explicit with what the upgrade entails?
The Rails Upgrade Guide (obviously much larger team, so can spend more bandwidth on this spec) provides a really great example of what an upgrade guide should entail.
† yes, yes, I learned my lesson, I won't upgrade immediately to production anymore
The Release Notes page will give the lowest level of changes to the application, but you're right in that a better documentation is needed.
I am using IBM UrbanCode udeploy, great tool but afraid IBM will kill it sooner or later with their slow response in supporting. What are the other choices?
UCD isn't going anywhere. Its in high demand these days, and is gaining traction in z/OS shops as well.
However, there other popular tools are Chef and Puppet. Just be prepared for manual data entry and less friendly interface. They work, but are more work to use.
Are there any essential differences between CouchDB and Couchbase?
I think there are some essential differences between CouchDB and Couchbase Server that need to be pointed out.
I will not write about the advantages of switching from CouchDB to the Couchbase Server because those are described pretty much everywhere (see The Future of CouchDB by Damien Katz or
Couchbase vs. Apache CouchDB
by Couchbase). Instead, I will try to enumerate features of CouchDB that you will not find in the Couchbase Server.
All of the names relating to CouchDB and Couchbase can be really confusing, so I've updated this answer, to begin with a brief explanation of the most important ones.
Names and confusion
There is CouchDB, CouchIO, CouchOne, Couchbase, Couchbase Server, Couchbase Mobile, Couchbase Lite, CouchApps, BigCouch, Touchbase, Membase, Memcached, MemcacheDB... all different and yet related in a way not at all obvious from the names alone.
First, there was CouchDB, a database created by Damien Katz, a former IBM developer. Its official name was changed to Apache CouchDB after it became an Apache project.
A company named CouchIO was founded to work on Apache CouchDB and later changed its name to CouchOne (by "its name" I mean the company name - not the database name).
CouchOne (formerly CouchIO) merged with Membase (formerly NorthScale) to form a new company called Couchbase. Membase (the company) developed Membase (a product of the same name). Membase was created by several leaders of the Memcached project and it used the Memcached protocol. After the merger of CouchOne and Membase, Couchbase continued the development of the Membase software and later changed its name to Couchbase Server.
Today I think most people believe that Couchbase Server is a new version of CouchDB but it is, in fact, a new version of Membase. It still uses the Memcached protocol and not the RESTful API of CouchDB. Meanwhile, CouchDB is still CouchDB, actively maintained and enhanced as an Apache project.
Now to the relevant differences:
Licensing
The Couchbase Server is not entirely open-source/free software. There are two versions: Community Edition (free but no latest bug fixes) and Enterprise Edition (there are restrictions on usage, confidentiality provisions, audits by Couchbase Inc. that "will be conducted during regular business hours at Licensee's facilities" and other terms typical to proprietary software that many people may find unacceptable).
CouchDB is an open-source/free software (no strings attached) project of The Apache Software Foundation and is released under the Apache License, Version 2.0 (DFSG-compatible, FSF-approved, OSI-approved, GPL-compatible, non-copyleft, commercial-friendly).
Philosophy
I have never seen it directly pointed out but this may be actually the most important difference between those two databases because it is deeply about the underlying philosophy of distributed computing models and not only about certain features, APIs or licensing. CouchDB and the Couchbase Server completely differ in their philosophy of building distributed systems and databases.
According to the CAP theorem it is impossible for a distributed database to simultaneously provide consistency, availability and partition tolerance.
CouchDB is an AP type system (provides Availability and Partition tolerance).
Couchbase Server is EITHER a CP type system (according to Wikipedia) OR a CA type system (according to Couchbase technical update) - WHICH OF THESE IS CORRECT? PLEASE COMMENT.
Features
This is what I found to be a list of CouchDB features that are not supported by the Couchbase Server:
no RESTful API (only for views, not for CRUD operations)
no _changes feed
no peer-to-peer replication
no CouchApps
no Futon (there is a different administration interface available)
no document IDs
no notion of databases (there are only buckets)
no replication between a CouchDB database and Couchbase Server
no explicit attachments (you have to store additional files as new key/value pairs)
no HTTP API for everything (you need to use the Couchbase Server SDKs or one of the Experimental Client Libraries at Couchbase Develop so no experiments with curl and wget)
no CouchDB API (it uses the Memcached API instead)
you can't do everything from the browser (you have to write a server-side application)
no two-tier architecture for Web apps is possible (you have to write a server-side application to sit between the browser and the database, like with relational databases)
no eventual consistency
not entirely open-source/free software
not a drop-in replacement for CouchDB (seems like a drop-in replacement for Memcached instead)
Those features of CouchDB may or may not be important to you so whether the lack of them is a disadvantage or not is strictly subjective, but I think that the decision whether to switch from CouchDB to Couchbase Server or not should be based on those differences and your dependence on those feature in your current CouchDB deployments.
For example if you've got interested in CouchDB after watching The CouchDB changes feed NodeCamp talk by Mikeal Rogers or one of the great CouchApp tutorials by J. Chris Anderson then you have to realize that if you want to switch to the Couchbase Server then you will have to forget about pretty much everything they were talking about.
Because of that, I would say that Couchbase Server looks like an evolution of Memcached and Membase (not an evolution of CouchDB) and as such it looks like a great product if you are currently using Memchached or Membase. If you are using CouchDB in the most basic way then you may consider using the Couchbase Server for the same things and it may or may not perform better (if you don't mind the license restrictions). But if you are actually using any of the features that are unique in CouchDB (like the changes feed, CouchApps, two-tier architecture, peer-to-peer replication etc.) then you can either forget about those features or stay with CouchDB.
In any case, make sure to read and understand the Migration to Couchbase for CouchDB Users tutorial before you think about switching.
People often get the wrong impression (maybe after reading things like "What's the future of CouchDB? It's Couchbase.") that CouchDB is somehow obsoleted by the Couchbase Server, or that it is an old, legacy version of Couchbase. Meanwhile CouchDB is an actively maintained open-source project, Couchbase server is a completely separate project (it is a newer project but it is not a newer version of CouchDB - they are not even compatible) and since even new tools for creating CouchApps still keep being developed (eg. see the Kanso project) then CouchDB is not going anywhere soon.
I hope it clarifies the confusion. Please correct me if I'm wrong on anything here.
Update:
Couchbase Server is actually a new name for the Membase Server (the Membase Server was renamed to Couchbase Server somewhere around version 1.8). See Couchbase 2011 Year in Review:
Unfortunately, we confused the heck out of many of our potential users. In addition to Membase Server and our new mobile products we also offered Couchbase Single Server which was a packaged “distribution” of Apache CouchDB. On top of that we began releasing developer previews of Couchbase Server 2.0, which incorporated CouchDB technology into Membase Server – but this product was not compatible with Couchbase Single Server (or CouchDB). [...] Membase Server will be renamed Couchbase Server 1.8 on its next release in January – a tiny step that simply alleviates “name” confusion. As has been planned from the beginning, the Couchbase Server 2.0 release (currently at Developer Preview 3) will add index and query functionality. While Couchbase Server 2.0 will incorporate substantial technology from the CouchDB project, it will not be upward compatible with CouchDB and it shouldn’t be viewed as a “version of CouchDB.” [emphasis added]
See also:
Comments to "The Future of CouchDB" by Damien Katz (removed in 2012 - available in the Web Archive)
Comments to "Why Couchbase?" by Damien Katz (removed in 2012 - available in the Web Archive)
Couchbase 2011 Year in Review
Membase Server is Now Couchbase Server
Couchbase technical update
Difference between Cloudant and CouchOne
They are different yet similar pieces of software. I've remixed the content from the top answer into a picture that might help clarify the "difference" as well as the common things:
A comment from Matt Ingenthron adds to this:
To add some context/corrections: NorthScale founders are Steve Yen and Dustin Sallings. I joined them shortly after founding. Also, Damien didn't later join Couchbase, he was part of CouchIO/Couch One prior to the merger. Citing a fun, historical source: https://youtube.com/watch?v=aZ_JOnU8tkI
I think CouchBase seem to be perceived as CouchDB's 'enterprise' alternative. Which in a way seem to be true.
Apart from lack of ability to attach files to records ( documents) and 'out-of-box' REST endpoints compared to CouchDB, CouchBase has sql like language i.e. N1QL (sometimes pronounced a Nickel, UPDATE renamed to SQL++ in Couchbase 7.0).
This is one of the reason why I don't really like / recommend using the term 'NoSQL'. I personally like term 'Non-relational'.
We have a small 3 developer team that is currently using Subversion for our source control. We expect the team to group to 8 members within the next 6 to 12 months. We are considering changing our source control to either TFS or Mercurial for improved branching. I know TFS is overkill for just branching, but that is the immediate need, and the other features of TFS could aid our team. One of our main concerns with TFS is we've heard that there is a lot of overhead deploying it, especially on a small team. I'm hoping to get some community insight into just how much overhead there may be involved, suggestions to make the process easier, and anything else the community may feel is useful in making the decision to implement.
In my experience, TFS works really well, even for small teams. If your total number of developers is five or less, you can use the relatively affordable Workgroup edition: above that, you'll have to pony up for the real thing, pricing for which is definitely in the 'Enterprise' realm...
The biggest hurdle to starting to use TFS is installing the darn thing: this process seems to be designed for maximum aggravation. (The extent to which the 'designers' of the 2005-to-2008 upgrade 'process' despise their users even manages to go beyond that: fortunately, you'll be able to start with TFS2008 and won't have to worry about upgrading for a while).
If you follow the instructions exactly, you should manage in 2-3 tries, though, and the hardware requirements aren't as bad as they seem. My 3-developer TFS setup runs quite comfortable on a previous-generation Dell laptop with 4GB of RAM.
One of the big advantages of TFS is the VS integration: this works just really, really well, and shelving and branching are implemented in a more straightforward way than with any other systems I've seen.
The process guidance and support in TFS are a bit less polished, but still quite usable. The pluggable support for several development methodologies is quite nice, and several third-party add-ons (for example for Scrum) are available already.
All in all, it definitely won't hurt to try TFS: if you have a MSDN subscription, you probably already have the Workgroup edition as well as a trial of the full version: otherwise, you can downloaded the latter from Microsoft as well.
UPDATE, April 12th, 2010: With the release of Team Foundation Server 2010, the installation and upgrade procedures have improved a lot. A new TFS2010 install shouldn't take you more than a few minutes (assuming you already have an instance of SQL Server 2008 up and running) and even an in-place upgrade of my TFS2008 setup proved to be entirely painless.
Setup of TFS is not too complicated, when you exactly follow the given guide step by step. We are using it on a small team for about one year now and i don't want to miss it any more.
Especially when you use more than one part of tfs like version control and work item tracking and maybe even teambuild, your team will benefit of the tight integration of the seperate parts.
For example, you can link to workitems when checking in code changes.
Then you run an automated build with teambuild and it will automatically update your workitems with the build number.
So afterwards you can see for example in a bug workitem the buildnumber which contains the bugfix.
We also use the sharepoint wiki for documentation and planning although i'm not the biggest sharepoint fan...
The main point is the great integration into the IDE and for workitem tracking the Teamsystem Web Access which allows you to control at least your workitems over a webinterface.
It's been awhile, but I'm thinking that it takes about a half-day to get setup, plus some time reading the manuals beforehand to make sure you know what you're doing. Configuration doesn't take too long -- you need to add all of your developers in as licensed users. Setting up projects is not too hard. I usually set up AD groups to map on the project roles and add those groups to the appropriate roles. I set up a new project in about 1/2 hour.
Note: I don't use any of the features of TFS except source control. If you plan to item tracking, use the project sharepoint site, etc., your mileage will vary quite a bit. I've found that on our projects (2-3 developers) a wiki works just as well for project management.