I would like to compare pipelineDB and PostgreSQL.
Reading the documentation, I found out that pipelineDB is the extension of PostgreSQL.
Then I'm getting curious : What am I have to do with PostgreSQL to compare pipelineDB?
Does system regards Postgres as pipelineDB?
Or, is there any option to switch Postgres and Pipeline extension?
There will be happen nothing. You will use Pipeline functionality via new special functions and database objects. It is great on extensions, so after installing you can use PostgreSQL as usual, and if you want, you can use special functions provided by extension.
Streams are implemented as foreign tables supported Pipelinedb - so this extension has full control over inserting, reading data to/from this object.
Related
PostgreSQL has excellent support for evaluating JSONPath expressions against JSON data.
For example, this query returns true because the value of the nested field is indeed "foo".
select '{"header": {"nested": "foo"}}'::jsonb #? '$.header ? (#.nested == "foo")'
Notably this query does not reference any schemas or tables. Ideally, I would like to use this functionality of PostgreSQL without creating or connecting to a full database instance. Is it possible to run PostgreSQL in such a way that it doesn't have schemas or tables, but is still able to evaluate "standalone" queries?
Some other context on the project, we need to evaluate JSONPath expressions against JSON data in both a Postgres database and Python application. Unfortunately, Python does not have any JSONPath libraries that support enough of the spec to be useful to us.
Ideally, I would like to use this functionality of PostgreSQL without creating or connecting to a full database instance.
Well, it is open source. You can always pull out the source code for this functionality you want and adapt it to compile by itself. But that seems like a large and annoying undertaking, and I probably wouldn't do it. And short of that, no.
Why do you need this? Are you worried about scalability or ease of installation or performance or what? If you are already using PostgreSQL anyway, firing up a dummy connection to just fire some queries at the JSONB engine doesn't seem too hard.
We are using dbt to manage our data models in MSSQL, and we are considering moving to PostgreSQL Citus.
I'm not clear on how I could use Citus's features via dbt? I'm thinking specifically of data compression and columnar storage.
Any ideas?
TBH, I don't have any experiences with dbt.
However, given that dbt docs state that they support PostgreSQL, and Citus Columnar is a table-access-method level abstraction that doesn't require any changes to the queries(*), I wouldn't expect any compatibility issues when combining both.
My guess is that after creating your tables with USING columnar option, then you can just follow dbt docs for the rest.
(*): You might still want to have a look into that page to see what Citus Columnar does or doesn't support atm.
https://github.com/citusdata/citus/tree/master/src/backend/columnar#introduction
was wondering if you had any updates on how this process has gone for you?
I'm looking to implement DBT at my company and was considering postgres/citus as well.
I believe you could use a combination of pre-hook/post-hook dbt functions and materializing incrementally instead of the table or view method.
Thinking is either pre-hook create columnar table and use the dbt incremental update, or materialize as view/table then post-hook SELECT alter_table_set_access_method('table_name', 'columnar');
Is Change tracking feature available in PostgreSQL similar to that of Microsoft SQL Server. Actually we are using PostgreSQL and MS SQL together and want to move changed data from PostgreSQL to MS SQL using change tracking. How to achieve this in best possible and lightweight way?
Yes, there is something like that.
It's called logical decoding and is part of the infrastructure for logical replication. While logical replication can only be used between two Postgres instances, logical decoding can be used independently of that (if you write the code to consume the messages).
Tools like Debezium make use of that
As there is no support for user defined functions or stored procedures in RedShift, how can i achieve UPSERT mechanism in RedShift which is using ParAccel, a PostgreSQL 8.0.2 fork.
Currently, i'm trying to achieve UPSERT mechanism using IF...THEN...ELSE... statement
e.g:-
IF NOT EXISTS(SELECT...WHERE(SELECT..))
THEN INSERT INTO tblABC() SELECT... FROM tblXYZ
ELSE UPDATE tblABC SET.,.,.,. FROM tblXYZ WHERE...
which is giving me error. As i'm writing this code independently without including it in function or SP's.
So, is there any solution to achieve UPSERT.
Thanks
You should probably read this article on upsert by depesz. You can't rely on SERIALIABLE for this since, AFAIK, ParAccel doesn't support full serializability support like in Pg 9.1+. As outlined in that post, you can't really do what you want purely in the DB anyway.
The short version is that even on current PostgreSQL versions that support writable CTEs it's still hard. On an 8.0 based ParAccel, you're pretty much out of luck.
I'd do a staged merge. COPY the new data to a temporary table on the server, LOCK the destination table, then do an UPDATE ... FROM followed by an INSERT INTO ... SELECT. Doing the data uploads in big chunks and locking the table for the upserts is reasonably in keeping with how Redshift is used anyway.
Another approach is to externally co-ordinate the upserts via something local to your application cluster. Have all your tools communicate via an external tool where they take an "insert-intent lock" before doing an insert. You want a distributed locking tool appropriate to your system. If everything's running inside one application server, it might be as simple as a synchronized singleton object.
Any idea how to go about doing that through a tool(preferred). Any alternate ways to do that.
You can check out the migration studio from EnterpriseDB here, although I have no experience with it.
There is no comparison to doing it yourself though - if you're not familiar with Postgres then this will get you familiar, and if you are, then aside from the data entry aspect, this should be old hat.
Use maxdb tools to generate a SQL text export of the database. Then import this file in PostgreSQL, luckily you won't need prior processing of the data dump.