I was surprised to find that Cloud Datastream does not support enum data types in source if replicating from PostgreSQL.
Datastream doesn't support replication of columns of the enumerated (ENUM) data type.
As we have quite a few fields created that way, that is not a viable option for us. Is any good workaround for this limitation?
From google issue tracker:
A workaround is to create a generated column which is of type text. DataStream will then sync the text column automatically and happily.
Also:
we're looking into adding support for ENUMs as part of our GA launch of the PostgreSQL source
Related
GCP now offers a lovely feature to replicate postgres tables directly into Bigquery.
I have managed to get this functionality setup pretty simply following the docs, however all of the columns of ENUM data type are not being converted and pushed into Bigquery.
Is there a solution for this?
https://cloud.google.com/datastream-for-bigquery
I need to show metrics in real time but my metrics are stored in a relational database not supported by the datasources listed here https://grafana.com/docs/grafana/latest/http_api/data_source/
Can I somehow provide the JDBC (or other DB driver) to Grafana?
As #danielle clearly mentioned, "There is no direct support for JDBC or ODBC currently. You could get this data in time series form and into Grafana if you are prepared to do some programming.
The simple json data source is a generic backend that could make JDBC/ODBC calls to MapD and then transform the data into the right form for Grafana."
https://github.com/grafana/grafana/issues/8739#issuecomment-312118425
Though this comment is a bit old, i'm pretty sure there is no out of the box way to visualize data using JDBC/ODBC, yet.
One possible approach can make use of:
Grafana can access PostgreSQL
PostgreSQL can transparently display data in other databases as though it was a PostgreSQL table through Foreign Data Wrappers
Doing it this way, you'd use PostgreSQL to act as a gateway to the data. Depending on the table structure, you might also need to create a view in PG to shape the data to match Grafana's requirements for PG data source.
I'm using the Debezium Postgres connector, and I'm finding that the connector doesn't capture columns that are composite types, or arrays of composite types. Is there some configuration involved in enabling this or is this not supported? I'm not seeing the definition of the composite type in the schema portion of the debezium message nor the composite type itself in the payload. Thanks.
Custom types and thus composite types are not supported yet. There's a feature request about enum types already, perhaps you can file one for composite types.
If you feel adventureous, you could try and use the include.unknown.datatypes connector option, which will propagate the raw representation as the connector obtains it from the WAL. But you should keep in mind that consumers deal with a PG-internal representation that way which might change down the road.
I have a database currently in Mongo running on an EC2 instance and would like to migrate the data to DynamoDB. Is this possible and what is the most cost effective way to achieve this?
When you ask for a "cost effective way" to migrate data, I assume you are looking for existing technologies that can ease your life. If so, you could do the following:
Export your MongoDB data to a text file, say in tsv format, using mongoexport.
Upload that file somewhere in S3.
Import this data, in S3, to DynamoDB using AWS Data Pipeline.
Of course, you should design & finalize your DynamoDB table schema before doing all this.
Whenever you are changing databases, you have to be very careful about the way you migrate data. Certain data formats maintain type consistency, while others do not.
Then there are just data formats that cannot handle your schema. For example, CSV is great at handling data when it is one row per entry, but how do you render an embedded array in CSV? It really isn't possible, JSON is good at this, but JSON has its own problems.
The easiest example of this is JSON and DateTime. JSON does not have a specification for storing DateTime values, they can end up as ISO8601 dates, or perhaps UNIX Epoch Timestamps, or really anything a developer can dream up. What about Longs, Doubles, Ints? JSON doesn't discriminate, it makes them all strings, which can cause loss of precision if not deserialized correctly.
This makes it very important that you choose the appropriate translation medium. The generally means you have to roll your own solution. This means loading up the drivers for both databases, reading an entry from one, translating, and writing to this other. This is the best way to be absolutely sure errors are handled properly for your environment, that types are kept consistently, and that the code properly translates schema from source to destination (if necessary).
What does this all mean for you? It means a lot of leg work for you. It is possible somebody has already rolled something that is broad enough for your case, but I have found in the past that it is best for you to do it yourself.
I know this post is old, Amazon made it possible with AWS DMS, check this document :
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MongoDB.html
Some relevant parts:
Using an Amazon DynamoDB Database as a Target for AWS Database
Migration Service
You can use AWS DMS to migrate data to an Amazon DynamoDB table.
Amazon DynamoDB is a fully managed NoSQL database service that
provides fast and predictable performance with seamless scalability.
AWS DMS supports using a relational database or MongoDB as a source.
I'm trying to understand PostgreSQL and Npgsql in regards to "Full Text Search". Is there something in the Npgsql project that helps doing those searches on a database?
I found the NpgsqlTsVector.cs/NpgsqlTsQuery.cs classes in the Npgsql source code project. Can they be used for "Full Text Search", and, if so, how?
Yes, since 3.0.0 Npgsql has special support for PostgreSQL's full text search types (tsvector and tsquery).
Make sure to read the PostgreSQL docs and understand the two types and how they work.
Npgsql's support for these types means that it allows you to seamlessly send and receive tsvector and tsquery from PostgreSQL. In other words, you can create an instance of NpgsqlTsVector, populate it with the lexemes you want, and then set it as a parameter in an NpgsqlCommand just like any other parameter type (the same goes for reading a tsvector or tsquery).
For more generic help on using Npgsql to interact with PostgreSQL you can read the Npgsql docs.