I am trying to find solution or function for converting Blob into dataframe.
My usecase is as GCS store data in the blob format. How can I access and convert that GCS blob data into data frame so as to operate on it.
I have Postgres 12.7 dump file (binary format), 300Gb, for simplicity, contains one table.
I want to read the data without installing a server. It would be nice to read the data by converting to csv format on the fly.
Is it possible to read from the dump in any way, preferably with c# or java?
I'm using ADF pipeline to copy data from data lake to blob storage and then from blob storage to table storage.
As you can see below, here are the column types in ADF Data Flow Sink - Blob Storage (integer, string, timestamp):
Here is the Mapping settings in Copy data activity:
On checking the output in table storage, I see all columns are of string type:
Why is table storage saving data in string values? How do I resolve this issue in table storage so that it will accept columns in the right type (integer, string, timestamp)? Please let me know. Thank you!
In usually, when load data from blob storage in Data Factory, all the default data type in blob file are String, Data Factory will help you convert the data type automatically to Sink.
But it also can not meet all our requests.
I tested copy data from Blob to Table Storage and found that: if we don't specify the data type manually in Source, after pipeline executed, all the data type will be String in Sink(Table Storage).
For example, this my Source blob file:
If I don't change the source data type, it seems that everything is ok in Sink table:
But after the pipeline executed, the data type in table storage are all String:
If we change the data type in Source blob manually, and it works ok!
For another question, a little confuse that just from you screenshot, that seems the UI of Mapping Data Flow Sink, but Mapping Data Flow doesn't support Table Storage as Sink.
Hope this helps.
Finally figured out the issue - I was using DelimitedText format for Blob Storage. After converting to Parquet format, I can see data being written to table storage in correct type.
Hi I am trying to create an application which would read a binary file and then depending on the data within the binary file I will have to build a sequence of steps.
I have tried using FlatFileItemReader but I understand that in order to read a Binary File you would have to use SimpleBinaryBufferedReaderFactory.
Can some one please help me on how to read the binary data.
I have a column with geometry data in postgres DB with postGIS enabled.Please help me with a way to encrypt/decrypt that column.
postgresql.org/docs/9.4/static/datatype-geometric.html - the column that i have can contain any of the geometry types. I need a way to encrypt the data while writing to the DB and way to decrypt it while reading the same data back
Thanks in advance
There are several options:
PostgreSQL encryption
This can be done on many levels:
Encryption For Specific Columns
Data Partition Encryption
Encrypting Data Across A Network
etc.
Source
Application-level encryption
Encryption:
application: has geometric data into intermediary format (e.g. JSON)
application: encrypts intermediary format into binary data
application: persists binary data to database
Decryption:
application: reads binary data from database
application: decrypts binary data into intermediary format
application: now has usable geometric data