I have a need to encrypt some data in Databricks. I'm currently using the built in aes_encrypt function. If I use the sql shown below, I get a distinct value for each record in the table. I do not get the same value for the encrypted value when the same input is used.
Is there a way to encrypt data in Databricks so the same input yields the same output?
drop table if exists encr;
create table encr as (
select
original_text,
base64(aes_encrypt(original_text,'abcdefabcdefabcdefabcdef')) as encrypted,
cast(aes_decrypt(unbase64(base64(aes_encrypt(original_text,'abcdefabcdefabcdefabcdef'))), 'abcdefabcdefabcdefabcdef') as string) as decrypted
from
my_table
)
;
Results:
select
count(*),
count(distinct original_text),
count(distinct encrypted)
from
encr
;
Setting the mode to 'ECB' gets the same output for the same input:
https://docs.databricks.com/sql/language-manual/functions/aes_encrypt.html
drop table if exists encr;
create table encr as (
select
original_text,
base64(aes_encrypt(original_text,'abcdefabcdefabcdefabcdef','ECB')) as encrypted,
cast(aes_decrypt(unbase64(base64(aes_encrypt(original_text,'abcdefabcdefabcdefabcdef'))), 'abcdefabcdefabcdefabcdef') as string) as decrypted
from
my_table
)
;
Related
I'm new with SAS and I try to copy my Code from Access vba into SAS.
In Access I use often the SELECT INTO funtion, but it seems to me this function is not in SAS.
I have two tables and I get each day new data and I want to update my table with the new lines. Now I Need to check if some new lines appear -> if yes insert this lines into the old table.
I tried some Code from stackoverflow and other stuff from Google, but I didn't find something which works.
INSERT INTO OLD_TABLE T
VALUES (GRVID = VTGONR)
FROM NEW_TABLE V
WHERE not exists (SELECT V.VTGONR FROM NEW_TABLE V WHERE T.GRVID = V.VTGONR);
Not sure what the purpose of using the VALUES keyword is in your example. PROC SQL uses VALUES() to list static values. Like:
VALUES (100)
SAS just uses normal SQL syntax instead. See for example: https://www.techonthenet.com/sql/insert.php
To specify the observations to insert just use SELECT. You can add a WHERE clause as part of the select to limit the rows that you select to insert. To tell INSERT which columns to insert into list them inside () after the table name. Otherwise it will expect the order that the columns are listed in the select statement to match the order of the columns in the target table.
insert into old_table(GRVID)
select VTGONR from new_table
where VTGONR not in (select GRVID from old_table)
;
Guessing this is straight forward but cant get it to run. The issue I am having is explicitly setting column data types in a view.
I need to do this as I will be unioning it to another table and need to match that tables datatypes.
Below is the code I have tried to run(I have tried without the sortkey aswell but still wont run)
DROP VIEW IF EXISTS testing.test_view;
CREATE OR REPLACE VIEW testing.test_view;
(
channel VARCHAR(80) ENCODE zstd,
trans_date TIMESTAMP ENCODE zstd
)
SORTKEY
(
trans_date
)
AS
SELECT channel,
trans_date
from (
SELECT to_date(date,'DD-MM-YYYY') as trans_date,channel
FROM testing.plan
group by date, channel
)
group by trans_date,channel;
The error message I am getting:
An error occurred when executing the SQL command: CREATE OR REPLACE
VIEW trading.trading_squads_plan_v_test ( channel , trans_date )
AS
SELECT channel VARCHAR(80) ENCODE zstd,
trans_date TIM...
Amazon Invalid operation: syntax error at or near "VARCHAR"
Position: 106;
Is this an issue with views where you cant set datatypes? If so is there a workaround?
Thanks
As Jon pointed out my error was trying to set a datatype at the view level, which is not possible as its only pulling this from the table.
So I cast the values in the select call from the table:
DROP VIEW IF EXISTS testing.test_view;
CREATE OR REPLACE VIEW testing.test_view;
(
channel,
trans_date,
source_region
)
AS
SELECT CAST(channel as varchar(80)),
CAST(trans_date as timestamp),
CAST(0 as varchar(80)) as source_region
from (
SELECT to_date(date,'DD-MM-YYYY') as trans_date,channel
FROM testing.plan
group by date, channel
)
group by trans_date,channel;
In the simplified canonical example (which I often see on forums and in the books) of a raster exporting script the OID (returned by lo_create(0), here is 9585208) has to be known before the penultimate script line (lo_export 9585208 'C:/temp/raster.png'):
SELECT oid, lowrite(lo_open(oid, 131072), img) As num_bytes
FROM (
VALUES (
lo_create(0),
(SELECT ST_AsPNG(rast)
FROM bag_o_rasters
LIMIT 1)
)
) As v(oid, img);
lo_export 9585208 'C:/temp/raster.png'
SELECT lo_unlink(9585208);
I have a hard time with figuring out how to make PSQL cli utility to run this script in one pass, i.e., how to communicate OID returned by lo_create(0) to lo_export command. It seems the lo_export is client side command and it is not able to digest any query results... please, help...
Finally, there is a PSQL command \gset which stores values into the script variables, the documentation says:
\gset sends the current query buffer to the server and stores the query's
output into psql variables. The query to be executed must return
exactly one row. Each column of the row is stored into a separate
variable, named the same as the column.
Here is correspondingly modified script:
SELECT lo_create(0) as blob_oid
\gset
SELECT oid, lowrite(lo_open(oid, 131072), img) As num_bytes
FROM (
VALUES (
:blob_oid,
(SELECT ST_AsPNG(rast)
FROM bag_o_rasters
LIMIT 1)
)
) As v(oid,img);
\lo_export :blob_oid 'C:/temp/raster.png'
SELECT lo_unlink(:blob_oid);
Is it possible to do a bulk insert into REdshift using the create table as syntax while defining data type and encoding at the same time? What's the correct syntax?
EG The following gives a syntax error near 'as':
create table my_table (
a int not null encode runlength,
b int not null encode runlength
) distkey(a) sortkey (a, b) as (
select * from other_table
);
I can only get it to work by defining column name only (a or b) and that's a huge limitation...
You can specify the DIST and SORT keys in a CREATE TABLE … AS query like this:
CREATE TABLE new_table
DISTSTYLE KEY
DISTKEY ( my_dist )
SORTKEY ( my_sort )
AS
SELECT *
FROM old_table
;
As per the docs: http://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_TABLE_AS.html I don't believe you can alter the compression encoding from the source table using CREATE TABLE AS.
More details on Redshift CTAS is given here: http://docs.aws.amazon.com/redshift/latest/dg/r_CTAS_usage_notes.html . In a nutshell, no where its mentioned that you can define the encoding in the CTAS statement. But you can define Sort Keys and Hash Keys. The default encoding chosesn by this statement is none.
However if you want to do a bulk insert, why don't you do in two steps.
Create table new_table with your encoding and sort/hash keys
Insert into new_table as select * from old_table
I'm trying to figure out how to insert a .JPG file into a SQL Server 2000 database field of type image using Transact SQL. Thanks.
Use OPENROWSET:
INSERT MyTable (ImageColumnName)
SELECT BulkColumn FROM OPENROWSET (BULK 'c:\myjpeg.jpg', SINGLE_BLOB) AS X
EDITED Whoops, you're using 2000--the previous solution is not supported. You have to use WRITETEXT:
CREATE TABLE MyTable
(
ID INT PRIMARY KEY IDENTITY (1,1),
ImageColumnName IMAGE NULL
)
GO
-- must insert a dummy value into the image column for TEXTPTR
-- to work in next bit
DECLARE #RowId INT
INSERT MyTable (ImageColumnName) VALUES (0xFFFFFFFF)
SELECT #RowId = SCOPE_IDENTITY()
-- get a pointer value to the row+column you want to
-- write the image to
DECLARE #Pointer_Value varbinary(16)
SELECT #Pointer_Value = TEXTPTR(ImageColumnName)
FROM MyTable
WHERE Id = #RowId
-- write the image to the row+column pointer
WRITETEXT MyTable.ImageColumnName #Pointer_Value 'c:\myjpeg.jpg'
There is a tool called textcopy.exe
You can find it under MSSQL\Binn or get it with SQL Server 2000 SP4
Alexander Chigrik wrote a nice stored procedure for usinig it with SQL query:
http://www.mssqlcity.com/Articles/KnowHow/Textcopy.htm
The stored procedure found in this tutorial worked for me:
Brief tutorial on text, ntext, and image