In AWS Athena you can write
SHOW CREATE TABLE my_table_name;
and see a SQL-like query that describes how to build the table's schema. It works for tables whose schema are defined in AWS Glue. This is very useful for creating tables in a regular RDBMS, for loading and exploring data views.
Interacting with Athena in this way is manual, and I would like to automate the process of creating regular RDBMS tables that have the same schema as those in Redshift Spectrum.
How can I do this through a query that can be run via psql? Or is there another way to get this via the aws-cli?
Redshift Spectrum does not support SHOW CREATE TABLE syntax, but there are system tables that can deliver same information. I have to say, it's not as useful as the ready to use sql returned by Athena though.
The tables are
svv_external_schemas - gives you information about glue database mapping and IAM roles bound to it
svv_external_tables - gives you the location information, and also data format and serdes used
svv_external_columns - gives you the column names, types and order information.
Using that data, you could reconstruct the table's DDL.
For example to get the list of columns and their types in the CREATE TABLE format one can do:
select distinct
listagg(columnname || ' ' || external_type, ',\n')
within group ( order by columnnum ) over ()
from svv_external_columns
where tablename = '<YOUR_TABLE_NAME>'
and schemaname = '<YOUR_SCHEM_NAME>'
the query give you the output similar to:
col1 int,
col2 string,
...
*) I am using listagg window function and not the aggregate function, as apparently listagg aggregate function can only be used with user defined tables. Bummer.
I had been doing something similar to #botchniaque's answer in the past, but recently stumbled across a solution in the AWS-Labs' amazon-redshift-utils code package that seems to be more reliable than my hand-spun queries:
amazon-redshift-utils: v_generate_external_tbl_ddl
If you don't have the ability to create a view backed with the ddl listed in that package, you can run it manually by removing the CREATE statement from the start of the query. Assuming you can create it as a view, usage would be:
SELECT ddl
FROM admin.v_generate_external_tbl_ddl
WHERE schemaname = '<external_schema_name>'
-- Optionally include specific table references:
-- AND tablename IN ('<table_name_1>', '<table_name_2>', ..., '<table_name_n>')
ORDER BY tablename, seq
;
They added show external table now.
SHOW EXTERNAL TABLE external_schema.table_name [ PARTITION ]
SHOW EXTERNAL TABLE my_schema.my_table;
https://docs.aws.amazon.com/redshift/latest/dg/r_SHOW_EXTERNAL_TABLE.html
We are building a web app that sits on top of a postgres db. We would like to implement authorization logic inside the database so that it is opaque to the app. For example, suppose a server side controller requests all users from a view v_user. We would like for the db to handle the authorization of which users the currently logged in user can or cannot see. Obviously the server is going to need to send over the login_pkey (user_pkey of logged in user) on every request for this to work.
The issue we are having is with reads. We were able to do this for inserts, updates and deletes by putting the logic in the triggers behind those operations on all views. The issue we are having is how to do this for reads. How can we include variable logic (i.e. logic that depends on which login_pkey is passed) in a view (or some other place) and how can we pass this information for each query.
If it is important, the server we are using is Node and the ORM is Sequelize.
Thanks in advance.
Ideally you really want row security to do this well. It's available in the 9.5 version in beta now.
But you can do what you need without.
To pass a user identity you can use a custom variable, e.g.
SET myapp.appuser = 'fred';
then access it with current_setting e.g.
SELECT current_setting('myapp.appuser')
This will raise an ERROR if the setting does not exist, so you should set a default blank value in postgresql.conf, with ALTER DATABASE SET, etc. Or use PostgreSQL 9.5's current_setting('settingname', true) to return null on missing values.
To filter what users can see, use views that check the user identity setting your app sets at connect-time, per the above.
This is not safe if your users can run arbitrary SQL, because nothing stops them RESETing the setting or doing a SET myapp.appuser = 'the-admin'.
It's very easy to implement this using Pl/Python global dict GD. First, you need to write auth() function:
create or replace function auth(login text, pass text) as $$
-- Check auth login here
GD['user_id'] = get_user_id_by_login(login)
$$ language plpythonu;
Then you have to write get_current_user() function
create or replace function get_current_user() returns integer as $$
return GD['user_id']
$$ langugage plpythonu;
Now, you can get current user any time you want. For example:
-- inside stored procedure
vUserId := get_current_user()
-- in query
select * from some_table where owner_id = get_current_user()
Remember, that GD is stored per session, so, as you wrote, you need to login every time you connect to database. In my ORM I do like this:
class MyORM():
def login(self, user, password):
cursor = self.__conn.cursor()
result = cursor.execute('select core.login(%s, %s)', (user, password,))
data = cursor.fetchone()
cursor.close()
return data[0]
def auth(self, cursor):
cursor.execute('select core.auth(%s)', (g.user_id,))
def query(self, query):
cursor = self.__conn.cursor()
self.auth(cursor)
cursor.execute(query)
data = cursor.fetchall()
cursor.close()
return data
I am logged in as the postgres user and am trying to export a blob from a Postgresql 9.3 database as using this query:
select lo_export(imgs.rast, '/tmp/img.tif') from imgs where rid = 1;
but I get this error:
ERROR: function lo_export(bytea, unknown) does not exist
Do I need to install the lo_export function? If so, how?
You're trying to use a function, which is used to export a large object, which is something different than a blob (bytea column type in Postgres).
There's no library function to export bytea to a file. You just use in your client program something like (Python example):
cursor = conn.execute("select rast from img were rid=%(rid)s", {"rid":1})
result = cursor.fetchone()
with tempfile.NamedTemporaryFile(mode="wb", suffix=".tif", delete=False) as f:
f.write(result[0])
result_filename = f.name
Please remember that this reads the whole data to memory multiple times, so if the data can be fairly large you can prefer to read it in chunks using substring(rast from %(chunk_start)s for %(chunk_size)s). And ensure that the column is saved externally using set storage external.
Looking through the documentation for the Postgres 9.4 datatype JSONB, it is not immediately obvious to me how to do updates on JSONB columns.
Documentation for JSONB types and functions:
http://www.postgresql.org/docs/9.4/static/functions-json.html
http://www.postgresql.org/docs/9.4/static/datatype-json.html
As an examples, I have this basic table structure:
CREATE TABLE test(id serial, data jsonb);
Inserting is easy, as in:
INSERT INTO test(data) values ('{"name": "my-name", "tags": ["tag1", "tag2"]}');
Now, how would I update the 'data' column? This is invalid syntax:
UPDATE test SET data->'name' = 'my-other-name' WHERE id = 1;
Is this documented somewhere obvious that I missed? Thanks.
If you're able to upgrade to Postgresql 9.5, the jsonb_set command is available, as others have mentioned.
In each of the following SQL statements, I've omitted the where clause for brevity; obviously, you'd want to add that back.
Update name:
UPDATE test SET data = jsonb_set(data, '{name}', '"my-other-name"');
Replace the tags (as oppose to adding or removing tags):
UPDATE test SET data = jsonb_set(data, '{tags}', '["tag3", "tag4"]');
Replacing the second tag (0-indexed):
UPDATE test SET data = jsonb_set(data, '{tags,1}', '"tag5"');
Append a tag (this will work as long as there are fewer than 999 tags; changing argument 999 to 1000 or above generates an error. This no longer appears to be the case in Postgres 9.5.3; a much larger index can be used):
UPDATE test SET data = jsonb_set(data, '{tags,999999999}', '"tag6"', true);
Remove the last tag:
UPDATE test SET data = data #- '{tags,-1}'
Complex update (delete the last tag, insert a new tag, and change the name):
UPDATE test SET data = jsonb_set(
jsonb_set(data #- '{tags,-1}', '{tags,999999999}', '"tag3"', true),
'{name}', '"my-other-name"');
It's important to note that in each of these examples, you're not actually updating a single field of the JSON data. Instead, you're creating a temporary, modified version of the data, and assigning that modified version back to the column. In practice, the result should be the same, but keeping this in mind should make complex updates, like the last example, more understandable.
In the complex example, there are three transformations and three temporary versions: First, the last tag is removed. Then, that version is transformed by adding a new tag. Next, the second version is transformed by changing the name field. The value in the data column is replaced with the final version.
Ideally, you don't use JSON documents for structured, regular data that you want to manipulate inside a relational database. Use a normalized relational design instead.
JSON is primarily intended to store whole documents that do not need to be manipulated inside the RDBMS. Related:
JSONB with indexing vs. hstore
Updating a row in Postgres always writes a new version of the whole row. That's the basic principle of Postgres' MVCC model. From a performance perspective, it hardly matters whether you change a single piece of data inside a JSON object or all of it: a new version of the row has to be written.
Thus the advice in the manual:
JSON data is subject to the same concurrency-control considerations as
any other data type when stored in a table. Although storing large
documents is practicable, keep in mind that any update acquires a
row-level lock on the whole row. Consider limiting JSON documents to a
manageable size in order to decrease lock contention among updating
transactions. Ideally, JSON documents should each represent an atomic
datum that business rules dictate cannot reasonably be further
subdivided into smaller datums that could be modified independently.
The gist of it: to modify anything inside a JSON object, you have to assign a modified object to the column. Postgres supplies limited means to build and manipulate json data in addition to its storage capabilities. The arsenal of tools has grown substantially with every new release since version 9.2. But the principal remains: You always have to assign a complete modified object to the column and Postgres always writes a new row version for any update.
Some techniques how to work with the tools of Postgres 9.3 or later:
How do I modify fields inside the new PostgreSQL JSON datatype?
This answer has attracted about as many downvotes as all my other answers on SO together. People don't seem to like the idea: a normalized design is superior for regular data. This excellent blog post by Craig Ringer explains in more detail:
"PostgreSQL anti-patterns: Unnecessary json/hstore dynamic columns"
Another blog post by Laurenz Albe, another official Postgres contributor like Craig and myself:
JSON in PostgreSQL: how to use it right
This is coming in 9.5 in the form of jsonb_set by Andrew Dunstan based on an existing extension jsonbx that does work with 9.4
For those that run into this issue and want a very quick fix (and are stuck on 9.4.5 or earlier), here is a potential solution:
Creation of test table
CREATE TABLE test(id serial, data jsonb);
INSERT INTO test(data) values ('{"name": "my-name", "tags": ["tag1", "tag2"]}');
Update statement to change jsonb value
UPDATE test
SET data = replace(data::TEXT,': "my-name"',': "my-other-name"')::jsonb
WHERE id = 1;
Ultimately, the accepted answer is correct in that you cannot modify an individual piece of a jsonb object (in 9.4.5 or earlier); however, you can cast the jsonb column to a string (::TEXT) and then manipulate the string and cast back to the jsonb form (::jsonb).
There are two important caveats
this will replace all values equaling "my-name" in the json (in the case you have multiple objects with the same value)
this is not as efficient as jsonb_set would be if you are using 9.5
update the 'name' attribute:
UPDATE test SET data=data||'{"name":"my-other-name"}' WHERE id = 1;
and if you wanted to remove for example the 'name' and 'tags' attributes:
UPDATE test SET data=data-'{"name","tags"}'::text[] WHERE id = 1;
This question was asked in the context of postgres 9.4,
however new viewers coming to this question should be aware that in postgres 9.5,
sub-document Create/Update/Delete operations on JSONB fields are natively supported by the database, without the need for extension functions.
See: JSONB modifying operators and functions
I wrote small function for myself that works recursively in Postgres 9.4. I had same problem (good they did solve some of this headache in Postgres 9.5).
Anyway here is the function (I hope it works well for you):
CREATE OR REPLACE FUNCTION jsonb_update(val1 JSONB,val2 JSONB)
RETURNS JSONB AS $$
DECLARE
result JSONB;
v RECORD;
BEGIN
IF jsonb_typeof(val2) = 'null'
THEN
RETURN val1;
END IF;
result = val1;
FOR v IN SELECT key, value FROM jsonb_each(val2) LOOP
IF jsonb_typeof(val2->v.key) = 'object'
THEN
result = result || jsonb_build_object(v.key, jsonb_update(val1->v.key, val2->v.key));
ELSE
result = result || jsonb_build_object(v.key, v.value);
END IF;
END LOOP;
RETURN result;
END;
$$ LANGUAGE plpgsql;
Here is sample use:
select jsonb_update('{"a":{"b":{"c":{"d":5,"dd":6},"cc":1}},"aaa":5}'::jsonb, '{"a":{"b":{"c":{"d":15}}},"aa":9}'::jsonb);
jsonb_update
---------------------------------------------------------------------
{"a": {"b": {"c": {"d": 15, "dd": 6}, "cc": 1}}, "aa": 9, "aaa": 5}
(1 row)
As you can see it analyze deep down and update/add values where needed.
Maybe:
UPDATE test SET data = '"my-other-name"'::json WHERE id = 1;
It worked with my case, where data is a json type
Matheus de Oliveira created handy functions for JSON CRUD operations in postgresql. They can be imported using the \i directive. Notice the jsonb fork of the functions if jsonb if your data type.
9.3 json https://gist.github.com/matheusoliveira/9488951
9.4 jsonb https://gist.github.com/inindev/2219dff96851928c2282
Updating the whole column worked for me:
UPDATE test SET data='{"name": "my-other-name", "tags": ["tag1", "tag2"]}' where id=1;
I have developed an application using postgresql and it works well.
Now I need to create several instances of the same application but I have only one database. So I am thinking about using schemas, so that I can group each instance tables in a different schema.
Now, I wouldn't like to rewrite all my functions and script, thus I am wondering if I can just use some directive to instruct the database to operate on a specific schema. Just to try to make it clearer, do you know when in c++ you do
using namespace std;
so that you can use cout instead of std::cout ? I would like to use someting similar if possible.
The parameter you are looking for is search_path - that lists the schemas a query will look in. So, you can do something like:
CREATE TABLE schema1.tt ...
CREATE TABLE schema2.tt ...
CREATE FUNCTION schema1.foo() ...
CREATE FUNCTION schema2.foo() ...
SET search_path = schema1, something_else;
SELECT * FROM tt; -- schema1.tt
SELECT * FROM schema2.tt -- schema2.tt
SELECT foo(); -- calls schema1.foo
SELECT schema2.foo(); -- calls schema2.foo
Note that if a query's plan gets saved inside the body of foo() then you may get an unexpected results. I would recommend you always explicitly list schemas for referenced tables in plpgsql functions if you are using duplicated tables. If not, make sure you have testing in place to check behaviour with a chaning search_path.
Oh - you can explicitly set search_path for a function's body too - see the manual's CREATE FUNCTION reference for details.