I create a migration to to convert a jsonb column into a one-to-many table.
-- upgrade
insert into device_component (
warranty_request_uuid,
serial_number,
component_type,
description
)
select
warranty_request.uuid,
value->>'serial_number',
value->>'type',
value->>'description'
from
warranty_request,
jsonb_array_elements(warranty_request.device_components)
I need to provide a corresponding downgrade statement. To revert the migration, I am trying something like the below.
-- downgrade
update
warranty_request
set
device_components = jsonb_set(
device_components,
'{}',
jsonb_build_object(
'serial_number', device_component.serial_number,
'type', device_component.component_type,
'description', device_component.description
)
)
from
device_component
where
warranty_request.uuid = device_component.warranty_request_uuid
The problem is that the device_components column contains only null values after downgrading. So nothing is inserted.
How, should the downgrade statement look like to make this work?
I want to upgrade and downgrade between these 2 formats.
Upgrade:
# warranty_request
uuid
-----
abc
# device_compoent
uuid | warranty_request_uuid | serial_number | device_type | description
-----|-----------------------|---------------|-------------|------------
efg | abc | 1 | foo | bar
hij | abc | 2 | foo | bar
Downgrade:
# warranty_request
uuid | device_components
------|----------------------------------------------------------------------------------------------------------------------
abc | [{"serial_number": 1, "type": "foo", "description": bar}, {"serial_number": 2, "type": "foo", "description": bar}]
You can try this :
UPDATE
warranty_request AS w
SET
device_components = a.device_components
FROM
( SELECT d.warranty_request_uuid
, jsonb_agg (jsonb_build_object
( 'serial_number', d.serial_number
, 'type', d.device_type
, 'description', d.description
)
) AS device_components
FROM device_component AS d
GROUP BY d.warranty_request_uuid
) AS a
WHERE w.uuid = a.warranty_request_uuid
see the demo result in dbfiddle.
Related
I have a source table, which lists like below:
public.source
Id | part_no | category
1 | 01270-4 | Landscape
2 | 01102-3 | Sports
Then, I have target table with jsonb column (combinations) , which list like below;
public.target
Id | part_no | combinations
7 | 01270-4 | {"subject":""}
8 | 01102-3 | {"subject":""}
My problem is - how I can update the target table with jsonb column (combinations) with the values coming from source table using the part_no column?
Output like:
Id | part_no | combinations
7 | 01270-4 | {"subject":"Landscape"}
8 | 01102-3 | {"subject":"Sports"}
I tried below but giving error:
UPDATE public.target t
SET combinations = jsonb_set(combinations,'{subject}','s.category',false)
FROM public.source s
WHERE s.part_no = t.part_no;
ERROR: invalid input syntax for type json
LINE 2: SET combinations = jsonb_set(combinations,'{subject}', 's.categor...
^
DETAIL: Token "s" is invalid.
CONTEXT: JSON data, line 1: s...
SQL state: 22P02
Character: 77
You should use to_jsonb function to convert s.category to JSON
Demo
UPDATE public.target t
SET combinations = jsonb_set(combinations,'{subject}',to_jsonb(s.category),false)
FROM public.source s
WHERE s.part_no = t.part_no
Or you can use sample structure for join and update two JSON field:
Demo
UPDATE public.target t
SET combinations = combinations || jsonb_build_object('subject', s.category)
FROM public.source s
WHERE s.part_no = t.part_no
Looking to use JSON_VALUE to access some data within a JSON column in an SQL Server table.
I have a table (tblMyTable) that has the following columns and data;
| Column1 | JSONColumn |
| -------- | -------------- |
| 1 | {some JSON} |
Where {some JSON} looks as follows;
[
{
"RoleName":"Client",
"Roles":[
{
"ContactID":21568,
"ContactName":"FullName1"
},
{
"ContactID":31568,
"ContactName":"FullName2"
}
]
},
{
"RoleName":"Owner",
"Roles":[
{
"ContactID":1,
"ContactName":"Billy Buxton"
}
]
}
]
I am wanting to use something like this to access the ContactName of Billy Buxton;
SELECT JSON_VALUE(JSONColumn, '$[Owner].Roles[0].ContactName') AS ContactName
FROM tblMyTable
WHERE Column1=1
RESULT:
| ContactName |
| ----------- |
| Billy Buxton|
The issue is that I don't know where the Owner RoleName is in the JSON so I can't use;
'$[0].Roles[0].ContactName' as it might be '$[1].Roles' or '$[10].Roles'.
Is there a way to do this by specifying the 'Owner' or 'Client' RoleName?
In addition, if I use the following I would get the array of ContactNames;
SELECT JSON_QUERY(JSONColumn, '$[Client].Roles.ContactName') AS ContactNames
FROM tblMyTable
WHERE Column1=1
RESULT:
| ContactNames |
| ------------------ |
| FullName1,FullName2|
I am able to change the structure of the JSON if needed, and would consider using OpenJSON if that works with specifying a path (eg; $[Owner].Roles[0].ContactName or similar)
I had come up with the following option, but it's using OPENJSON which is much messier than using JSON_VALUE or JSON_QUERY;
SELECT Y.RoleName,STRING_AGG(Z.ContactName,',') AS ContactName
FROM (
SELECT '[{"RoleName":"Client","Roles":[{"ContactID":21568,"ContactName":"FullName1"},{"ContactID":31568,"ContactName":"FullName2"}]},{"RoleName":"Owner","Roles":[{"ContactID":1,"ContactName":"Billy Buxton"}]}]' AS Roles
)X
CROSS APPLY OPENJSON (X.Roles)
WITH (
RoleName VarChar(50),
Roles nVarChar(MAX) AS JSON
)Y
CROSS APPLY OPENJSON (Y.Roles)
WITH (
ContactID BigInt,
ContactName VarChar(255)
)Z
WHERE Y.RoleName='Client'
GROUP BY Y.RoleName
RESULT:
| RoleName | ContactName |
| -------- | ------------------ |
| Client | FullName1,FullName2|
Thanks in advance for any assistance you can provide.
Check if this solve your needs
---------------- DDL+DML
use tempdb
GO
CREATE TABLE tblMyTable (ID INT, JSONColumn NVARCHAR(MAX))
GO
INSERT tblMyTable (ID, JSONColumn)
VALUES (1, ' [
{
"RoleName":"Client",
"Roles":[
{
"ContactID":21568,
"ContactName":"FullName1"
},
{
"ContactID":31568,
"ContactName":"FullName2"
}
]
},
{
"RoleName":"Owner",
"Roles":[
{
"ContactID":1,
"ContactName":"Billy Buxton"
}
]
}
]')
GO
---------- Solution
select ID, RoleName, Roles, Soluion = JSON_VALUE(Roles, '$[0].ContactName')
from tblMyTable t
CROSS APPLY OpenJson(t.JSONColumn)
WITH (
RoleName VarChar(50),
Roles nVarChar(MAX) AS JSON
)Y
WHERE RoleName = 'Owner'
GO
Note! I assume that there is only one role for the Owner, which is why I can use $[0]
Say I have a Postgres table where one of its column has jsonb type to store the Multilanguage value.
Here's the example.
| word |
|----------------------------------------|
| {"en": "Hello", "es": "Hola"} |
| {"it": "Rosso", "es": "Rojo"} |
|{"en": "Sea", "it": "Mare", "es": "Mar"}|
As you can see, the values do not always have all language version. My question is how I can select only the English version or any first value from json if the English version does not exist?
Thank you.
You'll have to unpack the JSON:
SELECT kv.value
FROM atable
CROSS JOIN LATERAL jsonb_each(atable.word) AS kv(key,value)
WHERE atable.id = ...
ORDER BY kv.key <> 'en'
LIMIT 1;
This relies on the fact that FALSE < TRUE.
I have the next table, and data:
/* script for people table, with field tsvector and gin */
CREATE TABLE public.people (
id INTEGER,
name VARCHAR(30),
lastname VARCHAR(30),
complete TSVECTOR
)
WITH (oids = false);
CREATE INDEX idx_complete ON public.people
USING gin (complete);
/* data for people table */
INSERT INTO public.people ("id", "name", "lastname", "complete")
VALUES
(1, 'MICHAEL', 'BRYANT BRYANT', '''bryant'':2,3 ''michael'':1'),
(2, 'HENRY STEVEN', 'BUSH TIESSEN', '''bush'':3 ''henri'':1 ''steven'':2 ''tiessen'':4'),
(3, 'WILLINGTON STEVEN', 'STEPHENS FLINN', '''flinn'':4 ''stephen'':3 ''steven'':2 ''willington'':1'),
(4, 'BRET', 'MARTINEZ AROCH', '''aroch'':3 ''bret'':1 ''martinez'':2'),
(5, 'TERENCE BERT', 'CAVALIERE ENRON', '''bert'':2 ''cavalier'':3 ''terenc'':1');
I need retrieve the names and lastnames, according the tsvector field. Actually I have the query:
SELECT * FROM people WHERE complete ## to_tsquery('WILLINGTON & FLINN');
And the result is right (the third record). BUT if I try with
SELECT * FROM people WHERE complete ## to_tsquery('STEVEN & FLINN');
/* the same record! */
I don't have results. Why? What can I do?
You should use the same language to search your table as the values in your field 'complete' where inserted.
Check the result of that query compared english and german:
select * ,
to_tsvector('english', concat_ws(' ', name, lastname )) as english,
to_tsvector('german', concat_ws(' ', name, lastname )) as german
from public.people
so that should work for you :
SELECT * FROM people WHERE complete ## to_tsquery('english','STEVEN & FLINN');
You are probably using a text search configuration where either STEVEN or FLINN are modified by stemming.
I can reproduce this here:
test=> SHOW default_text_search_config;
default_text_search_config
----------------------------
pg_catalog.german
(1 row)
test=> SELECT complete FROM public.people WHERE id = 3;
complete
-------------------------------------------------
'flinn':4 'stephen':3 'steven':2 'willington':1
(1 row)
test=> SELECT * FROM ts_debug('STEVEN & FLINN');
alias | description | token | dictionaries | dictionary | lexemes
-----------+-----------------+--------+---------------+-------------+---------
asciiword | Word, all ASCII | STEVEN | {german_stem} | german_stem | {stev}
blank | Space symbols | | {} | |
blank | Space symbols | & | {} | |
asciiword | Word, all ASCII | FLINN | {german_stem} | german_stem | {flinn}
(4 rows)
test=> SELECT * FROM public.people
WHERE complete ## to_tsquery('STEVEN & FLINN');
id | name | lastname | complete
----+------+----------+----------
(0 rows)
So you see, the German Snowball dictionary stems STEVEN to stev.
Since complete contains the unstemmed version steven, no match is found.
You should use the same text search configuration when you populate complete and in the query.
Introduction
Considering the tables given at the end of this question, I would like an algorithm or a simple solution that returns a nested tree from a YAML description. Using yaml format is an optional need. In fact, the output I need is an array of ordered hashes that may or may not contain nested ordered hashes or arrays of ordered hashes.
In short, I am talking about a tree-like structure.
For a better understanding of my question I will treat a simple example that covers all my needs. Actually this example is the one I am using to implement this algorithm.
I decided to ask this question in parallel with my own investigations as my knowledge in Perl is limited. I don't want to dig into the wrong tunnel and that's why I am asking for help.
I am currently focussing on the DBI module. I tried to look at other modules such as DBIx::Tree::NestedSet, but I don't think it is what I need.
So, let's get down to the details of my example.
Example
The inital idea is to write a perl program that takes a yaml description and outputs the extracted data.
This input description follows simple rules:
query is what data we are looking for. It can contains the following keys
sql is the SQL query
hide hides columns from the final output. This field is used when a column is required only in a subquery but not wanted in the end.
subquery is a nested query executed for each row of the parent query
bind to bind columns values to the sql query
hash tells the program to group the results not in an array of hashes but an hash of hashes. Actually this could be directly given to DBI::selectall_hashref. If this field is omitted the output is listed as an array of ordered hashes.
key is the name of the key listed at the same level of the parent's result. We will see
later that a key name can mask a result column.
list tells the program to list the result into an array. Notice that only one column can be displayed i.e. array: name displays the list of names
connect is the DBI connection string
format is the output format. It can be either XML, YAML or JSON. I primarly focus on the
YAML format as it can be easily translated. When omitted, the default ouput is YAML.
indent how many spaces is one identation. The tabs or tab value is also supported.
In addition, we know that in Perl hashes are not ordered. Here, the output keys order is important and should appear as they appear in the sql query.
From this I simpy use the YAML module :(
In summary, in the end we will just execute this command:
$ cat desc.yml | ./fetch > data.yml
The desc.yml description is given below:
---
connect: "dbi:SQLite:dbname=einstein-puzzle.sqlite"
ident: 4
query:
- sql: SELECT * from people
hide:
- pet_id
- house_id
- id
subquery:
- key: brevage
bind: id
sql: |
SELECT name, calories, potassium FROM drink
LEFT JOIN people_has_drink ON drink.id = people_has_drink.id_drink
WHERE people_has_drink.id_people = 1
hash:
- name
- key: house
sql: SELECT color as paint, size, id from house WHERE id = ?
hide: id
bind: paint
subquery:
- key: color
sql: SELECT name, ral, hex from color WHERE short LIKE ?
bind: color
- key: pet
sql: SELECT name from pet WHERE id = ?
bind: pet_id
list: name
Expected Output
From the description above, the output data would be this:
---
- nationality: Norvegian
smoke: Dunhill
brevage:
orange juice:
calories: 45
potassium: 200 mg
water:
calories: 0
potassium: 3 mg
house:
color:
name: Zinc yellow
ral: RAL 1018
hex: #F8F32B
paint: yellow
size: small
pet:
- cats
- nationality: Brit
smoke: Pall Mall
brevage:
milk:
calories: 42
potassium: 150 mg
house:
color:
name: Vermilion
ral: RAL 2002
hex: #CB2821
paint: red
size: big
pet:
- birds
- phasmatodea
Where I am
I still did not fully implemented the nested queries. My current sate is given here:
#!/usr/bin/env perl
use 5.010;
use strict;
use warnings;
use DBI;
use YAML;
use Data::Dumper;
use Tie::IxHash;
# Read configuration and databse connection
my %yml = %{ Load(do { local $/; <DATA>}) };
my $dbh = DBI->connect($yml{connect});
# Fill the bind values of the first query with command-line information
my %bind;
for(#ARGV) {
next unless /--(\w+)=(.*)/;
$bind{$1} = $2;
}
my $q0 = $yml{query}[0];
if ($q0->{bind} and keys %bind > 0) {
$q0->{bind_values} = arrayref($q0->{bind});
$q0->{bind_values}[$_] = $bind{$q0->{bind}[$_]} foreach (0 .. #{$q0->{bind}} - 1);
}
# Fetch all data from the database recursively
my $out = fetch($q0);
sub fetch {
# As long we have a query, one processes it
my $query = shift;
return undef unless $query;
$query->{bind_values} = [] unless ref $query->{bind_values} eq 'ARRAY';
# Execute SQL query
my $sth = $dbh->prepare($query->{sql});
$sth->execute(#{$query->{bind_values}});
my #columns = #{$sth->{NAME}};
# Fetch all the current level's data and preserve columns order
my #return;
for my $row (#{$sth->fetchall_arrayref()}) {
my %data;
tie %data, 'Tie::IxHash';
$data{$columns[$_]} = $row->[$_] for (0 .. $#columns);
for my $subquery (#{ $query->{subquery} }) {
my #bind;
push #bind, $data{$_} for (#{ arrayref($subquery->{bind}) });
$subquery->{bind_values} = \#bind;
my $sub = fetch($subquery);
# Present output as a list
if ($subquery->{list}) {
#if ( map ( $query->{list} eq $_ , keys $sub ) )
my #list;
for (#$sub) {
push #list, $_->{$subquery->{list}};
}
$sub = \#list;
}
if ($subquery->{key}) {
$data{$subquery->{key}} = $sub;
} else {
die "[Error] Key is missing for query '$subquery->{sql}'";
}
}
# Remove unwanted columns from the output
if ($query->{hide}) {
delete $data{$_} for( #{ arrayref($query->{hide}) } );
}
push #return, \%data;
}
\#return;
}
DumpYaml($out);
sub arrayref {
my $ref = shift;
return (ref $ref ne 'ARRAY') ? [$ref] : $ref;
}
sub DumpYaml {
# I am not happy with this current dumper. I cannot specify the indent and it does
# not preserve the extraction order
print Dump shift;
}
__DATA__
---
connect: "dbi:SQLite:dbname=einstein-puzzle.sqlite"
ident: 4
query:
- sql: SELECT * from people
hide:
- pet_id
- house_id
- id
subquery:
- key: brevage
bind: id
sql: |
SELECT name, calories, potassium FROM drink
LEFT JOIN people_has_drink ON drink.id = people_has_drink.id_drink
WHERE people_has_drink.id_people = ?
hash:
- name
- key: house
sql: SELECT color as paint, size, id from house WHERE id = ?
hide: id
bind: house_id
subquery:
- key: color
sql: SELECT short, ral, hex from color WHERE short LIKE ?
bind: paint
- key: pet
sql: SELECT name from pet WHERE id = ?
bind: pet_id
list: name
And this is what output I get:
---
- brevage:
- calories: 0
name: water
potassium: 3 mg
- calories: 45
name: orange juice
potassium: 200 mg
house:
- color:
- hex: '#F8F32B'
ral: RAL 1018
short: yellow
paint: yellow
size: small
nationality: Norvegian
pet:
- cats
smoke: Dunhill
- brevage:
- calories: 42
name: milk
potassium: 150 mg
house:
- color:
- hex: '#CB2821'
ral: RAL 2002
short: red
paint: red
size: big
nationality: Brit
pet:
- birds
- phasmatodea
smoke: Pall Mall
Database
My test databse is a sqlite db where the tables are listed below:
Table People
.----+-------------+----------+--------+-----------.
| id | nationality | house_id | pet_id | smoke |
+----+-------------+----------+--------+-----------+
| 1 | Norvegian | 4 | 3 | Dunhill |
| 2 | Brit | 1 | 2 | Pall Mall |
'----+-------------+----------+--------+-----------'
Table Drink
.----+--------------+----------+-----------.
| id | name | calories | potassium |
+----+--------------+----------+-----------+
| 1 | tea | 1 | 18 mg |
| 2 | coffee | 0 | 49 mg |
| 3 | milk | 42 | 150 mg |
| 4 | beer | 43 | 27 mg |
| 5 | water | 0 | 3 mg |
| 6 | orange juice | 45 | 200 mg |
'----+--------------+----------+-----------'
Table People Has Drink
.-----------+----------.
| id_people | id_drink |
+-----------+----------+
| 1 | 5 |
| 1 | 6 |
| 2 | 3 |
'-----------+----------'
Table House
+----+--------+--------+
| id | color | size |
+----+--------+--------+
| 1 | red | big |
| 2 | green | small |
| 3 | white | middle |
| 4 | yellow | small |
| 5 | blue | huge |
+----+--------+--------+
Table Color
.--------+-------------+----------+---------.
| short | color | ral | hex |
+--------+-------------+----------+---------+
| red | Vermilion | RAL 2002 | #CB2821 |
| green | Pale green | RAL 6021 | #89AC76 |
| white | Light grey | RAL 7035 | #D7D7D7 |
| yellow | Zinc yellow | RAL 1018 | #F8F32B |
| blue | Capri blue | RAL 5019 | #1B5583 |
'--------+-------------+----------+---------'
Table Pet
+----+-------------+
| id | name |
+----+-------------+
| 1 | dogs |
| 2 | birds |
| 3 | cats |
| 4 | horses |
| 5 | fishes |
| 2 | phasmatodea |
+----+-------------+
Database data
If you wish use the same data as mine also give you all what you need:
BEGIN TRANSACTION;
CREATE TABLE "pet" (
`id` INTEGER,
`name` TEXT
);
INSERT INTO `pet` VALUES (1,'dogs');
INSERT INTO `pet` VALUES (2,'birds');
INSERT INTO `pet` VALUES (3,'cats');
INSERT INTO `pet` VALUES (4,'horses');
INSERT INTO `pet` VALUES (5,'fishes');
INSERT INTO `pet` VALUES (2,'phasmatodea');
CREATE TABLE `people_has_drink` (
`id_people` INTEGER NOT NULL,
`id_drink` INTEGER NOT NULL,
PRIMARY KEY(id_people,id_drink)
);
INSERT INTO `people_has_drink` VALUES (1,5);
INSERT INTO `people_has_drink` VALUES (1,6);
INSERT INTO `people_has_drink` VALUES (2,3);
CREATE TABLE "people" (
`id` INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
`nationality` VARCHAR(45),
`house_id` INT,
`pet_id` INT,
`smoke` VARCHAR(45)
);
INSERT INTO `people` VALUES (1,'Norvegian',4,3,'Dunhill');
INSERT INTO `people` VALUES (2,'Brit',1,2,'Pall Mall');
CREATE TABLE "house" (
`id` INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
`color` TEXT,
`size` TEXT
);
INSERT INTO `house` VALUES (1,'red','big');
INSERT INTO `house` VALUES (2,'green','small');
INSERT INTO `house` VALUES (3,'white','middle');
INSERT INTO `house` VALUES (4,'yellow','small');
INSERT INTO `house` VALUES (5,'blue','huge');
CREATE TABLE `drink` (
`id` INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
`name` TEXT,
`calories` INTEGER,
`potassium` TEXT
);
INSERT INTO `drink` VALUES (1,'tea',1,'18 mg');
INSERT INTO `drink` VALUES (2,'coffee',0,'49 mg');
INSERT INTO `drink` VALUES (3,'milk',42,'150 mg');
INSERT INTO `drink` VALUES (4,'beer',43,'27 mg');
INSERT INTO `drink` VALUES (5,'water',0,'3 mg');
INSERT INTO `drink` VALUES (6,'orange juice',45,'200 mg');
CREATE TABLE `color` (
`short` TEXT UNIQUE,
`color` TEXT,
`ral` TEXT,
`hex` TEXT,
PRIMARY KEY(short)
);
INSERT INTO `color` VALUES ('red','Vermilion','RAL 2002','#CB2821');
INSERT INTO `color` VALUES ('green','Pale green','RAL 6021','#89AC76');
INSERT INTO `color` VALUES ('white','Light grey','RAL 7035','#D7D7D7');
INSERT INTO `color` VALUES ('yellow','Zinc yellow','RAL 1018','#F8F32B');
INSERT INTO `color` VALUES ('blue','Capri blue','RAL 5019','#1B5583');
COMMIT;
Is my implementation good
This is a rather broad question, and the answer probably depends on what you want from your code. For instance:
Does it work? Does it have all the features you need? Does it do what you want? Does it respond appropriately for all the ranges of inputs you want to cater for (and input you don't)? If you aren't sure, write some tests.
Is it fast enough? If not, what are the slow bits? Use Devel::NYTProf to find them.
If it's working, you probably also want to turn your code into a module rather than just a script so you can use it again.
and if not (I'm supposing that I am doing all wrong), what modules should I use to get the desired behavior?
It sounds very much like you're trying to do something like DBIx::Class (aka DBIC) does when you ask it to prefetch; it will build you a data structure of objects.
If you need to do this dynamically in response to arbitrary databases and YAML, that's not quite what DBIC was designed to do; it's probably possible but will probably involve you dynamically creating packages, which will not be easy.