Weird ghost records in PostgreSQL - what are they? - postgresql

I have a very weird issue on our postgresql DB. I have a table called "statement" which has some strange records in it.
Using the command line console psql, I query select * from customer.statement where type in ('QUOTE'); and get 12 rows back. 7 rows look normal, 5 are missing all data except a single column which is a nullable column but seems to hold real values entered by the user. psql tells me that 7 rows were returned even though there are 12. Most of the other columns are not nullable. The weird records look like this:
select * from customer.statement where type = 'QUOTE';
id | issuer_id | recipient_id | recipient_name | recipient_reference | source_statement_id | catalogue_id | reference | issue_date | due_date | description | total | currency | type | tax_level | rounding_mode | status | recall_requested | time_created | time_updated | time_paid
------------------+------------------+------------------+----------------+---------------------+---------------------+--------------+-----------+------------+------------+------------------------------------------------------------------+-----------+----------+-------+-----------+---------------+-----------+------------------+----------------------------+----------------------------+-----------
... 7 valid records removed ...
| | | | | | | | | | Build bulkheads and sheet with plasterboard. +| | | | | | | | | |
| | | | | | | | | | Patch all patches. +| | | | | | | | | |
| | | | | | | | | | Set and sand all joints ready for painting. +| | | | | | | | | |
| | | | | | | | | | Use wall angle on bulkhead in main bedroom. +| | | | | | | | | |
| | | | | | | | | | Build nib and sheet and set in entrance | | | | | | | | | |
(7 rows)
If I run the same query using pgAdmin, I don't see those weird records.
Anyone know what these are?

The plus sign before the separator (+|) indicates a newline character in the displayed string value in psql. So no additional rows, just the same row continued with line breaks. The final line of output in your quote confirms as much: (7 rows).
In pgAdmin you don't see the extra lines as long as you don't increase the height of the field (or copy / paste the content somewhere), but there are multiple lines as well.
Try in psql and in pgAdmin:
test=# SELECT E'This\nis\na\ntest.' AS multi_line, 'foo' AS single_line;
multi_line | single_line
--------------+-------------
This +| foo
is +|
a +|
test. |
(1 row)
The manual about psql:
linestyle
Sets the border line drawing style to one of ascii, old-ascii, or unicode. [...] The default setting is ascii. [...]
ascii style uses plain ASCII characters. Newlines in data are shown using a + symbol in the right-hand margin. [...]

Related

T-SQL : Pivot table without aggregate

I am trying to understand how to pivot data within T-SQL but can't seem to get it working. I have the following table structure
+-------------------+-----------------------+
| Name | Value |
+-------------------+-----------------------+
| TaskId | 12417 |
| TaskUid | XX00044497 |
| TaskDefId | 23 |
| TaskStatusId | 4 |
| Notes | |
| TaskActivityIndex | 0 |
| ModifiedBy | Orange |
| Modified | /Date(1554540200000)/ |
| CreatedBy | Apple |
| Created | /Date(2121212100000)/ |
| TaskPriorityId | 40 |
| OId | 2 |
+-------------------+-----------------------+
I want to pivot the name column to be columns expected output
+--------+------------------------+-----------+--------------+-------+-------------------+------------+-----------------------+-----------+-----------------------+----------------+-----+
| TASKID | TASKUID | TASKDEFID | TASKSTATUSID | NOTES | TASKACTIVITYINDEX | MODIFIEDBY | MODIFIED | CREATEDBY | CREATED | TASKPRIORITYID | OID |
+--------+------------------------+-----------+--------------+-------+-------------------+------------+-----------------------+-----------+-----------------------+----------------+-----+
| | | | | | | | | | | | |
| 12417 | XX00044497 | 23 | 4 | | 0 | Orange | /Date(1554540200000)/ | Apple | /Date(2121212100000)/ | 40 | 2 |
+--------+------------------------+-----------+--------------+-------+-------------------+------------+-----------------------+-----------+-----------------------+----------------+-----+
Is there an easy way of doing it? The columns are fixed (not dynamic).
Any help appreciated
Try this:
select * from yourtable
pivot
(
min(value)
for Name in ([TaskID],[TaskUID],[TaskDefID]......)
) as pivotable
You can also use case statements.
You must use the aggregate function in the pivot table.
If you want to learn more, here is the reference:
https://learn.microsoft.com/en-us/sql/t-sql/queries/from-using-pivot-and-unpivot?view=sql-server-2017
Output (I only tried three columns):
DB<>Fiddle

in postgresql how to get the last 4 numbers from a field and copy it to a new field

I'm trying to get the last four digits of the field "SERIAL8" and put that in a new field called "SS4". Here is the query I'm trying to use but it isn't working. I'm new at this, so any help would be appreciated
SELECT * FROM CUSTOMER_TABLE
SUBSTRING (SERIAL,4,4) as 'SS4'
CUSTOMER_TABLE
+-----------------------+------------+----------+--+
| "Complaint Full Date" | Source | SERIAL | |
+-----------------------+------------+----------+--+
| 02/04/16 | DAPIS_CAIR | DG540732 | |
| 04/18/16 | DAPIS_CAIR | DG553384 | |
| 03/23/17 | RO | DG559515 | |
| 03/29/16 | CAIR | DG559781 | |
| 12/10/14 | DAPIS_CAIR | DG561621 | |
+-----------------------+------------+----------+--+

Redshift: tables info query not working via spark

I am trying to run this query from spark code using databricks:
select * from svv_table_info
but I am getting this error msg:
Exception in thread "main" java.sql.SQLException: Amazon Invalid operation: Specified types or functions (one per INFO message) not supported on Redshift tables.;
any opinion why I am getting this?
That view returns table_id which is in the Postgres system type OID.
psql=# \d+ svv_table_info
Column | Type | Modifiers | Storage | Description
---------------+---------------+-----------+----------+-------------
database | text | | extended |
schema | text | | extended |
table_id | oid | | plain |
table | text | | extended |
encoded | text | | extended |
diststyle | text | | extended |
sortkey1 | text | | extended |
max_varchar | integer | | plain |
sortkey1_enc | character(32) | | extended |
sortkey_num | integer | | plain |
size | bigint | | plain |
pct_used | numeric(10,4) | | main |
empty | bigint | | plain |
unsorted | numeric(5,2) | | main |
stats_off | numeric(5,2) | | main |
tbl_rows | numeric(38,0) | | main |
skew_sortkey1 | numeric(19,2) | | main |
skew_rows | numeric(19,2) | | main |
You can cast it to INTEGER and Spark should be able to handle it.
SELECT database,schema,table_id::INT
,"table",encoded,diststyle,sortkey1
,max_varchar,sortkey1_enc,sortkey_num
,size,pct_used,empty,unsorted,stats_off
,tbl_rows,skew_sortkey1,skew_rows
FROM svv_table_info;

psycopg2 is mixing column names with values depending on the table selected

I query a whole postgres table using
c.execute("select * from train_temp")
trans=np.array(c.fetchall())
and amid the expected data I got one row with the column names.
trans[-1,]
Out[63]:
array(['ACTION', 'RESOURCE', 'MGR_ID', 'ROLE_ROLLUP_1', 'ROLE_ROLLUP_2',
'ROLE_DEPTNAME', 'ROLE_TITLE', 'ROLE_FAMILY_DESC', 'ROLE_FAMILY',
'ROLE_CODE', None, None, None, None, None, None, None, None, None], dtype=object)
More puzzling is the fact the the number of rows returned match the number of row in the table
trans.shape
Out[67]: (32770, 19)
select count(1) from train_temp ;
count
-------
32770
(1 row)
Here's the schema of the table
Table "public.train_temp"
Column | Type | Modifiers | Storage | Description
---------------------+------------------+-----------+----------+-------------
action | text | | extended |
resource | text | | extended |
mgr_id | text | | extended |
role_rollup_1 | text | | extended |
role_rollup_2 | text | | extended |
role_deptname | text | | extended |
role_title | text | | extended |
role_family_desc | text | | extended |
role_family | text | | extended |
role_code | text | | extended |
av_role_code | double precision | | plain |
av_role_family | double precision | | plain |
av_role_family_desc | double precision | | plain |
av_role_title | double precision | | plain |
av_role_deptname | double precision | | plain |
av_role_rollup_2 | double precision | | plain |
av_role_rollup_1 | double precision | | plain |
av_mgr_id | double precision | | plain |
av_resource | double precision | | plain |
Has OIDs: no
What's going on here? Note it does not happen with all tables. Actually for this last one the process works fine
Table "public.play"
Column | Type | Modifiers | Storage | Description
-----------+------------------+-----------+----------+-------------
row.names | text | | extended |
action | double precision | | plain |
color | text | | extended |
type | text | | extended |
Has OIDs: no
This last table is completely passed as string, while the problematic one respects the data types.
play[1,]
Out[73]:
array(['2', '0.0', 'blue', 'car'],
dtype='|S5')
trans[1,]
Out[74]:
array(['1', '0', '36', '117961', '118413', '119968', '118321', '117906',
'290919', '118322', 0.920412992041299, 0.942349726775956,
0.933439675174014, 0.920412992041299, 0.976, 0.964478764478764,
0.949222217031812, 0.909090909090909, 0.923076923076923], dtype=object)
Thanks for insight.
Actually I just wrote the headers myself when importing the *csv into postgres.
I should have used the header option in psql such
\copy test from 'test.csv' with (delimiter ',' , format csv, header TRUE);

Using named fields to determine ranges with vsum in Emacs org-table-mode, impossible?

I have been trying to simplify a semi-complex table that I have by adding named fields, without a problem, until I get to the vsum operator. I had the formula set to $M=vsum($3..#-4) which works, however I am continuously having to add and remove items from those fields, which changes the column numbering. This results in me having to change the field specifications of the vsum range after every update/change. I thus tried naming the top field and bottom fields with the thought of supplying the named variables to vsum, giving me a table similar to the following:
| / | <> | <> |
|---+--------+---------|
| | Title1 | Title 2 |
|---+--------+---------|
| _ | | START |
| | name | 1000 |
| | name | 3456 |
| | name | 123 |
| ^ | | END |
|---+--------+---------|
| _ | | MT |
| # | Total | #ERROR |
| # | | |
|---+--------+---------|
#+TBLFM: $MT=vsum($START..$END)
This is the debug formula output from the above table:
Substitution history of formula
Orig: vsum($START..$END)
$xyz-> vsum((1000)..(123))
#r$c-> vsum((1000)..(123))
$1-> vsum((1000)..(123))
-----------^
Error: Expected `)'
I have tried embrasing the named field variables in parenthesis, and several other ways but have thus far not been able to get this to work. I am hoping I am just missing something and being blind, but perhaps this is not possible to do?
I have also tried the sum-up function with no success as well. Thank you in advance for your assistance.
The following solution works by using #II and #III to refer to all entries between the second and third hline.
| / | <> | <> |
|---+--------+---------|
| | Title1 | Title 2 |
|---+--------+---------|
| | name | 1000 |
| | name | 3456 |
| | name | 123 |
|---+--------+---------|
| _ | | MT |
| # | Total | 4579 |
| # | | |
|---+--------+---------|
#+TBLFM: $MT=vsum(#II..#III)
Documentation: http://orgmode.org/manual/References.html#References