I'm trying to display hypertable information but can't seem to access the information table.
This request succeeds
select * from _timescaledb_catalog.hypertable;
But this one doesn't, says the table doesn't exist:
select * from timescaledb_information.hypertable;
As expected, creating a hypertable doesn't make any difference.
The former command was found in https://github.com/timescale/timescaledb/issues/648 and I would understand if it was obsolete as the user refers to the 0.10 docs.
The latter comes from the docs: https://docs.timescale.com/latest/api#utilities so it should work.
I'm using Timescale DB 2.2.0 (official Timescale repos) with PostgreSQL 11 (Debian repos).
timescaledb_information.hypertable is old name for this information view. From 2.0 all information views use plural instead of singular in the names. So this information view is renamed to timescaledb_information.hypertables. Its definition was updated too, see it in the docs.
The following query should work in 2.2.0:
select * from timescaledb_information.hypertables;
I also suggest to check the overall changes in 2.0.
Related
I am upserting some data to a Postgres table using jOOQ's insertInto() and onDuplicateKeyUpdate() methods. I want to know later how many duplicates were in my data and hence need to return if a row was inserted or updated.
From my postgres specific research so far, I found RETURNING (not MY_TABLE.xmax = 0) AS updated to be a valid option. However, the auto-generated Java table classes from jOOQ don't seem to give me access to the system columns of postgres like xmax.
Here is my query so far:
dsl.insertInto(MY_TABLE)
.columns(
// pkey columns
MY_TABLE.SHIFT,
MY_TABLE.DATE_UTC,
MY_TABLE.TIME_UTC,
MY_TABLE.DURATION,
)
.values(
shiftId,
utcDateId,
utcTime,
duration
)
.onDuplicateKeyUpdate()
.set(MY_TABLE.DURATION, newDuration)
.returning((MY_TABLE.xmax = 0).`às`("inserted"))
.execute()
This causes the following compile time error:
Error: Kotlin: Unresolved reference: XMAX
I have rechecked my Maven jOOQ table generation configuration and I am not excluding any columns. I have also read through everything I could find on jOOQ's own website but found no useful information for this specific use-case.
Any tips on what I could do here?
In this case you should use jOOQ's SQL templating. Specifically look at the DSL.field() method. Something like this: field("my_table.xmax", int.class).eq(0).
I run into a Cannot enlarge string buffer message on my running postgREST API. I guess some tables are too large to work successful with the API.
I am using the docker postgrest/postgrest container from https://hub.docker.com/r/postgrest/postgrest with the version PostgREST 5.1.0.
Everything is working as expected but if the tables size getting too large, I get following error message.
hint null
details "Cannot enlarge string buffer containing 1073741822 bytes by 1 more bytes."
code "54000"
message "out of memory"
I can't determine the threshold when it's working or not.
Is there a possibility to enlarge the string buffer in some config file or is this hardcoded?
Are there any limits from the table size working with the API. So far I couldn’t find any information in the docu.
=========== Update
The postgres logs give me following SQL query:
WITH pg_source AS (
SELECT "public"."n_osm_bawue_line".*
FROM "public"."n_osm_bawue_line"
)
SELECT null AS total_result_set,
pg_catalog.count(_postgrest_t) AS page_total,
array[]::text[] AS header,
coalesce(json_agg(_postgrest_t), '[]')::character varying AS body
FROM (
SELECT *
FROM pg_source
) _postgrest_t
I use following postgres version:
"PostgreSQL 11.1 (Debian 11.1-1.pgdg90+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516, 64-bit"
Unless you recompile PostgreSQL is not possible to upgrade the limit(defined here).
My suggestion would be to try to reduce the size of the payload(are you sure you need all the data?) or get all of the payload in multiple requests.
With PostgREST you can do vertical filtering(just select the columns that you need) or paginate to reduce the number of rows you get in one request.
The error message comes from PostgreSQL. PostgREST just wraps the message in JSON and sends the HTTP response.
As a first step for finding the problem, look what is the exact HTTP request you do to trigger the error.
Then, enable PostgreSQL logging and repeat the request, check the logs and then you'll see what is the SQL query that causes this error. Run the query through pgAdmin or psql to make sure you got the problematic query.
Update your question with your findings. The SQL query would be what is needed to continue.
After that you could add a postgresql tag to your question.
There is always the possibility that the file being imported is either corrupted or malformed because of any number of reasons.
I just happened to have discovered in my case that my file had something like incorrect line endings (long story, unnecessary here) which caused the whole file to appear as one line, thus causing the obvious result. You may have something similar in your case that requires a find+replace kind of solution.
For whatever benefit to anyone else, I used this to resolve it:
tr -d '\0' < bad_file.csv > bad_file.csv.fixed
I am trying to extend a item profile table by parsering the its part_number column further down into properties. It works fine outside a function.
ALTER TABLE tbl_item_info
ADD prop1 varchar(2),
ADD prop2 varchar(1),
ADD prop3 numeric(4,3);
UPDATE tbl_item_info
SET prop1 = substr(part_num,5,2)
, prop2 = substr(part_num,7,1)
, prop3 = to_number( substr(part_num,8,5) , '9G999')
WHERE ARRAY[left(part_num,3)] <# ARRAY['NTX','EXC'] ;
But when I try to put the statements into a function. It always fail with error "relation with OID xxxxx does not exist" pointing to the UPDATE statements.
I have no clue what it is trying to say. Any idea why ?
I wish I had a definitive answer, but this seems to be related to a known bug in PostgreSQL as described here:
https://github.com/greenplum-db/gpdb/issues/1094
Bear in mind that the greenplum implementation of PostgreSQL is proprietary to Dell EMC, however, the core code issue is likely the same for all major PostgreSQL distributions. I am still researching this to determine if there is a good resolution to the problem. The database in which I experienced a markedly similar error is not the greenplum implementation of PostgreSQL. The error was thrown when I called the pg_relation_filepath() function in a query on an oid that was dynamically obtained from a record in the pg_class table that should have had an associated external file in a subdirectory of the ./base/ path. The error that was thrown was:
ERROR: relation "pg_toast_34474_index" does not exist
The point here is that for a toast entity to exist, it is supposed to be tied to another relation and acts as a reference to additional files created out on the storage media to accommodate additional data that does not fit into the owning relation's top level file - in this case, most likely a table. But when I search for the owning relation's oid (34474), the owner doesn't exist. Since the owner doesn't exist I think the logic assumes that the toast entity doesn't either, even though it has a record in the pg_class table.
This is as close as I can get to a root cause for now. Although the above link suggests code to improve the issue is supposed to have been released in version 8.3, my database has been upgraded from version 8.1 to version 9.4.7, so it appears that even though code may have improved between those two version to prevent new occurrences of the problem, if the problem was created before the database was upgraded, the newer code does not know how to reassemble the tinker toys left behind from issues created by this apparent bug before the fix was implemented.
At present I am investigating if a PLPGSQL function can wrap and trap the error for all relations so I can identify which ones have the problem (as well as to solve my original problem of determining which relation is hosted in a specific file that the server.postmaster log tells me it is unable to read from - hopefully it is just an index I can drop/create).
I found this issue at server 13.7. It was not at server 14.3.
It happened when I changed the signature (parameters) of the stored procedure:
SQL Error [42883]: ERROR: function with OID 894070 does not exist
I removed the old procedure and created new one.
But when I called a function which used that procedure it triggered the error.
To fix it I recreated the function which used changed object.
So general rule:
look where error happens, make sure to recreate object that triggers error, and recompile the code which uses it.
Hope it will help.
i have a DB2 data source and an Oracle 12c target.
The Oracle has a DB link to the DB2 defined which is working in general.
Now i have a huge table in the DB2 which has a timestamp column (lets call it ROW_CHANGED) for row changes. I want to retrieve rows which have changed after a particular time.
Running
SELECT * FROM lib.tbl WHERE ROW_CHANGED >'2016-08-01 10:00:00'
on the DB2 returns exactly 1 row after ca. 90 secs which is fine.
Now i try the same query from the Oracle via the db link:
SELECT * FROM lib.tbl#dblink_name WHERE ROW_CHANGED >TO_TIMESTAMP('2016-08-01 10:00:00')
This runs for hours and ends up in a timeout.
I read some Oracle docs and found distributed query optimization tips but most of them refer to joining a local to a remote table which is not my case.
In my desperation, i have tried the DRIVING_SITE hint, without effect.
Now i wonder when the WHERE part of the query will be evaluated. Since i have to use Oracle syntax and not DB2 syntax for the query, is it possible the Oracle will try to first copy the full table and apply the where clause afterwards? I did some research but did not find anything which would help me in this direction.
The ROW_CHANGED is a hidden column in the DB2, if that matters.
Thx for any hint in advance.
Update
Thanks#all for help. I'll share what did the trick for me.
First of all i have used TO_TIMESTAMP since the DB2 column is also Timestamp (not date) and i had expected to circumvent implicit conversions by this.
Without the explicit conversion i ran into ORA-28534: Heterogeneous Services preprocessing error and i have no hope of touching the DB config within reasonable time.
The explain plan btw did not bring much. It showed a FULL hint and no conversion on the predicates. Indeed it showed the ROW_CHANGED column as Date, i wonder why.
I have tried Justins suggestion to use a bind variable, however i got ORA-28534 again. Next thing i did was to wrap it into a pl/sql block (will run in a SP anyway later).
declare
v_tmstmp TIMESTAMP := 01.08.16 10:00:00;
begin
INSERT INTO ORAUSER.TMP_TBL (SRC_PK,ROW_CHANGED)
SELECT SRC_PK,ROW_CHANGED
FROM lib.tbl#dblink_name
WHERE ROW_CHANGED > v_tmstmp;
end;
This was executing in the same time as in DB2 itself. The date format is DD.MM.YY here since it is the default unfortunately.
When changing the variable assignment to
v_tmstmp TIMESTAMP := TO_TIMESTAMP('01.08.16 10:00:00','DD.MM.YY HH24:MI:SS');
I got the same problem as before.
Meanwhile the DB2 operators have created an index in the ROW_CHANGED column which i requested earlier that day. This has solved the problem in general it seems. Even my original query finishes in no time now.
If you are actually using an Oracle-specific conversion function like to_timestamp, that forces the predicate to be evaluated on the Oracle side. Oracle isn't going to know how to convert a built-in function like to_timestamp into an exactly equivalent function call in DB2.
If you used a bind variable, that would be more likely to get evaluated on the DB2 side. But that may be complicated by the data type mapping between different databases-- there may not be a perfect mapping between one engine's date and another engine's timestamp data type. If this was a numeric column, a bind variable would be almost certain to get pushed. In this case, it probably involves playing around a bit to figure out exactly what data type to use for your variable that works for your framework, Oracle, and DB2.
If using a bind variable doesn't work, you can force the predicate to be evaluated on the remote server using the dbms_hs_passthrough package. That lets you send a query verbatim to the remote server which allows you to do things like use functions defined in your DB2 database. That's a bit of overkill in this situation, hopefully, but it's nice to have the hammer as your backup if the simpler solution doesn't work quickly enough.
I'm trying to update a table (in pgsql) with a complex expression that needs to occur several times in the UPDATE statement. WITH seems perfect for this:
WITH newtz AS (SELECT timezone FROM timezonebyzipcode WHERE zip=(SELECT zip_code FROM company WHERE id=company_id))
UPDATE cross_rental
SET return_timezone=newtz
return_time=(return_time AT TIME ZONE return_timezone) AT TIME ZONE newtz
WHERE return_to='Vendor' AND return_timezone<>newtz
Unfortunately, it doesn't work:
ERROR: syntax error at or near "UPDATE"
LINE 2: UPDATE cross_rental
^
I've searched and couldn't find any examples of using WITH with UPDATE in this way, but I also don't see anything indicating it shouldn't work. Is this just unsupported, or am I making some silly mistake?
And, if it's unsupported, should I just copy that nasty long expression into each of the three places where I'm using "newtz" in the UPDATE clause? Or is there some better way to accomplish this update?
ERROR: syntax error at or near "UPDATE"
LINE 2: UPDATE cross_rental
This specific error message reveals you're using a PostgreSQL version 9.0 or older. The two major versions before 9.1 featured CTEs and WITH, but not in the context of data modifying queries.
This appeared in 9.1. See 7.8.2. Data-Modifying Statements in WITH in PostgreSQL 9.1 doc.
Assuming a newer version, a CTE must be used as a table with rows and columns (not as a scalar variable), so the query should be fixed as mentioned in Richard Huxton's answer.
It's a table (or a from-recordset-source anyway).
WITH calculated AS (
SELECT ....
)
UPDATE foo
SET bar = calculated.something
FROM calculated
WHERE ...
WITH (SELECT timezone FROM timezonebyzipcode WHERE zip=(SELECT zip_code FROM company WHERE id=company_id)) AS newtz
You have the alias and referent backwards...