I have a query that looks like the following:
SELECT * FROM table WHERE timestamp = NULL;
The timestamp column is a timestamp with time zone data type (second type in this table). This is in PostgreSQL 8.4.
What I'm trying to accomplish is to only select rows that have not had a timestamp inserted. When I look at the data in pgAdmin the field is empty and shows no value. I've tried where timestamp = NULL, 'EPOCH' (which you would think would be the default value), a valid timestamp of zeros (0000-00-00 00:00:00-00, which results in a out of range error), the lowest date possible according to the docs (January 1, 4713 BC) and a blank string ('', which just gets a data type mismatch error). There also appears to be no is_timestamp() function that I can use to check if the result is not a valid timestamp.
So, the question is, what value is in that empty field that I can check for?
Thanks.
EDIT: The field does not have a default value.
null in SQL means 'unknown'.
This means that the result of using any comparison operator, like =, with a null is also 'unknown'.
To check if a column is NULL (or not NULL), use the special syntax of IS NULL (or IS NOT NULL) instead of using =.
Applying that to your statement,
SELECT * FROM table WHERE timestamp IS NULL;
should work.
Related
I am tying to insert null value to a postgres timestamp datatype variable using python psycopg2.
The problem is the other data types such as char or int takes None, whereas the timestamp variable does not recognize None.
I tried to insert Null , null as a string because I am using a dictionary to get append the values for insert statement.
Below is the code.
queryDictOrdered[column] = queryDictOrdered[column] if isNull(queryDictOrdered[column]) is False else NULL
and the function is
def isNull(key):
if str(key).lower() in ('null','n.a','none'):
return True
else:
False
I get the below error messages:
DataError: invalid input syntax for type timestamp: "NULL"
DataError: invalid input syntax for type timestamp: "None"
Empty timestamps in Pandas dataframes come through as NaT (not a time), which is NOT pg compatible with NULL. A quick work around is to send it as a varchar and then run these 2 queries:
update <<schema.table_name>> set <<column_name>> = Null where
<<column_name>> = 'NULL';
or (depending on what you hard coded empty values as)
update <<schema.table_name>> set <<column_name>> = Null where <<column_name>> = 'NaT';
Finally run:
alter table <<schema.table_name>>
alter COLUMN <<column_name>> TYPE timestamp USING <<column_name>>::timestamp without time zone;
Surely you are adding quotes around the placeholder. Read psycopg documentation about passing parameters to queries.
Dropping this here incase it's helpful for anyone.
Using psycopg2 and the cursor object's copy_from method, you can copy missing or NaT datetime values from a pandas DataFrame to a Postgres timestamp field.
The copy_from method has a null parameter that is a "textual representation of NULL in the file. The default is the two characters string \N". See this link for more information.
Using pandas' fillna method, you can replace any missing datetime values with \N via data["my_datetime_field"].fillna("\\N"). Notice the double backslash here, where the first backslash is necessary to escape the second backslash.
Using the select_columns method from the pyjanitor module (or .loc[] and some subsetting with the column names of your DataFrame), you can coerce multiple columns at once via something akin to this, where all of your datetime fields end with an _at suffix.
data_datetime_fields = \
(data
.select_columns("*_at")
.apply(lambda x: x.fillna("\\N")))
I have a table named "temp_table" and a column named "temp_column" of type varchar. The problem is "temp_column" must be of type integer. If I will just automatically update the table into type integer, it will generate an error since some data has non-numeric data in it.
I want a query that will show all rows if "temp_column" has non-numeric values in it (or the other way around) and update or SET the value accordingly. I'm having a hard time since ISNUMERIC is not available in postgresql.
how to do this?
This will show all rows where you have non-integer values in that column. It uses a regular expression to find all values that have anything else than just numbers in it:
select *
from temp_table
where temp_column ~ '[^0-9]';
this can also be used in an update statement:
update temp_table
set temp_column = null
where temp_column ~ '[^0-9]';
This will also filter out "numeric" values like 3.14 as those aren't integers.
I'm migrating from SQLite to PostgreSQL, and the following query is not working anymore:
where("my_timestamp is NOT NULL and my_timestamp != ''")
How can I find all records that have a certain (datetime) attribute present?
Assuming that your my_timestamp column is a real timestamp (i.e. t.datetime in ActiveRecord parlance) then a simple NOT NULL test is sufficient:
where('my_timstamp is not null')
If this is the case then your query should be giving you an error like:
invalid input syntax for type timestamp: ""
pointing at your my_timestamp != '' test. Your comparison with an empty string worked fine in SQLite because SQLite doesn't have a real timestamp type, it just uses ISO 8601 formatted strings in text columns; this data type problem is also why you ended up with '' in your timestamp columns in SQLite in the first place.
create table Department
(Dname varchar(255) NOT NULL, Dnumber int NOT NULL PRIMARY KEY, Mgr_SSN char(9) NOT NULL, Mgr_start_Date DATE);
insert into Department values('HR', '1', '11001', '2012-04-05 10:15:00');
I am getting the error "not valid month".
Should we define date format when we create the table?
I am using Oracle11g.
When you have a DATE column, you should always insert a DATE, not a VARCHAR2. Relying on implicit casting to correctly convert the string is a bad idea-- it is very easy for different sessions to have different NLS settings and, thus, to do the implicit conversion differently (either resulting in a different DATE or an error). The easiest way to do that is to use the to_date function.
insert into Department( dname,
dnumber,
mgr_ssn,
mgr_start_date )
values('HR',
1,
'11001',
to_date( '2012-04-05 10:15:00', 'yyyy-mm-dd hh24:mi:ss') );
I also modified the statement to list the columns, which is generally a good practice since it ensures that you don't have to look up the physical order of columns in the table every time and since it allows the INSERT statement to work in the future if you add new columns to the table. Since dnumber is a NUMBER, I also changed the INSERT statement to insert a number rather than inserting a string (again, don't rely on implicit conversion if there is no need to do so). I did not correct the apparent bug that you have a CHAR(9) column representing a social security number for which you are inserting a 5 character string.
I'm trying to create an index on the cast of a varchar column to date. I'm doing something like this:
CREATE INDEX date_index ON table_name (CAST(varchar_column AS DATE));
I'm getting the error: functions in index expression must be marked IMMUTABLE But I don't get why, the cast to date doesn't depends on the timezone or something like that (which makes a cast to timestamp with time zone give this error).
Any help?
Your first error was to store a date as a varchar column. You should not do that.
The proper fix for your problem is to convert the column to a real date column.
Now I'm pretty sure the answer to that statement is "I didn't design the database and I cannot change it", so here is a workaround:
CAST and to_char() are not immutable because they can return different values for the same input value depending on the current session's settings.
If you know you have a consistent format of all values in the table (which - if you had - would mean you can convert the column to a real date column) then you can create your own function that converts a varchar to a date and is marked as immutable.
create or replace function fix_bad_datatype(the_date varchar)
returns date
language sql
immutable
as
$body$
select to_date(the_date, 'yyyy-mm-dd');
$body$
ROWS 1
/
With that definition you can create an index on the expression:
CREATE INDEX date_index ON table_name (fix_bad_datatype(varchar_column));
But you have to use exactly that function call in your query so that Postgres uses it:
select *
from foo
where fix_bad_datatype(varchar_column) < current_date;
Note that this approach will fail badly if you have just one "illegal" value in your varchar column. The only sensible solution is to store dates as dates,
Please provide the database version, table ddl, and some example data.
Would making your own immutable function do what you want, like this? Also look into creating a new cast in the docs and see if that does anything for you.
create table emp2 (emp2_id integer, hire_date VARCHAR(100));
insert into emp2(hire_date)
select now();
select cast(hire_date as DATE)
from emp2
CREATE FUNCTION my_date_cast(VARCHAR) RETURNS DATE
AS 'select cast($1 as DATE)'
LANGUAGE SQL
IMMUTABLE
RETURNS NULL ON NULL INPUT;
CREATE INDEX idx_emp2_hire_date ON emp2 (my_date_cast(hire_date));