Nxlog im_dbi is not working - postgresql

I am able to insert data into PostgreSQL using nxlog(om_dbi).
But I am not able to select data(or fetch data) from PostgreSQL using nxlog. I tried many options nothing is working.
And in nxlog document also for IM_DBI module description has only "FIXME" mentioned.
Document Link: http://nxlog.org/documentation/nxlog-community-edition-reference-manual-v20928#im_dbi
Please help me to solve this.
Logs:
<Input dbiin>
Module im_dbi
SavePos TRUE
SQL SELECT * FROM NEW_TABLE
Driver pgsql
Option host 127.0.0.1
Option username chitta
Option password ''
Option dbname db
</Input>
2014-10-16 14:29:17 WARNING nxlog-ce received a termination request signal, exiting...
2014-10-16 14:29:18 INFO nxlog-ce-2.8.1248 started
2014-10-16 14:29:18 ERROR im_dbi failed to execute SQL statement. ERROR: column "id" does not exist;LINE 1: SELECT * FROM NEW_TABLE WHERE id = 1;
Note:
the module will automatically prepends a "WHERE id > %d" clause.

Not an answer, but here's some help.
The most important directive is missing: SQL Select ID as id,
DateOccured as EventTime, data from logtable
Source: https://www.mail-archive.com/nxlog-ce-users#lists.sourceforge.net/msg00225.html
I'm currently in the same boat. My assumption is that your data is not formatted in a way that nxlog can interpret. Troubleshooting and will get back to you if I can find a resolution.
Also digging through the source code for the im_dbi module.
https://github.com/lamby/pkg-nxlog-ce/blob/master/src/modules/input/dbi/im_dbi.c

The answer by SoMuchToGrok is valid.
Actually the question already has this: "ERROR: column "id" does not exist".
The table must have an id column or you must use SELECT x as id so that the result set has id in it

Related

Trying to select all rows in this column with a certain value, and failing [duplicate]

I'm writing a Java application to automatically build and run SQL queries. For many tables my code works fine but on a certain table it gets stuck by throwing the following exception:
Exception in thread "main" org.postgresql.util.PSQLException: ERROR: column "continent" does not exist
Hint: Perhaps you meant to reference the column "countries.Continent".
Position: 8
The query that has been run is the following:
SELECT Continent
FROM network.countries
WHERE Continent IS NOT NULL
AND Continent <> ''
LIMIT 5
This essentially returns 5 non-empty values from the column.
I don't understand why I'm getting the "column does not exist" error when it clearly does in pgAdmin 4. I can see that there is a schema with the name Network which contains the table countries and that table has a column called Continent just as expected.
Since all column, schema and table names are retrieved by the application itself I don't think there has been a spelling or semantical error so why does PostgreSQL cause problems regardless? Running the query in pgAdmin4 nor using the suggested countries.Continent is working.
My PostgreSQL version is the newest as of now:
$ psql --version
psql (PostgreSQL) 9.6.1
How can I successfully run the query?
Try to take it into double quotes - like "Continent" in the query:
SELECT "Continent"
FROM network.countries
...
In working with SQLAlchemy environment, i have got this error with the SQL like this,
db.session.execute(
text('SELECT name,type,ST_Area(geom) FROM buildings WHERE type == "plaza" '))
ERROR: column "plaza" does not exist
Well, i changed == by = , Error still persists, then i interchanged the quotes, like follows. It worked. Weird!
....
text("SELECT name,type,ST_Area(geom) FROM buildings WHERE type = 'plaza' "))
This problem occurs in postgres because the table name is not tablename instead it is "tablename".
for eg.
If it shows user as table name,
than table name is "user".
See this:
Such an error can appear when you add a space in the name of a column by mistake (for example "users ").
QUICK FIX (TRICK)
If you have recently added a field which you have already deleted before and now trying to add the same field back then let me share you this simple trick! i did this and the problem was gone!!
so, now just delete the migration folder entirely on the app,then instead of adding that field you need to now add a field but with the name of which you have never declared on this app before, example if you are trying to add title field then create it by the name of heading and now do the migration process separately on the app and runserver, now go to admin page and look for that model and delete all the objects and come to models back and rename the field that you recently made and name it to which you were wishing it with earlier and do the migrations again and now your problem must have been gone!!
this occurs when the objects are there in the db but you added a field which wasn't there when the earlier objs were made, so by this we can delete those objs and make fresh ones again!
I got the same error when I do PIVOT in RedShift.
My code is similar to
SELECT *
INTO output_table
FROM (
SELECT name, year_month, sales
FROM input_table
)
PIVOT
(
SUM(sales)
FOR year_month IN ('nov_2020', 'dec_2020', 'jan_2021', 'feb_2021', 'mar_2021', 'apr_2021', 'may_2021', 'jun_2021', 'jul_2021', 'aug_2021',
'sep_2021', 'oct_2021', 'nov_2021', 'dec_2021', 'jan_2022', 'feb_2022', 'mar_2022', 'apr_2022', 'may_2022', 'jun_2022',
'jul_2022', 'aug_2022', 'sep_2022', 'oct_2022', 'nov_2022')
)
I tried year_month without any quote (got the error), year_month with double quote (got the error), and finally year_month with single quote (it works this time). This may help if someone in the same situation like my example.

Postgres Copy csv to table sql [duplicate]

I'm trying to run the following PHP script to do a simple database query:
$db_host = "localhost";
$db_name = "showfinder";
$username = "user";
$password = "password";
$dbconn = pg_connect("host=$db_host dbname=$db_name user=$username password=$password")
or die('Could not connect: ' . pg_last_error());
$query = 'SELECT * FROM sf_bands LIMIT 10';
$result = pg_query($query) or die('Query failed: ' . pg_last_error());
This produces the following error:
Query failed: ERROR: relation "sf_bands" does not exist
In all the examples I can find where someone gets an error stating the relation does not exist, it's because they use uppercase letters in their table name. My table name does not have uppercase letters. Is there a way to query my table without including the database name, i.e. showfinder.sf_bands?
From what I've read, this error means that you're not referencing the table name correctly. One common reason is that the table is defined with a mixed-case spelling, and you're trying to query it with all lower-case.
In other words, the following fails:
CREATE TABLE "SF_Bands" ( ... );
SELECT * FROM sf_bands; -- ERROR!
Use double-quotes to delimit identifiers so you can use the specific mixed-case spelling as the table is defined.
SELECT * FROM "SF_Bands";
Re your comment, you can add a schema to the "search_path" so that when you reference a table name without qualifying its schema, the query will match that table name by checked each schema in order. Just like PATH in the shell or include_path in PHP, etc. You can check your current schema search path:
SHOW search_path
"$user",public
You can change your schema search path:
SET search_path TO showfinder,public;
See also http://www.postgresql.org/docs/8.3/static/ddl-schemas.html
I had problems with this and this is the story (sad but true) :
If your table name is all lower case like : accounts
you can use: select * from AcCounTs and it will work fine
If your table name is all lower case like : accounts
The following will fail:
select * from "AcCounTs"
If your table name is mixed case like : Accounts
The following will fail:
select * from accounts
If your table name is mixed case like : Accounts
The following will work OK:
select * from "Accounts"
I dont like remembering useless stuff like this but you have to ;)
Postgres process query different from other RDMS. Put schema name in double quote before your table name like this, "SCHEMA_NAME"."SF_Bands"
Put the dbname parameter in your connection string. It works for me while everything else failed.
Also when doing the select, specify the your_schema.your_table like this:
select * from my_schema.your_table
If a table name contains underscores or upper case, you need to surround it in double-quotes.
SELECT * from "Table_Name";
I had a similar problem on OSX but tried to play around with double and single quotes. For your case, you could try something like this
$query = 'SELECT * FROM "sf_bands"'; // NOTE: double quotes on "sf_Bands"
This is realy helpfull
SET search_path TO schema,public;
I digged this issues more, and found out about how to set this "search_path" by defoult for a new user in current database.
Open DataBase Properties then open Sheet "Variables"
and simply add this variable for your user with actual value.
So now your user will get this schema_name by defoult and you could use tableName without schemaName.
You must write schema name and table name in qutotation mark. As below:
select * from "schemaName"."tableName";
I had the same issue as above and I am using PostgreSQL 10.5.
I tried everything as above but nothing seems to be working.
Then I closed the pgadmin and opened a session for the PSQL terminal.
Logged into the PSQL and connected to the database and schema respectively :
\c <DATABASE_NAME>;
set search_path to <SCHEMA_NAME>;
Then, restarted the pgadmin console and then I was able to work without issue in the query-tool of the pagadmin.
For me the problem was, that I had used a query to that particular table while Django was initialized. Of course it will then throw an error, because those tables did not exist. In my case, it was a get_or_create method within a admin.py file, that was executed whenever the software ran any kind of operation (in this case the migration). Hope that helps someone.
In addition to Bill Karwin's answer =>
Yes, you should surround the table name with double quotes. However, be aware that most probably php will not allow you to just write simply:
$query = "SELECT * FROM "SF_Bands"";
Instead, you should use single quotes while surrounding the query as sav said.
$query = 'SELECT * FROM "SF_Bands"';
You have to add the schema first e.g.
SELECT * FROM place.user_place;
If you don't want to add that in all queries then try this:
SET search_path TO place;
Now it will works:
SELECT * FROM user_place;
Easiest workaround is Just change the table name and all column names to lowercase and your issue will be resolved.
For example:
Change Table_Name to table_name and
Change ColumnName to columnname
It might be silly for a few, but in my case - once I created the table I could able to query the table on the same session, but if I relogin with new session table does not exits.
Then I used commit just after creating the table and now I could able to find and query the table in the new session as well. Like this:
select * from my_schema.my_tbl;
Hope this would help a few.
Make sure that Table name doesn't contain any trailing whitespaces
Try this: SCHEMA_NAME.TABLE_NAME
I'd suggest checking if you run the migrations or if the table exists in the database.
I tried every good answer ( upvote > 10) but not works.
I met this problem in pgAdmin4.
so my solution is quite simple:
find the target table / scheme.
mouse right click, and click: query-tool
in this new query tool window, you can run your SQL without specifying set search_path to <SCHEMA_NAME>;
you can see the result:

Pgadmin is not able to read the column names due to header in column name [duplicate]

I'm writing a Java application to automatically build and run SQL queries. For many tables my code works fine but on a certain table it gets stuck by throwing the following exception:
Exception in thread "main" org.postgresql.util.PSQLException: ERROR: column "continent" does not exist
Hint: Perhaps you meant to reference the column "countries.Continent".
Position: 8
The query that has been run is the following:
SELECT Continent
FROM network.countries
WHERE Continent IS NOT NULL
AND Continent <> ''
LIMIT 5
This essentially returns 5 non-empty values from the column.
I don't understand why I'm getting the "column does not exist" error when it clearly does in pgAdmin 4. I can see that there is a schema with the name Network which contains the table countries and that table has a column called Continent just as expected.
Since all column, schema and table names are retrieved by the application itself I don't think there has been a spelling or semantical error so why does PostgreSQL cause problems regardless? Running the query in pgAdmin4 nor using the suggested countries.Continent is working.
My PostgreSQL version is the newest as of now:
$ psql --version
psql (PostgreSQL) 9.6.1
How can I successfully run the query?
Try to take it into double quotes - like "Continent" in the query:
SELECT "Continent"
FROM network.countries
...
In working with SQLAlchemy environment, i have got this error with the SQL like this,
db.session.execute(
text('SELECT name,type,ST_Area(geom) FROM buildings WHERE type == "plaza" '))
ERROR: column "plaza" does not exist
Well, i changed == by = , Error still persists, then i interchanged the quotes, like follows. It worked. Weird!
....
text("SELECT name,type,ST_Area(geom) FROM buildings WHERE type = 'plaza' "))
This problem occurs in postgres because the table name is not tablename instead it is "tablename".
for eg.
If it shows user as table name,
than table name is "user".
See this:
Such an error can appear when you add a space in the name of a column by mistake (for example "users ").
QUICK FIX (TRICK)
If you have recently added a field which you have already deleted before and now trying to add the same field back then let me share you this simple trick! i did this and the problem was gone!!
so, now just delete the migration folder entirely on the app,then instead of adding that field you need to now add a field but with the name of which you have never declared on this app before, example if you are trying to add title field then create it by the name of heading and now do the migration process separately on the app and runserver, now go to admin page and look for that model and delete all the objects and come to models back and rename the field that you recently made and name it to which you were wishing it with earlier and do the migrations again and now your problem must have been gone!!
this occurs when the objects are there in the db but you added a field which wasn't there when the earlier objs were made, so by this we can delete those objs and make fresh ones again!
I got the same error when I do PIVOT in RedShift.
My code is similar to
SELECT *
INTO output_table
FROM (
SELECT name, year_month, sales
FROM input_table
)
PIVOT
(
SUM(sales)
FOR year_month IN ('nov_2020', 'dec_2020', 'jan_2021', 'feb_2021', 'mar_2021', 'apr_2021', 'may_2021', 'jun_2021', 'jul_2021', 'aug_2021',
'sep_2021', 'oct_2021', 'nov_2021', 'dec_2021', 'jan_2022', 'feb_2022', 'mar_2022', 'apr_2022', 'may_2022', 'jun_2022',
'jul_2022', 'aug_2022', 'sep_2022', 'oct_2022', 'nov_2022')
)
I tried year_month without any quote (got the error), year_month with double quote (got the error), and finally year_month with single quote (it works this time). This may help if someone in the same situation like my example.

Dreamfactory: Database connection to Postgres

First I set up the dream factory by "Bitnami Installer for Windows". Following to https://github.com/dreamfactorysoftware/dsp-core/wiki/Install-Microsoft-Windows
Then I follow this add-a-rest-api-to-any-sql-db-in-minutes to add Services to my Remote Postgres Database.
On the "API Docs" tab, it is success to call the GET /db operation. ( getTables() - List all table names).
I got the following error when trying to call GET /db/{table_name} operation ( getRecordsByFilter() - Retrieve one or more records by using a filter).
Please help
[app][ERROR ] CDbCommand::fetchAll() failed: SQLSTATE[42601]: Syntax error: 7 ERROR: syntax error at or near "field_name"
LINE 1: SELECT k.column_name field_name
^. The SQL statement executed was: SELECT k.column_name field_name
FROM "information_schema"."key_column_usage" k
LEFT JOIN "information_schema"."table_constraints" c
ON k.table_name = c.table_name
AND k.constraint_name = c.constraint_name
WHERE c.constraint_type ='PRIMARY KEY'
AND k.table_name = :table
AND k.table_schema = :schema.
Please check the version of PostgreSQL you are connecting to. It turns out that omission of as keyword is supported starting from 8.4.
Here's a quote from 8.4 release notes section E.23.3.3. Queries:
Allow AS to be optional when specifying a SELECT (or RETURNING) column
output label (Hiroshi Saito)
This works so long as the column label is not any PostgreSQL keyword;
otherwise AS is still needed.
Therefore SELECT k.column_name field_name is not valid for 8.3 and below, but SELECT k.column_name AS field_name would work.
If PostgreSQL version is the cause of your problem you have several options:
update the database to 8.4 and above;
patch the Dreamfactory codebase yourself to work around this problem;
raise a ticket in Dreamfactory's bug tracker and wait them to fix it for you.

Create table with postgreSQL [duplicate]

I'm trying to run the following PHP script to do a simple database query:
$db_host = "localhost";
$db_name = "showfinder";
$username = "user";
$password = "password";
$dbconn = pg_connect("host=$db_host dbname=$db_name user=$username password=$password")
or die('Could not connect: ' . pg_last_error());
$query = 'SELECT * FROM sf_bands LIMIT 10';
$result = pg_query($query) or die('Query failed: ' . pg_last_error());
This produces the following error:
Query failed: ERROR: relation "sf_bands" does not exist
In all the examples I can find where someone gets an error stating the relation does not exist, it's because they use uppercase letters in their table name. My table name does not have uppercase letters. Is there a way to query my table without including the database name, i.e. showfinder.sf_bands?
From what I've read, this error means that you're not referencing the table name correctly. One common reason is that the table is defined with a mixed-case spelling, and you're trying to query it with all lower-case.
In other words, the following fails:
CREATE TABLE "SF_Bands" ( ... );
SELECT * FROM sf_bands; -- ERROR!
Use double-quotes to delimit identifiers so you can use the specific mixed-case spelling as the table is defined.
SELECT * FROM "SF_Bands";
Re your comment, you can add a schema to the "search_path" so that when you reference a table name without qualifying its schema, the query will match that table name by checked each schema in order. Just like PATH in the shell or include_path in PHP, etc. You can check your current schema search path:
SHOW search_path
"$user",public
You can change your schema search path:
SET search_path TO showfinder,public;
See also http://www.postgresql.org/docs/8.3/static/ddl-schemas.html
I had problems with this and this is the story (sad but true) :
If your table name is all lower case like : accounts
you can use: select * from AcCounTs and it will work fine
If your table name is all lower case like : accounts
The following will fail:
select * from "AcCounTs"
If your table name is mixed case like : Accounts
The following will fail:
select * from accounts
If your table name is mixed case like : Accounts
The following will work OK:
select * from "Accounts"
I dont like remembering useless stuff like this but you have to ;)
Postgres process query different from other RDMS. Put schema name in double quote before your table name like this, "SCHEMA_NAME"."SF_Bands"
Put the dbname parameter in your connection string. It works for me while everything else failed.
Also when doing the select, specify the your_schema.your_table like this:
select * from my_schema.your_table
If a table name contains underscores or upper case, you need to surround it in double-quotes.
SELECT * from "Table_Name";
I had a similar problem on OSX but tried to play around with double and single quotes. For your case, you could try something like this
$query = 'SELECT * FROM "sf_bands"'; // NOTE: double quotes on "sf_Bands"
This is realy helpfull
SET search_path TO schema,public;
I digged this issues more, and found out about how to set this "search_path" by defoult for a new user in current database.
Open DataBase Properties then open Sheet "Variables"
and simply add this variable for your user with actual value.
So now your user will get this schema_name by defoult and you could use tableName without schemaName.
You must write schema name and table name in qutotation mark. As below:
select * from "schemaName"."tableName";
I had the same issue as above and I am using PostgreSQL 10.5.
I tried everything as above but nothing seems to be working.
Then I closed the pgadmin and opened a session for the PSQL terminal.
Logged into the PSQL and connected to the database and schema respectively :
\c <DATABASE_NAME>;
set search_path to <SCHEMA_NAME>;
Then, restarted the pgadmin console and then I was able to work without issue in the query-tool of the pagadmin.
For me the problem was, that I had used a query to that particular table while Django was initialized. Of course it will then throw an error, because those tables did not exist. In my case, it was a get_or_create method within a admin.py file, that was executed whenever the software ran any kind of operation (in this case the migration). Hope that helps someone.
In addition to Bill Karwin's answer =>
Yes, you should surround the table name with double quotes. However, be aware that most probably php will not allow you to just write simply:
$query = "SELECT * FROM "SF_Bands"";
Instead, you should use single quotes while surrounding the query as sav said.
$query = 'SELECT * FROM "SF_Bands"';
You have to add the schema first e.g.
SELECT * FROM place.user_place;
If you don't want to add that in all queries then try this:
SET search_path TO place;
Now it will works:
SELECT * FROM user_place;
Easiest workaround is Just change the table name and all column names to lowercase and your issue will be resolved.
For example:
Change Table_Name to table_name and
Change ColumnName to columnname
It might be silly for a few, but in my case - once I created the table I could able to query the table on the same session, but if I relogin with new session table does not exits.
Then I used commit just after creating the table and now I could able to find and query the table in the new session as well. Like this:
select * from my_schema.my_tbl;
Hope this would help a few.
Make sure that Table name doesn't contain any trailing whitespaces
Try this: SCHEMA_NAME.TABLE_NAME
I'd suggest checking if you run the migrations or if the table exists in the database.
I tried every good answer ( upvote > 10) but not works.
I met this problem in pgAdmin4.
so my solution is quite simple:
find the target table / scheme.
mouse right click, and click: query-tool
in this new query tool window, you can run your SQL without specifying set search_path to <SCHEMA_NAME>;
you can see the result: