Sometimes postgres does not return records when queried remotely from JDBC Client - postgresql

public static ArrayList<String[]> getDailyRecords () throws ClassNotFoundException {
ArrayList<String[]> list=new ArrayList<String[]> ();
String[] header = {"City", "Location", "Asset", "Number of Alerts", "Time spent in alerts", "Last seen temparature", "Limit"};
list.add (header);
String myDriver = "org.postgresql.Driver";
Class.forName (myDriver);
try( Connection conn = DriverManager.getConnection( ApplicationSettings.DATABASE_URL, ApplicationSettings.DATABASE_USER, ApplicationSettings.DATABASE_PASSWORD);
Statement st = conn.createStatement ();) {
conn.setAutoCommit (true);
ResultSet rs=null;
st.setFetchSize (200);
String dailyQuery="select * from sch.reports();";
rs= st.executeQuery (dailyQuery);
while (rs.next ()) {
String[] ar = new String[7];
ar[0] = rs.getString ("location");
ar[1] = rs.getString ("sublocation");
ar[2] = rs.getString ("zone");
ar[3] = rs.getString ("total_count");
ar[4] = rs.getString ("period");
String temp = rs.getString ("temparature");
ar[5] = temp;
list.add (ar);
}
conn.close ();
st.close ();
rs.close ();
}catch(Exception e){
e.printStackTrace ();
}catch(Error e){
e.printStackTrace ();
}
finally {
if(list.size ()==1){
System.out.println ("NO records found");
}else{
System.out.println ("Foudn some records");
}
return list;
}
I have a sql function, which does return records when queried locally from postgres clients. I invoke this function at scheduled timings from a java application. This worked fine for few months. But all of sudden, st.executeQuery() returning empty result set occasionally. Like 4 out of 10 attempts of executeQuery() in a day return empty resultset.
Things I have tried out:
Made sure query returns some records always. It should return 60 records.
Captured Errors,Exceptions. There were no errors seen.
Made sure JDBC connection is closed.
Debugged Postgres JDBC driver classes. Found that in cases when resultset is empty, data was not received from postgres data stream.
can size of data in tables influence query response?
Any help is kindly appreciated!
Update: SQL function definition added.
DECLARE
exception_error_code text;
exception_message text;
exception_detail text;
exception_hint text;
exception_context text;
BEGIN
RETURN QUERY
select l.name as location,sl.name as sublocation,z.name as zone,al.lim as lim,count(al.id) total_count,sum(al.timestamp_to-al.timestamp_from) period,t2.temperature temparature
from
map.location l
left join
map.sub_location sl
on l.id=sl.location_id
left join
map.zone z
on sl.id=z.sub_location_id
left join
map.subzone sz
on z.id=sz.zone_id
inner join
(select *,(info::json -> 'breach_type')::text as lim from alr.live where date(timestamp_from)= date(now() + INTERVAL '5 hours 30 minutes')) al
on sz.id= al.sub_zone_id
left join
(select subzone_id,max(nd_timestamp) as nd_timestamp from tel.temperature_sh group by subzone_id) t1
on al.sub_zone_id=t1.subzone_id
left join
tel.temperature_sh t2
on t1.subzone_id=t2.subzone_id
and t1.nd_timestamp=t2.nd_timestamp
group by l.name ,sl.name,z.name,t2.temperature , al.lim
order by l.name ,sl.name,z.name;
-- if something "breaks" do the following
EXCEPTION WHEN others THEN
get stacked diagnostics
exception_error_code = RETURNED_SQLSTATE
,exception_message = MESSAGE_TEXT
,exception_detail = PG_EXCEPTION_DETAIL
,exception_hint = PG_EXCEPTION_HINT
,exception_context = PG_EXCEPTION_CONTEXT
;
-- log exception for debugging
PERFORM public.insert_db_exception(
exception_error_code
,exception_message
,exception_detail
,exception_hint
,exception_context
);
END;

Related

How can I read 23 million records from postgres using JDBC? I have to read from a table in postgres and write to another table

When I write a simple JPA code to findAll() data, I run into memory issues. For writing, I can do batch update. But how to read 23 million records and save them in list for storing into another table?
Java is a poor choice for processing "batch" stuff (and I love java!).
Instead, do it using pure SQL:
insert into target_table (col1, col2, ...)
select col1, col2, ....
from ...
where ...
or, if you must do some processing in java that can't be done within the query, open a cursor for the query and read rows 1 at a time and write the target row before reading the next row. This approach however will take a looooong time to finish.
I fully agree with Bohemian's answer.
If the source and the destination tables you can read and write within the same loop
something in a try - catch block like:
PreparedStatement reader = null;
PreparedStatement writer = null;
ResultSet rs = null;
try {
reader = sourceConnection.prepareStatement("select....");
writer = destinationConnection.prepareStatement("insert into...");
rs = reader.executeQuery();
int chunksize = 10000; // this is you batch size, depends on your system
int counter = 0;
while ( rs.next() {
writer.set.... // do for every field to insert the corresponding set
writer.addBatch();
if ( counter++ % chunksize == 0 ) {
int rowsWritten = writer.executeBatch();
System.out.println("wrote " + counter + " rows"); // probably a simple message to see a progress
}
}
// when finished, do not forget to flush the rest of the batch job
writer.executeBatch();
} catch (SQLException sqlex ) {
// an Errormessage to your gusto
System.out.println("SQLException: " + sqlex.getMessage());
} finally {
try {
if ( rs != null ) rs.close();
if ( reader != null ) reader.close();
if ( writer != null ) writer.close();
// probably you want to clsoe the connections as well
} catch (SQLException e ) {
System.out.println("Exception while closing " + e.getMessage());
}
}

Dblink is not working in java

String id="";
List list = null;
Query query = null;
Session session = getSession();
String sql = "select id from student#dblink s, location#dblink l where s.id in (select name from Employ where id='" +Id +"') and s.country_id = l.country_id and l.country_end = '"+countryEnd+"'";
query = session.createSQLQuery(sql);
System.out.println("sql.1."+sql);
list = query.list();
if(list.size()>0)
id = list.get(0).toString();
else
id= "0";
System.out.println("Stage2Ubr Id is:"+id);
return id;
Here I got exception like "org.hibernate.exception.GenericJDBCException: could not execute query". Sometimes this exception is not coming. Sometimes it's working fine without exception. This exception won't come if dblink is working fine.Please help me. How to check wheather the dblink is working fine or not?

Column must appear in the GROUP BY clause or be used in an aggregate function

I'm updating a Qt software, to make it compatible with both SQLite and PostgreSQL.
I have a C++ method that is used to count elements of a given table with given clauses.
In SQLite, the following worked and gave me a number N (the count).
SELECT COUNT(*) FROM table_a
INNER JOIN table_b AS
ON table_b.fk_table_a = table_a.id
WHERE table_a.start_date_time <> 0
ORDER BY table_a.creation_date_time DESC
With PostgreSQL (I'm using 9.3), I have the following error :
ERROR: column "table_a.creation_date_time" must appear in the
GROUP BY clause or be used in an aggregate function
LINE 5: ORDER BY
table_a.creation_date_time DESC
If I add, GROUP BY table_a.creation_date_time, it gives me a table with N rows.
I've read a lot of stuff about how different DBMS allow you to omit columns in the GROUP BY clause. Now, I'm just confused.
For those who are curious, the C++ method is:
static int count(const QString &table, const QString &clauses = QString(""))
{
int success = -1;
if (!table.isEmpty())
{
QString statement = QString("SELECT COUNT(*) FROM ");
statement.append(table);
if (!clauses.isEmpty())
{
statement.append(" ").append(clauses) ;
}
QSqlQuery query;
if(!query.exec(statement))
{
qWarning() << query.lastError();
qWarning() << statement;
}
else
{
if (query.isActive() && query.isSelect() && query.first())
{
bool ok = false;
success = query.value(0).toInt(&ok);
if (ok == false)
{
success = -1;
return success;
}
}
}
}
return success;
}
If you're just doing a count(*) on the table in order to get a single scalar-value result, then surely having the order by present is obsolete ?
solution
Remove the obsolete order by to get "standard" query behavior across multiple dbms

Postgresql query for this situation?

I am using the following statement
select *
from table
where column1 in(groups)
Where "groups" is a String array of size n.
If i use it as it is, It wont get executed so can anyone suggest exact query to perform this?
EDIT 1
If I use the following code
try{
System.out.println("before execute query");
ps1.setArray(1,conn.createArrayOf("text",gs));
ps1.setArray(2,conn.createArrayOf("text",gs));
System.out.println("after execute query");
}
catch(Exception e)
{
System.out.println("hrer----"+e);
}
First,It prints "before execute query" and then it gives the following exception
javax.servlet.ServletException:servlet execution threw an exception
*NOTE : * It does not print "hrer-----" in catch(Exception e) block
This should work:
PreparedStatement stmt = conn.prepareStatement(
"SELECT * FROM users WHERE username = any(?)");
String[] usernames = {"admin", "guest"};
stmt.setArray(1, conn.createArrayOf("varchar", usernames));
Credit goes to Boris's answer at https://stackoverflow.com/a/10240302
select * from table where column1 in (?, ?)
except that you have n question marks.
StringBuilder q = new StringBuilder("select * from table where column1 in (");
for(int i=0; i<groups.length; i++) {
q.append("?");
if(i != groups.length - 1) {
q.append(",");
}
}
q.append(")");
PreparedStatement query = con.prepareStatement(q.toString());
for(int i=1; i<=groups.length; i++) {
query.setString(i, groups[i-1]);
}
ResultSet rs = query.getResultSet();

How to use Multiple resultsets with POSTGRES JDBC?

I am using JDBC on a PostgreSQL database.
When I query for an entity in a resultset, it returns 5 rows.
Related to that entity is another entity, for which I query while i am using a row in the above resultset.
When I execute this query, the above resultset is closed.
This means that it is allowing only 1 resultset to be active on 1 connection at a time.
Previously the same code was working perfect for Oracle DB server.
Is it that I need to ask the DB admin to configure the server to allow multiple resultsets?
Or to do some change in the code?
Or is it impossible to do it in postgre?
Here is the code for more details:
Connection conn = PTSConnection.getConnection();
Statement stmt = conn.createStatement();
ResultSet lines = stmt.executeQuery("SELECT LINEID,STARTSTOPID,ENDSTOPID FROM LINES"); **//first resultset is active**
while (lines.next()){
int lineId= lines.getInt(1);
Stop ss = StopStorage.getByID(lines.getInt(2));
Stop es = StopStorage.getByID(lines.getInt(3));
ResultSet stops = stmt.executeQuery("SELECT STOPID FROM STOPSINLINES WHERE LINEID=" + lineId); **//first resultset dies**
List<Stop> lineStops = new ArrayList<Stop>();
while(stops.next()){
Stop stop = StopStorage.getByID(stops.getInt(1));
lineStops.add(stop);
}
stops.close();
Line aLine = null;
ResultSet emergencyLine = stmt.executeQuery("SELECT CAUSE, STARTTIME, ENDTIME FROM EMERGENCYLINES WHERE LINEID =" + lineId);
if(emergencyLine.next()){
String cause = emergencyLine.getString(1);
Time startTime = emergencyLine.getTime(2);
Time endTime = emergencyLine.getTime(3);
aLine = new EmergencyLine(ss, es, cause, startTime, endTime, (Stop[]) lineStops.toArray(new Stop[lineStops.size()]));
} else {
aLine = new Line(ss, es, (Stop[]) lineStops.toArray(new Stop[lineStops.size()]));
}
emergencyLine.close();
LineRepository.getInstance().addLine(aLine);
}
lines.close();
The reason is not that you are using two resultsets on the same connection, but you are re-using the same Statement object for a new query. When you run executeQuery() on a Statement instance, any previous result will be closed (I'm surprised that your code did work with Oracle...)
Simply create a new Statement object before executing the second query:
Statement stmt = conn.createStatement();
Statement nestedStmt = conn.createStatement();
ResultSet lines = stmt.executeQuery("SELECT LINEID,STARTSTOPID,ENDSTOPID FROM LINES"); **//first resultset is active**
while (lines.next()){
...
ResultSet stops = nestedStmt.executeQuery("SELECT STOPID FROM STOPSINLINES WHERE LINEID=" + lineId); **//first resultset dies**
List lineStops = new ArrayList();
while(stops.next()){
Stop stop = StopStorage.getByID(stops.getInt(1));
lineStops.add(stop);
}
stops.close();
...
ResultSet emergencyLine = nestedStmt.executeQuery("SELECT CAUSE, STARTTIME, ENDTIME FROM EMERGENCYLINES WHERE LINEID =" + lineId);
if(emergencyLine.next()){
String cause = emergencyLine.getString(1);
....
}
emergencyLine.close();
And don't for get to properly close all Statements and ResultSets !