Using JPA to call native Postgresql command - postgresql

I am using a third party library to perform mass inserts into a database PgBulkInsert . It takes inserts that would normally take 30 minutes and performs them in 30 seconds. We have noticed that overtime there is disk usage leakage, but we figure out that performing a table reindex appears to corrcect the issue. I am trying to use my JPA Entity Manager to perform a native update. The below code works but contains potential SQL injection vulnerability.
#Stateless
public class ReindexService {
#PersistenceContext(unitName = "my-ds")
private EntityManager em;
public void reindexTable(String table) {
String queryStr = "REINDEX TABLE " + table;
Query query = em.createNativeQuery(queryStr);
query.executeUpdate();
}
}
When I pass in string "alert" to index the alert table it yields the following SQL output
/* dynamic native SQL query */ REINDEX TABLE alert
When I attempt to use a positional parameter it yields a SQL error
String queryStr = "REINDEX TABLE ?";
Query query = em.createNativeQuery(queryStr);
query.setParameter(1, table);
query.executeUpdate();
This yields the following error output
/* dynamic native SQL query */ REINDEX TABLE ?
SQL Error: 0, SQLState: 42601
ERROR: syntax error at or near "$1"
Position: 46
I get a similar error when I try to use a name parameter
String queryStr = "REINDEX TABLE :table";
Query query = em.createNativeQuery(queryStr);
query.setParameter("table", table);
query.executeUpdate();
This yields the same error
/* dynamic native SQL query */ REINDEX TABLE ?
SQL Error: 0, SQLState: 42601
ERROR: syntax error at or near "$1"
Position: 46
Does anyone know how I can call a the native Postgresql reindex table command using my entity manager in a manner without adding a SQL injection vulnerability? I am using Hibernate 5.3.6.Final but would prefer a non-implementation specific solution.
I also tried to access the Connection and perform a JDBC call and it seems to give the error
final Session session = //get session from entity manager
session.doWork(conn -> {
try (PreparedStatement stmt = conn.prepareCall(REINDEX TABLE ?)) {
stmt.setString(1, table);
stmt.execute();
}
});
Yields the same errors as above
SQL Error: 0, SQLState: 42601
ERROR: syntax error at or near "$1"
Position: 15

Identifiers can't be passed as parameters. If you don't expect the table name to come from user input (it sounds a bit strange), you may try using an enumeration of all the tables which you want to reindex, and pass enumeration values to your service (and just concatenate strings).
If you do expect table names to come from untrusted sources, you can try enclosing identifier with double-quotes and escaping existing double-quotes.
There is also a function quote_ident in PostgreSQL which can be used to quote identifiers properly. So you can create a stored procedure which takes a regular argument from your JPA code and uses quote_ident and EXECUTE a constructed query
.

Our workaround was to create a Database Function and call it using a native query
The database function
CREATE OR REPLACE FUNCTION reindex_table(table_in text)
RETURNS void
SECURITY DEFINER
AS $$
BEGIN
EXECUTE FORMAT('REINDEX (VERBOSE) TABLE %I', table_in);
RETURN;
END;
$$ LANGUAGE plpgsql;
Here is the Service code for calling the function
public void reindexTable(String table) {
String queryStr = "select reindex_table(?)";
final Session session = //obtain Hibernate Session from Entitymanager
session.doWork(conn -> {
try (PreparedStatement stmt = conn.prepareCall(queryStr)) {
stmt.setString(1, table);
stmt.execute();
}
});
}

This is something weird ,But Just try, table is a reserved keyword in PostgreSQL. So try changing the variable name.
String queryStr = "REINDEX TABLE :tableName";
Query query = em.createNativeQuery(queryStr);
query.setParameter("tableName", "AUTH_IND");
query.executeUpdate();
From the Documentation :
"select" could be used to refer to a column or table named “select”, whereas an unquoted select would be taken as a key word and would therefore provoke a parse error when used where a table or column name is expected.
https://www.postgresql.org/docs/current/sql-syntax-lexical.html

Related

Spring simplejdbc and Postgresql Function with Table Return type

I am using PostgreSQL with Spring JDBC Template.
I have a postgressql function that returns a TABLE type. As a standalone the function returns number of records based on input.
Table function looks like below (not exact implementation but a sample)
CREATE OR REPLACE FUNCTION
search( p_query varchar(20))
RETURNS table (name varchar(20),
address varchar(100),
salary numeric)
AS $$
select name,
address,
salary
from emp_table
where
upper(name) = upper(p_query);
$$LANGUAGE 'sql';
In DAO class, I am making use of simplejdbccall to execute the function.
Map<String, Object> results = simpleJdbcCall.withSchemaName(DB_SCHEMA_NAME)
.withFunctionName(SEARCH_FUNCTION)
.execute(INPUT_PARAMETERS);
There are no errors. However the results is having only one record(the first record).
So I tried with,
List<Map<String, Object>> results = (List<Map<String, Object>>) simpleJdbcCall.withSchemaName(DB_SCHEMA_NAME)
.withFunctionName(SEARCH_FUNCTION)
.execute(INPUT_PARAMETERS);
Which resulted in ClassCastException as execute returns a Map.
How to invoke PostgreSQL function using spring jdbctemplate instead of direct insert query in Java?
The above solution worked for a single return value but not for table type. Without using Hibernate JPA is it possible to implement this use case.
Note: I have done a similar implementation using Oracle Stored procedure which returns a cursor. Not sure how to make this work with PostgreSQL function. I am a starter
for PostgreSQL.
Instead of simplejdbccall, I used a slightly different approach to get the result
i. Used jdbcTemplate.query and written select statement on the Function
ii. Used Rowmapper to map the results into Bean.
Example below.
List<TableObjectBean> outputList = jdbcTemplate.query("select name,address,salary from search(?)"
,new Object[] {queryParam}
,new RowMapper<TableObjectBean>() {
#Nullable
#Override
public TableObjectBean mapRow(ResultSet resultSet, int i) throws SQLException {
TableObjectBean tableObjectBean = new TableObjectBean();
tableObjectBean.setName(resultSet.getString(1));
tableObjectBean.setAddress(resultSet.getString(2));
tableObjectBean.setSalary(resultSet.getDouble(3));
return tableObjectBean;
}
});

How SpringBoot JPA run DDL sql with dynamic tableName?

Yestody,I got this question, how jpa run DDL sql with dynamic tableName?
usually,I just used DQL and DML like 'select,insert,update,delete'.
such as :
public interface UserRepository extends JpaRepository<User, Integer> {
#Query(value = "select a.* from user a where a.username = ? and a.password = ?", nativeQuery = true)
List<User> loginCheck(String username, String password);
}
but when I required run DDL sql below
String sql = "create table " + tableName + " as select * from user where login_flag = '1'";
I don't find a way to solve this with Jpa (or EntityManager).
Finally I used JDBC to run the DDL sql,but I think it's ugly...
Connection conn = null;
PreparedStatement ps = null;
String sql=" create table " + tableName + " as select * from user where login_flag = '1' ";
try {
Class.forName(drive);
conn = DriverManager.getConnection(url, username, password);
ps = conn.prepareStatement(sql);
ps.executeUpdate();
ps.close();
conn.close();
} catch (Exception e) {
e.printStackTrace();
}
So,can jpa run DDL sql(such as CREATE/DROP/ALTER) wiht dynamic tableName in an easy way?
Your question seems to consist of two parts
The first part
can jpa run DDL sql
Sure, just use entityManager.createNativeQuery("CREATE TABLE ...").executeUpdate(). This is probably not the best idea (you should be using a database migration tool like Flyway or Liquibase for DB creation), but it will work.
Note that you might run into some issues, e.g. different RDBMSes have different requirements regarding transactions around DDL statements, but they can be solved quite easily most of the time.
You're probably wondering how to get hold of an EntityManager when using Spring Data. See here for an explanation on how to create custom repository fragments where you can inject virtually anything you need.
The second part
with dynamic tableName
JPA only supports parameters in certain clauses within the query, and identifiers are not one of them. You'll need to use string concatenation, I'm afraid.
Why dynamic table names, though? It's not like your entity definitions are going to change at runtime. Static DDL scripts are generally less error-prone.

Must declare the scalar variable "##RowCount" on SaveChangesAsync

I have a simple table insert in entity framework to add a record to a Azure SQL Data-Warehouse table. I get this error on context.SaveChanges() - SQLException: Must declare the scalar variable "##ROWCOUNT
Reading a table works perfectly fine only the saving to a table fails.
context.Users.Add(user);
context.SaveChanges(); -> fails here.
Expected result - record should get inserted in the table
Actual result - Microsoft.EntityFrameworkCore.DbUpdateException: 'An error occurred while updating the entries. See the inner exception for details.'
Inner Exception
SqlException: Must declare the scalar variable "##ROWCOUNT".
I found out that EntityFramework is not supported for Azure SQL Data Warehouse. https://feedback.azure.com/forums/307516-sql-data-warehouse/suggestions/12868725-support-for-entity-framework
I used the SqlConnection and SqlCommand as a workaround.
`using (var cn = new SqlConnection(connectionString))
{
var query = "insert into Users([Id]) values (#Id)";
using (var cmd = new SqlCommand(query, cn))
{
cmd.Parameters.AddWithValue("#Id", 1);
cn.Open();
cmd.ExecuteNonQuery();
cn.Close();
}
}`

playframework selecting from h2 database

I am trying downloading all records from table in h2 buildin database in playframework.
I am facing an error:
[IllegalArgumentException: org.hibernate.hql.internal.ast.QuerySyntaxException: unexpected token: * near line 1, column 8 [SELECT * FROM TABLE]]
Method CODE i class Table:
#Transactional(readOnly=true)
public static Result view() {
Query query = JPA.em().createQuery("SELECT * FROM TABLE");
List<Table> downloaded_from_db = query.getResultList();
System.out.println(downloaded_from_db.getClass());
return ok(view.render("none"));
}
Please help me. I would like to see downloaded records in console in simple view.
Please give me some tips or good tutorial.
After changing my class loooks like this:
#Transactional(readOnly=true)
public static Result view() {
List<MedicalIncidents> data = JPA.em()
.createNativeQuery("SELECT * FROM MedicalIncident")
//.createQuery("Select m from MedicalIncident m")
.getResultList();
System.out.println(data);
AND I think it works, cause I have 2 entries in that table in database:
But System.out.println(data) return in plaay console:
[[Ljava.lang.Object;#70a0c9be, [Ljava.lang.Object;#4c1d12b6]
But it should return this object by model name like in example: computer-database-jpa:
[models.Computer#214c6fde, models.Computer#63728eb3, models.Computer#75f6bcc6, models.Computer#19e3a7ab, models.Computer#3114d8d4, models.Computer#4fa75f78, models.Computer#756ce822, models.Computer#40fc4c68, models.Computer#73fc612c, models.Computer#3e4fcb31]
So I think that there is something wrong with it. Please help
You mxied SQL queries with JPQL query. The method you used createQuery needs an JPQL query:
SELECT e FROM Entity e
Also please note in JPQL there is no SELECT *. If you want to write a SQL query, use the method em.createNtiveQuery().

org.postgresql.util.PSQLException: ERROR: unterminated dollar-quoted string at or near "$BODY$

I am trying to use dbunit-express in my java project to create some tables and functions on postgress in Junit tests.
I use this driver :
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>9.2-1003-jdbc4</version>
The java class...
#Rule
public EmbeddedDbTesterRule testDb = new EmbeddedDbTesterRule(); // with this you don't neet to call onSetup
#Test
public void testIt() throws Exception {
try {
DatabaseCreator databaseCreator = new DatabaseCreator();
databaseCreator.setDdlFile("HistoryTables.ddl");
databaseCreator.doCreateDbSchemaFromDdl(testDb.getSqlConnection());
} catch (Exception e) {
Assert.fail(e.getMessage());
}
}
But I get this error...
org.postgresql.util.PSQLException: ERROR: unterminated dollar-quoted string at or near "$BODY$
The function looks like this.
CREATE OR REPLACE FUNCTION product_history()
RETURNS trigger AS
$BODY$
BEGIN
INSERT INTO product_history (id, product_id, edit_ts, name, print_provider_id, description)
VALUES (nextval('product_history_sequence'), OLD.id, now(), OLD.name, OLD.print_provider_id, OLD.description);
RETURN NULL;
END;
$BODY$
LANGUAGE plpgsql;
The create works fine in PGAdmin 1.14.3 and in DBVisualizer 9.0.8
Is the create function code included in HistoryTables.ddl? If yes the error might be caused by a limitation of DatabaseCreator. It splits statements read from the ddl file at ; (see line 127 of the source code. The function is therefore splitted into multiple statements disturbing the syntax.
Try to move the function code into an extra file and send this file as one statement to your server.