I have the following interface which is connected to a PostgreSQL database:
public interface ExampleDAO extends JpaRepository<Example, Integer>
{
#Modifying
#Query(value="COPY my_table FROM :filename CSV HEADER", nativeQuery=true)
public int copyMyTable(#Param("filename") String filename);
}
But when I run it I get
org.springframework.dao.InvalidDataAccessResourceUsageException: could not execute statement; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: could not execute statement
I remove the #Param and hardcode the filename in and it works fine. The filename passed is:
copyMyTable("C:\\path\\to\\my\\file.csv");
I am totally at a loss on this one!
Judging from this thread Postgres doesn't support bind parameters in this kind of statement.
Related
I am using a third party library to perform mass inserts into a database PgBulkInsert . It takes inserts that would normally take 30 minutes and performs them in 30 seconds. We have noticed that overtime there is disk usage leakage, but we figure out that performing a table reindex appears to corrcect the issue. I am trying to use my JPA Entity Manager to perform a native update. The below code works but contains potential SQL injection vulnerability.
#Stateless
public class ReindexService {
#PersistenceContext(unitName = "my-ds")
private EntityManager em;
public void reindexTable(String table) {
String queryStr = "REINDEX TABLE " + table;
Query query = em.createNativeQuery(queryStr);
query.executeUpdate();
}
}
When I pass in string "alert" to index the alert table it yields the following SQL output
/* dynamic native SQL query */ REINDEX TABLE alert
When I attempt to use a positional parameter it yields a SQL error
String queryStr = "REINDEX TABLE ?";
Query query = em.createNativeQuery(queryStr);
query.setParameter(1, table);
query.executeUpdate();
This yields the following error output
/* dynamic native SQL query */ REINDEX TABLE ?
SQL Error: 0, SQLState: 42601
ERROR: syntax error at or near "$1"
Position: 46
I get a similar error when I try to use a name parameter
String queryStr = "REINDEX TABLE :table";
Query query = em.createNativeQuery(queryStr);
query.setParameter("table", table);
query.executeUpdate();
This yields the same error
/* dynamic native SQL query */ REINDEX TABLE ?
SQL Error: 0, SQLState: 42601
ERROR: syntax error at or near "$1"
Position: 46
Does anyone know how I can call a the native Postgresql reindex table command using my entity manager in a manner without adding a SQL injection vulnerability? I am using Hibernate 5.3.6.Final but would prefer a non-implementation specific solution.
I also tried to access the Connection and perform a JDBC call and it seems to give the error
final Session session = //get session from entity manager
session.doWork(conn -> {
try (PreparedStatement stmt = conn.prepareCall(REINDEX TABLE ?)) {
stmt.setString(1, table);
stmt.execute();
}
});
Yields the same errors as above
SQL Error: 0, SQLState: 42601
ERROR: syntax error at or near "$1"
Position: 15
Identifiers can't be passed as parameters. If you don't expect the table name to come from user input (it sounds a bit strange), you may try using an enumeration of all the tables which you want to reindex, and pass enumeration values to your service (and just concatenate strings).
If you do expect table names to come from untrusted sources, you can try enclosing identifier with double-quotes and escaping existing double-quotes.
There is also a function quote_ident in PostgreSQL which can be used to quote identifiers properly. So you can create a stored procedure which takes a regular argument from your JPA code and uses quote_ident and EXECUTE a constructed query
.
Our workaround was to create a Database Function and call it using a native query
The database function
CREATE OR REPLACE FUNCTION reindex_table(table_in text)
RETURNS void
SECURITY DEFINER
AS $$
BEGIN
EXECUTE FORMAT('REINDEX (VERBOSE) TABLE %I', table_in);
RETURN;
END;
$$ LANGUAGE plpgsql;
Here is the Service code for calling the function
public void reindexTable(String table) {
String queryStr = "select reindex_table(?)";
final Session session = //obtain Hibernate Session from Entitymanager
session.doWork(conn -> {
try (PreparedStatement stmt = conn.prepareCall(queryStr)) {
stmt.setString(1, table);
stmt.execute();
}
});
}
This is something weird ,But Just try, table is a reserved keyword in PostgreSQL. So try changing the variable name.
String queryStr = "REINDEX TABLE :tableName";
Query query = em.createNativeQuery(queryStr);
query.setParameter("tableName", "AUTH_IND");
query.executeUpdate();
From the Documentation :
"select" could be used to refer to a column or table named “select”, whereas an unquoted select would be taken as a key word and would therefore provoke a parse error when used where a table or column name is expected.
https://www.postgresql.org/docs/current/sql-syntax-lexical.html
Yestody,I got this question, how jpa run DDL sql with dynamic tableName?
usually,I just used DQL and DML like 'select,insert,update,delete'.
such as :
public interface UserRepository extends JpaRepository<User, Integer> {
#Query(value = "select a.* from user a where a.username = ? and a.password = ?", nativeQuery = true)
List<User> loginCheck(String username, String password);
}
but when I required run DDL sql below
String sql = "create table " + tableName + " as select * from user where login_flag = '1'";
I don't find a way to solve this with Jpa (or EntityManager).
Finally I used JDBC to run the DDL sql,but I think it's ugly...
Connection conn = null;
PreparedStatement ps = null;
String sql=" create table " + tableName + " as select * from user where login_flag = '1' ";
try {
Class.forName(drive);
conn = DriverManager.getConnection(url, username, password);
ps = conn.prepareStatement(sql);
ps.executeUpdate();
ps.close();
conn.close();
} catch (Exception e) {
e.printStackTrace();
}
So,can jpa run DDL sql(such as CREATE/DROP/ALTER) wiht dynamic tableName in an easy way?
Your question seems to consist of two parts
The first part
can jpa run DDL sql
Sure, just use entityManager.createNativeQuery("CREATE TABLE ...").executeUpdate(). This is probably not the best idea (you should be using a database migration tool like Flyway or Liquibase for DB creation), but it will work.
Note that you might run into some issues, e.g. different RDBMSes have different requirements regarding transactions around DDL statements, but they can be solved quite easily most of the time.
You're probably wondering how to get hold of an EntityManager when using Spring Data. See here for an explanation on how to create custom repository fragments where you can inject virtually anything you need.
The second part
with dynamic tableName
JPA only supports parameters in certain clauses within the query, and identifiers are not one of them. You'll need to use string concatenation, I'm afraid.
Why dynamic table names, though? It's not like your entity definitions are going to change at runtime. Static DDL scripts are generally less error-prone.
My code is running fine in Postgres, but when I switched to Greenplum, the following exception occurs：
Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: lastval() not supported
Error Code: 0
Call: select lastval()
Query: ValueReadQuery(name="SEQ_GEN_IDENTITY" sql="select lastval()")
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1620)
...
at org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:874)
at org.eclipse.persistence.internal.jpa.QueryImpl.performPreQueryFlush(QueryImpl.java:967)
at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:207)
at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:521)
at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:400)
at module.AuthREST.login(AuthREST.java:103)
...
Caused by: org.postgresql.util.PSQLException: ERROR: lastval() not supported
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:555)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:302)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:1009)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:644)
This is User model class:
public class User implements Serializable {
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Basic(optional = false)
#Column(name = "id", nullable = false)
private Long id;
...
}
This is the code in AuthREST class:
...
#POST
#Path("login")
#Consumes({"application/x-www-form-urlencoded"})
public Response login(#Context HttpServletRequest request,
#Context HttpServletResponse response,
#FormParam("username") String username,
#FormParam("password") String password) {
request.login(username, password);
...
}
...
So how should i do avoid the exception in Greenplum?
The way sequences work in Greenplum:
the master has a "sequence server" running (if you look into the process list, you see a separate process)
every time a segment needs the next value from a sequence, it connects to the master and asks for the next value (this is overhead, by the way: http://engineering.pivotal.io/post/SERIAL_Datatype_Performance_in_Greenplum_Database/ )
The underlaying problem: the sequence server does not keep tab on the requests from the segment servers, therefore it does not know the last value (or values) per segment. Hence lastval() cannot be answered.
Even if the sequence server keeps a log of assigned sequence values: the way your query works, lastval() is executed on the master - but the master never actually inserted any data.
tl;dr: Retrieving the last value of a sequence in Greenplum is problematic. Even if lastval() is supported, the returned answer is likely not what you are looking for.
Reference your sequence table using sql last_value column. You can create your own last value function.
Example:
Select last_value from wcdl_dictionary.sq_dic_table_definition_table_id
Using Spring data JPA, trying to create a generic method to upload csv data to database using Load Data mysql. The below code is my attempt. The problem is, the second parameter tablename is getting replaced with quotes. Eg: LOAD DATA LOCAL INFILE Client.csv INTO TABLE 'Client' FIELDS
Any idea on how to overcome this ??
#NoRepositoryBean public interface CustomJpaRepository<T, ID extends Serializable> extends JpaRepository<T, ID> {
#Query(value = "LOAD DATA LOCAL INFILE :filepath INTO TABLE :tablename FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'", nativeQuery = true)
public void bulkInsertUsingCLI(#Param("filepath") String filepath, #Param("tablename") String tablename);
}
I'm using Spring JDBC. Is a simple way to get last inserted ID using Spring Framework or i need to use some JDBC tricks ?
jdbcTemplate.update("insert into test (name) values(?)", params, types);
// last inserted id ...
I found something like below, but i get: org.postgresql.util.PSQLException: Returning autogenerated keys is not supported.
KeyHolder keyHolder = new GeneratedKeyHolder();
jdbcTemplate.update(new PreparedStatementCreator() {
#Override
public PreparedStatement createPreparedStatement(
Connection connection) throws SQLException {
PreparedStatement ps = connection.prepareStatement("insert into test (name) values(?)", new String[] {"id"});
ps.setString(1, "test");
return ps;
}
}, keyHolder);
lastId = (Long) keyHolder.getKey();
The old/standard way is to use call currval() after the insert (ref). Simple and secure.
Support for "generated keys for PreparedStatements" started only since PostgreSql Ver 8.4-701.