I am getting this error while executing transformation in pentaho for execute sql script.
2022/04/04 13:00:18 - Execute SQL script.0 - offending row : [batch_id Integer(9)]
2022/04/04 13:00:18 - Execute SQL script.0 -
2022/04/04 13:00:18 - Execute SQL script.0 - Error setting value #1 [Integer(9)] on prepared statement
2022/04/04 13:00:18 - Execute SQL script.0 - The column index is out of range: 1, number of columns: 0.
2022/04/04 13:00:18 - Execute SQL script.0 -
2022/04/04 13:00:18 - Execute SQL script.0 -
2022/04/04 13:00:18 - Execute SQL script.0 - at org.pentaho.di.core.database.Database.setValues(Database.java:1102)
2022/04/04 13:00:18 - Execute SQL script.0 - at org.pentaho.di.core.database.Database.execStatement(Database.java:1540)
2022/04/04 13:00:18 - Execute SQL script.0 - ... 3 more
2022/04/04 13:00:18 - Execute SQL script.0 - Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
2022/04/04 13:00:18 - Execute SQL script.0 - Error setting value #1 [Integer(9)] on prepared statement
2022/04/04 13:00:18 - Execute SQL script.0 - The column index is out of range: 1, number of columns: 0.
2022/04/04 13:00:18 - Execute SQL script.0 -
2022/04/04 13:00:18 - Execute SQL script.0 - at org.pentaho.di.core.row.value.ValueMetaBase.setPreparedStatementValue(ValueMetaBase.java:5490)
2022/04/04 13:00:18 - Execute SQL script.0 - at org.pentaho.di.core.database.Database.setValue(Database.java:1084)
2022/04/04 13:00:18 - Execute SQL script.0 - at org.pentaho.di.core.database.Database.setValues(Database.java:1100)
2022/04/04 13:00:18 - Execute SQL script.0 - ... 4 more
Here is the code of execute sql script for which I changed value from :1 to ? which is nothing but the column batch_id.even this is working fine in postgres if i replace it with actual id
Do
$$
DECLARE
p_err_no text:='0';
p_err_str text; -- (datatype cannot be varchar)
BEGIN
call DASHBOARD_ETL.update_fact_module_status(
?::int,
p_err_no,
p_err_str);
END;
$$
Related
I am using DACPAC approach for my automatic database deployment on Azure Sql Server database from visual studio 2019,I have added DACPAC database for my local database now problem is that dacpac file is created in CI pipeline and going to release it on Azure SQL database using 'Azure SQL DacpacTask' but their I am facing issue while releasing the dacpac file.
*** Could not deploy package.
Error SQL72014: .Net SqlClient Data Provider: Msg 40515, Level 15, State 1, Procedure ddltrg_CREATE_Activity_LOG, Line 16 Reference to database and/or server name in 'TrackDBChanges..tblDDLEventLog' is not supported in this v
ersion of SQL Server.
Error SQL72045: Script execution error. The executed script:
CREATE TRIGGER [ddltrg_CREATE_Activity_LOG]
ON DATABASE
FOR DDL_DATABASE_LEVEL_EVENTS
AS SET NOCOUNT ON;
BEGIN
DECLARE #xmlEventData AS XML;
SET #xmlEventData = eventdata();
IF CONVERT (VARCHAR (250), #xmlEventData.query('data(/EVENT_INSTANCE/DatabaseName)')) NOT IN ('model')
INSERT INTO TrackDBChanges..tblDDLEventLog (EventTime, EventType, ServerName, DatabaseName, ObjectType, ObjectName, UserName, CommandText, CommandTextXML, HostName, LoginName)
SELECT REPLACE(CONVERT (VARCHAR (MAX), #xmlEventData.query('data(/EVENT_INSTANCE/PostTime)')), 'T', ' '),
CONVERT (VARCHAR (MAX), #xmlEventData.query('data(/EVENT_INSTANCE/EventType)')),
CONVERT (VARCHAR (MAX), #xmlEventData.query('data(/EVENT_INSTANCE/ServerName)')),
CONVERT (VARCHAR (MAX), #xmlEventData.query('data(/EVENT_INSTANCE/DatabaseName)')),
CONVERT (VARCHAR (MAX), #xmlEventDa
1 more errors. Click on expand view in the context menu to view complete logs.
my structure of Dacpac project
I have a spring batch job which copies data from tables from one oracle schema to another.
To do this, I have written a partitioned job.
For example:
Case A:
I have a table A with 100000 rows and I split into 100 steps of 1000 rows each. all these inserts are done in parallel using ThreadPool task executor. If 4 steps failed due to some issue. I restarted the job, it successfully ran the failed 4 steps within seconds as I expected.
Case B:
Say a table A containing 32 million rows is to be copied from source to destination.
So I split this job into steps of 1000 rows each so 32000 steps are created.
Out of these 32000 steps 4 steps fails due to some db issue. When I try to restart this job, Spring batch just hangs or I dont know what processing is going on behind restart it just does not do anything. I waited for 5 hours and then killed the job.
so Can someone help me with what happens behind the restart, how the total number of steps affects the restart ? and what should I do to improve this speed?
Please let me know if any more info is needed.
Updates:
I was able to speed up the Case B by implementing PartitionNameProvider and was returning the names of failed steps alone via getPartitionNames during restart. Great. I was testing this restart with more number of failures and I have one more case now.
Case C:
If I have 20000 steps and all 20000 steps failed. When I try to restart this case. the getPartitionNames returns all 20000 steps. In this case, Im facing the problem I mentioned above. The process hangs.
I tried to understand what was going on behind the job by enabling spring debug logs (which took me so long to discover but worth it). I saw a specific set of queries running. they are:
2019-02-22 06:40:27 DEBUG ResourcelessTransactionManager:755 - Initiating transaction commit
2019-02-22 06:40:27 DEBUG ResourcelessTransactionManager:39 - Committing resourceless transaction on [org.springframework.batch.support.transaction.ResourcelessTransactionManager$ResourcelessTransaction#5743a94e]
2019-02-22 06:40:27 DEBUG ResourcelessTransactionManager:367 - Creating new transaction with name [org.springframework.batch.core.repository.support.SimpleJobRepository.getLastStepExecution]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
2019-02-22 06:40:27 DEBUG JdbcTemplate:682 - Executing prepared SQL query
2019-02-22 06:40:27 DEBUG JdbcTemplate:616 - Executing prepared SQL statement [SELECT JOB_EXECUTION_ID, START_TIME, END_TIME, STATUS, EXIT_CODE, EXIT_MESSAGE, CREATE_TIME, LAST_UPDATED, VERSION, JOB_CONFIGURATION_LOCATION from COPY_JOB_EXECUTION where JOB_INSTANCE_ID = ? order by JOB_EXECUTION_ID desc]
2019-02-22 06:40:27 DEBUG DataSourceUtils:110 - Fetching JDBC Connection from DataSource
2019-02-22 06:40:27 DEBUG DataSourceUtils:114 - Registering transaction synchronization for JDBC Connection
2019-02-22 06:40:27 DEBUG JdbcTemplate:682 - Executing prepared SQL query
2019-02-22 06:40:27 DEBUG JdbcTemplate:616 - Executing prepared SQL statement [SELECT JOB_EXECUTION_ID, KEY_NAME, TYPE_CD, STRING_VAL, DATE_VAL, LONG_VAL, DOUBLE_VAL, IDENTIFYING from COPY_JOB_EXECUTION_PARAMS where JOB_EXECUTION_ID = ?]
2019-02-22 06:40:27 DEBUG JdbcTemplate:682 - Executing prepared SQL query
2019-02-22 06:40:27 DEBUG JdbcTemplate:616 - Executing prepared SQL statement [SELECT STEP_EXECUTION_ID, STEP_NAME, START_TIME, END_TIME, STATUS, COMMIT_COUNT, READ_COUNT, FILTER_COUNT, WRITE_COUNT, EXIT_CODE, EXIT_MESSAGE, READ_SKIP_COUNT, WRITE_SKIP_COUNT, PROCESS_SKIP_COUNT, ROLLBACK_COUNT, LAST_UPDATED, VERSION from COPY_STEP_EXECUTION where JOB_EXECUTION_ID = ? order by STEP_EXECUTION_ID]
2019-02-22 06:40:27 DEBUG JdbcTemplate:682 - Executing prepared SQL query
2019-02-22 06:40:27 DEBUG JdbcTemplate:616 - Executing prepared SQL statement [SELECT STEP_EXECUTION_ID, STEP_NAME, START_TIME, END_TIME, STATUS, COMMIT_COUNT, READ_COUNT, FILTER_COUNT, WRITE_COUNT, EXIT_CODE, EXIT_MESSAGE, READ_SKIP_COUNT, WRITE_SKIP_COUNT, PROCESS_SKIP_COUNT, ROLLBACK_COUNT, LAST_UPDATED, VERSION from COPY_STEP_EXECUTION where JOB_EXECUTION_ID = ? order by STEP_EXECUTION_ID]
2019-02-22 06:40:30 DEBUG JdbcTemplate:682 - Executing prepared SQL query
2019-02-22 06:40:30 DEBUG JdbcTemplate:616 - Executing prepared SQL statement [SELECT SHORT_CONTEXT, SERIALIZED_CONTEXT FROM COPY_STEP_EXECUTION_CONTEXT WHERE STEP_EXECUTION_ID = ?]
2019-02-22 06:40:30 DEBUG JdbcTemplate:682 - Executing prepared SQL query
2019-02-22 06:40:30 DEBUG JdbcTemplate:616 - Executing prepared SQL statement [SELECT SHORT_CONTEXT, SERIALIZED_CONTEXT FROM COPY_JOB_EXECUTION_CONTEXT WHERE JOB_EXECUTION_ID = ?]
2019-02-22 06:40:30 DEBUG DataSourceUtils:327 - Returning JDBC Connection to DataSource
and so on...
I understood spring is executing getLastStepExecution for each failed step one by one. But why is getLastStepExecution done this way? or is there any way we can configure this? like reading all the step executions in bulk and start processing so as to reduce the restart time.
Thanks in advance.
I have created a database link from my 10g database to 12c database. When I am trying to access table using 10g database it works fine. But when we try to execute the procedure from 10g database, it popups following error
SQL> exec popcust#abcd;
BEGIN popcust#abcd; END;
*
ERROR at line 1:
ORA-06550: line 1, column 7:
PLS-00201: identifier 'POPCUST#ABCD' must be declared
ORA-06550: line 1, column 7:
PL/SQL: Statement ignored
When we describe a procedure from 10g DB. it comes with like this.
SQL> desc popcust#abcd;
PROCEDURE st#abcd
Please can somebody help me
I am using flyway for a deployment and the tables built as a result of flyway are all fine.
The issue I have is with the schema_version table. I am unable to query an individual column in the table. I am only able to perform a select *.
The error message I am getting is:
10:35:49 [SELECT - 0 row(s), 0.000 secs] 1) [Error Code: -206, SQL State: 42703] DB2 SQL Error: SQLCODE=-206, SQLSTATE=42703, SQLERRMC=SCRIPT, DRIVER=4.13.127. 2) [Error Code: -727, SQL State: 56098] DB2 SQL Error: SQLCODE=-727, SQLSTATE=56098, SQLERRMC=2;-206;42703;SCRIPT, DRIVER=4.13.127
Try enclosing the column name in double quotes:
select "SCRIPT" from flyway.schema_version
This might help.
Flyway Schema table is designed to be in lower case.
You can change the table name using below config property.
flyway.table=SCHEMA_VERSION
For more details you can check this
https://flywaydb.org/documentation/faq#case-sensitive
Try to make your query like this:
SELECT "version", "installed_on" FROM "schema_version";
I am connecting postgres in 192.168.4.12 I want to insert into table in server IP 192.168.4.13, I have tried but I get error.
ERROR: syntax error at end of input
CONTEXT: Error occurred on dblink connection named "unnamed": could not execute command.
********** Error **********
ERROR: syntax error at end of input
SQL state: 42601
Context: Error occurred on dblink connection named "unnamed": could not execute command.
Here is the syntax.
SELECT dblink_exec
('hostaddr=192.168.4.13 dbname=mrp-pti-coba user=postgres password=passwordb',
'insert into mst_item(item_id) values
(dblink_exec("hostaddr=192.168.4.12 dbname=mrp-exp2 user=postgres password=passworda","select item_id from mst_item")');