I have a fairly complex Db2 V11.1 SP that will compile and deploy manually, but when i add the SQL to a migration script I get this issue
https://github.com/flyway/flyway/issues/2795
As the SP compiles and deploys manually, I am confident the SP SQL is ok.
Does anyone have any idea what the underlying issue might be
DB2 11.1
Flyway 6.4.1 (I have tried 7.x versions with same result)
the SP uses pure xml functions, so the SP SQL includes $ and # characters
I tried using obscure statement terminator chars ( ~ ^) but a simple test with Pure xml functions and # as statement terminator seemed to work
--#SET TERMINATOR #
SET SCHEMA CORE
#
CREATE OR REPLACE PROCEDURE CORE.XML_QUERY
LANGUAGE SQL
BEGIN
DECLARE GLOBAL TEMPORARY TABLE OPTIONAL_ELEMENT (
LEG_SEG_ID BIGINT,
OPTIONAL_ELEMENT_NUM INTEGER,
OPTIONAL_ELEMENT_LIST VARCHAR(100),
CLSEQ INTEGER
) ON COMMIT PRESERVE ROWS NOT LOGGED WITH REPLACE;
insert into session.optional_element
select distinct LEG_SEG_ID, A.OPTIONAL_ELEMENT_NUM, A.OPTIONAL_ELEMENT_LIST, A.CLSEQ
from core.leg_seg , XMLTABLE('$d/LO/O' passing XMLPARSE(DOCUMENT(optional_element_xml)) as "d"
COLUMNS
OPTIONAL_ELEMENT_NUM INTEGER PATH '#Num',
OPTIONAL_ELEMENT_LIST VARCHAR(100) PATH 'text()',
CLSEQ INTEGER PATH '#Seq') AS A
WHERE iv_id = 6497222690 and optional_element_xml is not null;
END
#
I'm trying to get database migrations into my release pipeline definition, roughly following this aproach - dotnet ef migrations script --idempotent as part of the build pipeline, and then an Invoke-SqlCmd task pointing to the resulting script as part of the release pipeline.
However, I have made one change from that blog post, that might be of importance: in order to avoid corrupt states where half a migration is deployed, I wrap the entire script in a transaction, so that it is effectively
SET XACT_ABORT ON
BEGIN TRANSACTION
-- output of dotnet ef migrations script --idempotent
COMMIT
However, when I try to run the script as part of my release pipeline, in an Azure SQL Database task just as the one he uses in the blog post, it fails on e.g. references to things that don't exist in the schema yet, but will by the point the script reaches the part that includes the reference.
For example, considering the following migrations script:
SET XACT_ABORT ON
BEGIN TRANSACTION
-- in the actual script, each of these statements are, individually, wrapped in an IF that
-- checks whether the migration has been run before or not, similar to the first one in
-- this listing
IF NOT EXISTS (SELECT * FROM [__EFMigrationsHistory] WHERE MigrationId = 'AddColumnBar')
BEGIN
ALTER TABLE Foo ADD Bar int NULL
END
GO
UPDATE Foo SET Bar = 3
GO
ALTER TABLE Foo ALTER COLUMN Bar int NOT NULL
GO
COMMIT
Executing this script with Invoke-SqlCmd as suggested in the blog post yields an error stating that the Foo column does not exist.
How do I tell Invoke-SqlCmd that I know it doesn't, but it will when it needs to? If that's not possible, what's the best approach to fix this deployment flow for Azure DevOps?
Additional info:
Here's the complete log from the SQL step in the release pipeline:
2019-01-09T14:57:52.7983184Z ##[section]Starting: Apply EF Migrations
2019-01-09T14:57:52.7989024Z ==============================================================================
2019-01-09T14:57:52.7989311Z Task : Azure SQL Database Deployment
2019-01-09T14:57:52.7989427Z Description : Deploy Azure SQL DB using DACPAC or run scripts using SQLCMD
2019-01-09T14:57:52.7989514Z Version : 1.2.9
2019-01-09T14:57:52.7989608Z Author : Microsoft Corporation
2019-01-09T14:57:52.7989703Z Help : [More Information](https://aka.ms/sqlazuredeployreadme)
2019-01-09T14:57:52.7989823Z ==============================================================================
2019-01-09T14:57:58.8012013Z Sql file: D:\a\r1\a\_db-migrations\migrations-with-transaction.sql
2019-01-09T14:57:58.8189093Z Invoke-Sqlcmd -ServerInstance "***" -Database "***" -Username "***" -Password ****** -Inputfile "D:\a\r1\a\db-migrations\migrations-with-transaction.sql" -ConnectionTimeout 120
2019-01-09T14:58:04.3140758Z ##[error]Invalid column name 'Group_DocumentTagGroupId'.Check out how to troubleshoot failures at https://aka.ms/sqlazuredeployreadme#troubleshooting-
2019-01-09T14:58:04.3480044Z ##[section]Finishing: Apply EF Migrations
Here are all the references to Group_DocumentTagGroupId in the script:
-- starting around line 1740
IF NOT EXISTS(SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'20181026122735_AddEntityForDocumentTagGroup')
BEGIN
ALTER TABLE [DocumentTags] ADD [Group_DocumentTagGroupId] bigint NOT NULL DEFAULT 0;
END;
GO
-- about 20 lines with no mention of the column (but several GO statements)
IF NOT EXISTS(SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'20181026122735_AddEntityForDocumentTagGroup')
BEGIN
EXEC('
UPDATE [dbo].[DocumentTags] SET [Group_DocumentTagGroupId] = (SELECT TOP 1 [DocumentTagGroupId] FROM [dbo].[DocumentTagGroups] g WHERE g.[Name] = [Type])')
END;
GO
IF NOT EXISTS(SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'20181026122735_AddEntityForDocumentTagGroup')
BEGIN
CREATE INDEX [IX_DocumentTags_Group_DocumentTagGroupId] ON [DocumentTags] ([Group_DocumentTagGroupId]);
END;
GO
IF NOT EXISTS(SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'20181026122735_AddEntityForDocumentTagGroup')
BEGIN
ALTER TABLE [DocumentTags] ADD CONSTRAINT [FK_DocumentTags_DocumentTagGroups_Group_DocumentTagGroupId] FOREIGN KEY ([Group_DocumentTagGroupId]) REFERENCES [DocumentTagGroups] ([DocumentTagGroupId]) ON DELETE CASCADE;
END;
GO
The SQL file that is sent to Invoke-SqlCmd is generated with the following PowerShell script
$migrationsWithoutTransaction = "./Migrations/scripts/migrations-without-transaction.sql"
dotnet ef migrations script --configuration Release --idempotent --output $migrationsWithoutTransaction
$migrationsWithTransaction = "./Migrations/scripts/migrations-with-transaction.sql"
Get-Content "./Migrations/begin-transaction.sql" | Out-File -Encoding Utf8 $migrationsWithTransaction
Get-Content $migrationsWithoutTransaction | Out-File -Encoding Utf8 -Append $migrationsWithTransaction
Get-Content "./Migrations/commit-transaction.sql" | Out-File -Encoding Utf8 -Append $migrationsWithTransaction
where this is the contents of the auxiliary sql scripts:
-- Migrations/begin-transaction.sql
SET XACT_ABORT ON
BEGIN TRANSACTION
-- Migrations/end-transaction.sql
COMMIT
Is using Azure Release pipeline (classic) an option for you? Because I am successfully running EF migrations on it, while creating the migrations script on the fly.
See my answer here: how to execute sql script using azure devops pipeline
I am trying to build View PostgreSQL table in Liquibase for my JHipster application. Therefore, I have tried createView and sqlFile methods.
My query has date_trunc() function as following:
CREATE OR REPLACE VIEW periodical_statistics AS
SELECT tran.id, date_trunc('day', tran.pub_date) as, ...
FROM transaction tran
...(LEFT JOINS - not relative)...
WHERE ...(condtions - not relative)...
When I run my JHispster app as Dev Mode in maven(./mvnw). It is working ver fine.
But when I run it as Prod Mode in maven(./mvnw -Pprod package). It gives following error.
20180817101122-1::liquibase-docs failed. Error: Function "DATE_TRUNC" not found;
SQL statement: CREATE OR REPLACE VIEW periodical_statistics AS ....
is there are any way to solve this error problem?
After adding dbms='postgresql' to the changeset. Liquibase recognized the 'date_trunc' function. It is as following:
<changeSet author="sanatbek" id="20180904094713" dbms="postgresql">
<createView replaceIfExists="true"
schemaName="public"
viewName="periodical_statistics">
SELECT
tran.id
date_trunc('day', tran.pub_date) as truncated_date,
....
...(LEFT JOINS - not relative)...
WHERE ...(condtions - not relative)...
</createView>
</changeSet>
First post, so I'll get right to it. Thank you in advance for your answers and consideration.
I have full privileges on the database engine that the DB in question is running on, including sysadmin.
To the best of my knowledge, I have enabled this correctly according to documentation, doing the following:
Running the command EXEC sys.sp_cdc_enable_db via a c# application
that I am using as an interface to deal with setting up, recording,
and comparing DML database changes.
From the same application, running the command
EXEC sys.sp_cdc_enable_table
#source_schema = N'dbo',
#source_name = N'ORD_ATTACHMENTS',
#role_name = NULL
I have verified that the DB in question is ready for CDC using SELECT [name], database_id, is_cdc_enabled FROM sys.databases.
The table's readiness I have also verified using SELECT [name], is_tracked_by_cdc FROM sys.tables.
Running SELECT * FROM [msdb].[dbo].[cdc_jobs] WHERE [database_id] = DB_ID() in the database context yields the following information for the capture job:
maxtrans: 7200
maxscans: 10
continuous: 1
pollinginterval: 5
retention and threshold are 0.
After inserting records into the table in question via SSMS, the related CDC table, though present, does not have any data in it. No errors were encountered, and the record was successfully added to the source table.
Additional information:
Database server used to use Windows fibers (lightweight pooling). I
have switched this off, reconfigured, and rebooted the server.
Database used to have compatibility set to SQL Server 2005 (90), but
I have updated this to SQL Server 2008 (100). Again rebooted the
server.
I also set the Change Tracking property to true for the
database in question, but I have since learned that this is
irrelevant.
The source table has the following fields:
[AttachmentID] [bigint] IDENTITY(1,1) NOT NULL,
[ORDNUM] [nvarchar](10) NOT NULL,
[FileName] [nvarchar](260) NOT NULL,
[FileContent] [varbinary](max) NULL,
[CreatedOn] [datetime] NOT NULL CONSTRAINT [DF_ORD_ATTACHMENTS_CreatedOn] DEFAULT (getdate())
No fields are excluded from CDC for this table.
Thank you in advance for any assistance.
Best regards,
Chris.
Update 2016-09-20 15:15:
Ran the following:
sp_configure 'show advanced options', 1;
GO
RECONFIGURE;
GO
sp_configure 'Agent XPs', 1;
GO
RECONFIGURE
GO
Have now switched to a test DB to simplify matters. Re-enabled the CDC on my new test table (fields are bigint PK identity field and an NVARCHAR(50) nullable field). Still not working. Also, the capture job has no history entries under SQL Server Agent.
Update 2016-09-20 20:09
Ran sp_MScdc_capture_job in the DB context. This can be, depending on job settings, a continuously executing procedure. Data was found in the CDC table upon running this. Will try to figure out how to automatically engage this.
Update 2016-09-28 17:19
The capture job is scripted as follows:
USE [msdb]
GO
/****** Object: Job [cdc.CDCTest_capture] Script Date: 2016-09-28 5:18:13 PM ******/
BEGIN TRANSACTION
DECLARE #ReturnCode INT
SELECT #ReturnCode = 0
/****** Object: JobCategory [REPL-LogReader] Script Date: 2016-09-28 5:18:13 PM ******/
IF NOT EXISTS (SELECT name FROM msdb.dbo.syscategories WHERE name=N'REPL-LogReader' AND category_class=1)
BEGIN
EXEC #ReturnCode = msdb.dbo.sp_add_category #class=N'JOB', #type=N'LOCAL', #name=N'REPL-LogReader'
IF (##ERROR <> 0 OR #ReturnCode <> 0) GOTO QuitWithRollback
END
DECLARE #jobId BINARY(16)
EXEC #ReturnCode = msdb.dbo.sp_add_job #job_name=N'cdc.CDCTest_capture',
#enabled=1,
#notify_level_eventlog=2,
#notify_level_email=0,
#notify_level_netsend=0,
#notify_level_page=0,
#delete_level=0,
#description=N'CDC Log Scan Job',
#category_name=N'REPL-LogReader',
#owner_login_name=N'sa', #job_id = #jobId OUTPUT
IF (##ERROR <> 0 OR #ReturnCode <> 0) GOTO QuitWithRollback
/****** Object: Step [Starting Change Data Capture Collection Agent] Script Date: 2016-09-28 5:18:14 PM ******/
EXEC #ReturnCode = msdb.dbo.sp_add_jobstep #job_id=#jobId, #step_name=N'Starting Change Data Capture Collection Agent',
#step_id=1,
#cmdexec_success_code=0,
#on_success_action=3,
#on_success_step_id=0,
#on_fail_action=3,
#on_fail_step_id=0,
#retry_attempts=10,
#retry_interval=1,
#os_run_priority=0, #subsystem=N'TSQL',
#command=N'RAISERROR(22801, 10, -1)',
#server=N'AECON-SQL',
#database_name=N'CDCTest',
#flags=4
IF (##ERROR <> 0 OR #ReturnCode <> 0) GOTO QuitWithRollback
/****** Object: Step [Change Data Capture Collection Agent] Script Date: 2016-09-28 5:18:14 PM ******/
EXEC #ReturnCode = msdb.dbo.sp_add_jobstep #job_id=#jobId, #step_name=N'Change Data Capture Collection Agent',
#step_id=2,
#cmdexec_success_code=0,
#on_success_action=1,
#on_success_step_id=0,
#on_fail_action=2,
#on_fail_step_id=0,
#retry_attempts=10,
#retry_interval=1,
#os_run_priority=0, #subsystem=N'TSQL',
#command=N'sys.sp_MScdc_capture_job',
#server=N'AECON-SQL',
#database_name=N'CDCTest',
#flags=4
IF (##ERROR <> 0 OR #ReturnCode <> 0) GOTO QuitWithRollback
EXEC #ReturnCode = msdb.dbo.sp_update_job #job_id = #jobId, #start_step_id = 1
IF (##ERROR <> 0 OR #ReturnCode <> 0) GOTO QuitWithRollback
EXEC #ReturnCode = msdb.dbo.sp_add_jobschedule #job_id=#jobId, #name=N'CDC capture agent schedule.',
#enabled=1,
#freq_type=64,
#freq_interval=0,
#freq_subday_type=0,
#freq_subday_interval=0,
#freq_relative_interval=0,
#freq_recurrence_factor=0,
#active_start_date=20160920,
#active_end_date=99991231,
#active_start_time=0,
#active_end_time=235959,
#schedule_uid=N'd1fc7d85-c051-4b24-af84-5505308caaf0'
IF (##ERROR <> 0 OR #ReturnCode <> 0) GOTO QuitWithRollback
EXEC #ReturnCode = msdb.dbo.sp_add_jobserver #job_id = #jobId, #server_name = N'(local)'
IF (##ERROR <> 0 OR #ReturnCode <> 0) GOTO QuitWithRollback
COMMIT TRANSACTION
GOTO EndSave
QuitWithRollback:
IF (##TRANCOUNT > 0) ROLLBACK TRANSACTION
EndSave:
GO
Chris,
When you enable CDC at the DB and table level, a number of further objects are created underneath the cdc schema. Of importance are the various functions, the _CT table, and the two jobs cdc.XXXX_capture & cdc.XXXX_cleanup (where XXXX is the full name of the database).
From your description thus far, especially considering the latest update, the error appears to possibly be with the jobs themselves.
At the outset, and it might sound obvious, but do you have SQL Agent running on this instance? I only ask because it is not mentioned in your initial description.
If it is already running, then you will need to get in a little deeper.
If you navigate to your SQL Agent/Jobs folder (under SSMS), locate the capture job, right click and request it to be scripted, you should find the following.
4 calls:
sp_add_job #job_name=N'cdc.XXXX_capture'
sp_add_jobstep #step_name=N'Starting Change Data Capture Collection Agent'
sp_add_jobstep #step_name=N'Change Data Capture Collection Agent'
sp_add_jobschedule #name=N'CDC capture agent schedule.'
The second of those sp_add_jobstep calls is the one that executes the same code you indicates above, #command=N'sys.sp_MScdc_capture_job'.
You can attempt to kick the job off manually to see if that kicks it into life, or, at the very least, provides some data into the _CT table.
In addition, check the last of those calls above, the schedule, sp_add_jobschedule. This should also be enabled, with #freq_type=64 (to ensure it starts when the Agent starts).
Please provide the results of what you find in the response to assist further troubleshooting.
Thanks,
The problem turned out to be that the SQL Server Agent job was not running, although SQL Server indicated that it was. Turned this service on in the services console and was able to capture data in CDC.
Special thanks to LogicalMan who very patiently worked with me through all of this.
I have several databases named very similar (my-db-1, my-db-2, my-db-3, my-db-4). I want to execute the same stored procedure on each of these databases. I decided to use cursors. However, I am getting some strange issues. First here is my simple code that I am executing through SQL Server Management Studio 2008.
DECLARE #db_cursor CURSOR
DECLARE #name varchar(255)
DECLARE #Sql nvarchar(4000)
SET #db_cursor = CURSOR FOR
SELECT name FROM sys.databases WHERE name LIKE 'my-db-%'
OPEN #db_cursor
FETCH NEXT FROM #db_cursor INTO #name
WHILE ##FETCH_STATUS = 0
BEGIN
SET #Sql = 'Use [' + #name + ']; PRINT DB_NAME();'
exec sp_sqlexec #Sql
FETCH NEXT FROM #db_cursor INTO #name
END
CLOSE #db_cursor
DEALLOCATE #db_cursor
Executing this multiple times in a row within 2 seconds, I get strange results:
Execution1:
my-db-1
my-db-2
my-db-3
my-db-4
Execution2:
my-db-1
my-db-2
Execution3:
my-db-1
my-db-2
my-db-3
my-db-4
Execution4:
my-db-1
It seems like its completely random. Sometimes I'll get all 4 databases to print after 10 executions. Sometimes after just 2 executions only 1 database will get printed.
This SQL is executing on Microsoft SQL Server 2008 R2 (RTM) - 10.50.1600.1 (X64) Apr 2 2010 15:48:46 Copyright (c) Microsoft Corporation Developer Edition (64-bit) on Windows NT 6.1 (Build 7600: ) through Microsoft SQL Server Management Studio 10.50.1600.1
Does anyone have any ideas?
Try declaring your cursor as FAST_FORWARD.
The default is an updateable cursor and these required update locks probably conflict with another process accessing sys.databases
Ref.: DECLARE CURSOR