My scenario is to export data from the real DB2 and then load it back for integration tests purpose. I am using liquibase (v3.5.3) to manage it.
I have found that TIMESTAMP values are changed during this cycle. When timestamp value is exported I can see it in changelog file as "2018-06-28 22:47:38.816343". After that when I load it back into DB2 it becomes "2018-06-28 23:01:14.343".
The reason is that the "816343" part is treated not as a part of a second but rather as milliseconds amount and that amount is added to the result timestamp.
In the tests a business decision criteria is made by comparing of those timestamps. I need them to be equal.
Any thoughts and proposals will be appreciated.
There are steps to reproduce:
1. Create a file "01_test_data_to_setup.xml" with content
<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd">
<changeSet author="OK" id="1">
<createTable tableName="TT">
<column name="TS_LOADED" type="TIMESTAMP"/>
<column name="TS_GENERATED" type="TIMESTAMP" defaultValueComputed="CURRENT_TIMESTAMP">
<constraints nullable="false"/>
</column>
</createTable>
</changeSet>
<changeSet author="OK" id="2">
<insert tableName="TT">
<column name="TS_LOADED"/>
</insert>
</changeSet>
</databaseChangeLog>
2. Execute above changelog file with liquibase update command
liquibase.bat --driver=com.ibm.db2.jcc.DB2Driver --logLevel=info --classpath=~jdbc driver path here~ --changeLogFile=01_test_data_to_setup.xml --url=jdbc:db2:~jdbc url here~ --defaultSchemaName=~schema name here~ --username=~user name here~ --password=~password here~ update
3. Export data from DB
liquibase.bat --driver=com.ibm.db2.jcc.DB2Driver --logLevel=info --classpath=~jdbc driver path here~ --changeLogFile=db2_exported_test_data.xml --url=jdbc:db2:~jdbc url here~ --defaultSchemaName=~schema name here~ --username=~user name here~ --password=~password here~ --diffTypes="data" generateChangeLog --includeObjects="table:TT"
As result you will have "db2_exported_test_data.xml" file
<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd">
<changeSet author="OK (generated)" id="1530226079041-1">
<insert tableName="TT">
<column name="TS_LOADED"/>
<column name="TS_GENERATED" valueDate=~your generated timestamp value here. in my case it's "2018-06-28 22:47:38.816343"/>
</insert>
</changeSet>
</databaseChangeLog>
4. A file to load exported timestamp back to DB "02_test_data_to_load.xml"
<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd">
<changeSet author="OK" id="3">
<insert tableName="TT">
<column name="TS_LOADED" valueDate=~value in TS_GENERATED from previous step, i.e."2018-06-28 22:47:38.816343"~/>
</insert>
</changeSet>
</databaseChangeLog>
Liquibase command
liquibase.bat --driver=com.ibm.db2.jcc.DB2Driver --logLevel=info --classpath=~jdbc driver path here~ --changeLogFile=02_test_data_to_load.xml --url=jdbc:db2:~jdbc url here~ --defaultSchemaName=~schema name here~ --username=~user name here~ --password=~password here~ update
5. Check exported and loaded timestamp in DB or export TT table data one more time.
TS_LOADED timestamp value in second row will be different from TS_GENERATED value in first row
I would guess this is a bug in Liquibase similar to this one
https://liquibase.jira.com/browse/CORE-1958
Or your datatype is TIMESTAMP(3) on the target table
Related
I use liquibase and I want to create enum in my xml file (PostgreSQL).
As an example below changelog file (only a piece of the file) that creates new table:
<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-2.0.xsd
http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd">
<changeSet id="r3.3.0_table_creation_research_document "
author="anonim">
<createTable tableName="research_document">
<column name="id" type="bigint">
<constraints primaryKey="true"
primaryKeyName="research_document_pkey" nullable="false" />
</column>
I can't find any info/examples in Internet!
I don't think liquibase supports enum for postgres natively in the xml format. However, since it is possible in postgres, you could always use liquibase's formatted sql instead of xml:
--liquibase formatted sql
--changeset ronak:1
CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy');
CREATE TABLE person (
name text,
current_mood mood
);
--rollback drop table person;
I am using Liquibase version 4.0.0 to deploy DB migration scripts in PostgreSQL.
I use a master changelog file and it looks like the following.
<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd">
<includeAll path="4.460.7" relativeToChangelogFile="true"/>
</databaseChangeLog>
My project structure looks like the following.
I use a docker container to run liquibase update in my Jenkinsfile as the following.
docker run --rm -v /home/jenkins/workspace/mate_DB_Migration_Scripts_master:/liquibase/changelog liquibase/liquibase --url="jdbc:postgresql://host:5432/postgres?currentSchema=schema1" --changeLogFile=../liquibase/changelog/changelog.xml --username=postgres --password=some_password update
docker run --rm -v /home/jenkins/workspace/mate_DB_Migration_Scripts_master:/liquibase/changelog liquibase/liquibase --url="jdbc:postgresql://host:5432/postgres?currentSchema=schema2" --changeLogFile=../liquibase/changelog/changelog.xml --username=postgres --password=some_password update
It runs the update just fine. The issue is I can see 2 rows in the databasechangelog file as the following for the same changeset.
Does anyone know why this happens? Please let me know if you want any other information to resolve this.
It looks like there is an issue with the Liquibase Docker image.
More specifically with the combination of v4.0.0 and the "includeAll" tag.
As a workaround you can try the "include" tag instead, but you will have to include every single file.
like this:
<include file="path/to/<filename>.sql" relativeToChangelogFile="true"/>
You can also try the sqlFile tag like this:
<changeSet author="SteveZ" id="external-sql+rollback-script-example" context="QA" labels="Jira1000">
<sqlFile dbms="mysql" splitStatements="true" endDelimiter="//" stripComments="true" path="objects/function/myFunction.sql"/>
<rollback>
<sqlFile dbms="mysql" splitStatements="true" endDelimiter="//" stripComments="true" path="objects/function/myFunction_rollback.sql"/>
</rollback>
</changeSet>
I would like to create or replace a view on DB2 using liquibase and its changeSet tag: XML Sample
This is what I include in the changelog.xml file:
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
logicalFilePath="lon-service-mpd/gin/15.100/15.100.0.0.changelog.xml"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.1.xsd">
<changeSet author="mas-gin-gestion-echelon-service" id="create-view-from-table-periodeavancement-type-personnel">
<createView schemaName="GIN" viewName="V_PERIODEAVANCEMENT_1">select IDPERIODE, CAMPAGNETA from GIN.PERIODEAVANCEMENT</createView>
</changeSet>
</databaseChangeLog>
However, during the creation of the view, DB2 returns the following error liquibase.exception.DatabaseException: DB2 SQL Error: SQLCODE=-206, SQLSTATE=42703
I do not find the way to fix the problem, even if the SQL syntax is correct.
I have fixed the problem by calling directly a sql file instead of using the XML sample. Here is my solution:
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
logicalFilePath="mas-gin-gestion-echelon-service-mpd/gin/15.100/15.100.0.0.changelog.xml"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.1.xsd">
<changeSet id="create_view_periodeavancement" author="mas-gin-gestion-echelon-service">
<sqlFile path="sql/create_view_periodeavancement.sql" relativeToChangelogFile="true"/>
</changeSet>
And the sql file:
CREATE OR REPLACE VIEW GIN.V_PERIODEAVANCEMENT_1 (IDPERIODE, TS_INSERT, BL_DELETE )
AS SELECT IDPERIODE, TS_INSERT, BL_DELETE
FROM PERIODEAVANCEMENT;
I've got a postgresql database that is created from liquibase. How can I configure liquibase to create CITEXT columns?
I ran across this URL but unsure where to put this configuration. I'm using version 3.3.2.
https://liquibase.jira.com/browse/CORE-1234
Thanks for any help.
You will need to edit the changelog file. You didn't specify what format your changelog is in, so this assumes XML, but the instructions are similar for other formats.
Basically, if you have a column that should be CITEXT, you can use that type in the XML declaration:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog">
<changeSet author="Pete" id="1430490076262-3216" objectQuotingStrategy="QUOTE_ALL_OBJECTS">
<createTable tableName="AA_BUILD">
<column name="STAGE" type="NUMBER">
<constraints nullable="false"/>
</column>
<column name="ITERATION" type="NUMBER">
<constraints nullable="false"/>
</column>
<column name="BUILD_NUM" type="NUMBER">
<constraints nullable="false"/>
</column>
<column name="CHANGED_IN_DEV_DT" type="date"/>
<column name="DESCRIPTION" type="CITEXT"/>
</createTable>
</changeSet>
I have a pipeline question.
I have a workflow where I have a master XML that references multiple XML files. It would be similar to a DITA map, but this is not DITA.
I am having to run an XSLT on the master XML to prepare it for a downstream process for output to the web or other medium.
Part of this transform includes resolving database file paths to relative file paths within the package that is exported from the content management system.
I can perform the transformation on the master XML just fine. However, my question involves running a XSLT on the referenced XML files. They also need the paths corrected.
Here is a simple sample...
<master>
<reference url="x-database01.xml"/>
</master>
The reference will resolve to something like url="/files/realXMLname.xml". The problem is that I don't know how to then transform realXMLname.XML to resolve the paths there.
<content>
<graphic href="x-database02.jpg"/>
</content>
The database and the downstream process are 2 different software packages with an out-of-the-box integration. I could write my own pipeline to do the transforms, but it may be cost prohibitive. The current integration only allows for 1 XSLT to be run a the pre-process step.
Is it possible to transform referenced XML files in a single XSLT transform step?
The following setup uses a main input XML input.xml which only contains a root node and two other XML files part1.xml and part2.xml which are read using the document() function. The XSLT uses <xsl:apply-templates> to trigger the conversion of the union of the two XML files. This way it should be possible to combine any number of XML files beforehand into one tree and execute the XSLT process on the resulting tree.
By the way: It turns out that this is already supported in XSLT 1.0 (at least by the xsltproc processor).
File input.xml:
<?xml version="1.0" encoding="ISO-8859-1"?>
<root/>
File part1.xml:
<?xml version="1.0" encoding="ISO-8859-1"?>
<root1>
<element>
<subelement1 message="Hello from part1.xml"/>
</element>
</root1>
File 'part2.xml':
<?xml version="1.0" encoding="ISO-8859-1"?>
<root2>
<element>
<subelement2 message="Hello from part2.xml"/>
</element>
</root2>
With the input above the following XSLT transformation
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet
version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes" />
<xsl:variable name="part1" select="document('part1.xml')/root1"/>
<xsl:variable name="part2" select="document('part2.xml')/root2"/>
<xsl:template match="/">
<messages>
<xsl:apply-templates select="$part1|$part2"/>
</messages>
</xsl:template>
<xsl:template match="*">
<xsl:apply-templates/>
</xsl:template>
<xsl:template match="text()"/>
<xsl:template match="subelement1">
<message>
<xsl:value-of select="#message"/>
</message>
</xsl:template>
<xsl:template match="subelement2">
<message>
<xsl:value-of select="#message"/>
</message>
</xsl:template>
</xsl:stylesheet>
yields this output:
<?xml version="1.0" encoding="UTF-8"?>
<messages>
<message>Hello from part1.xml</message>
<message>Hello from part2.xml</message>
</messages>