TSQL Parse XML with namespace - tsql

I'm trying to parse some xml that's stored in the database inside of stored procedure. The procedure should return 2 columns, id and value. I'm only part of the way through this, I'm stuck on the fact that I can't list even list the "Setting" nodes.
declare #PolicySettingsXml xml
set #PolicySettingsXml = '<?xml version="1.0" encoding="utf-8"?>
<Policy xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://www.myurl.com/app/Policies">
<Setting id="VendorInfo0">
<string>fsdfdsfds</string>
</Setting>
<Setting id="VendorInfo1">
<string />
</Setting>
<Setting id="VendorInfo2">
<string />
</Setting>
<Setting id="SendSurchargeAsSeparateTransaction">
<boolean>false</boolean>
</Setting>
<Setting id="VendorSurchargeInfo0">
<string />
</Setting>
<Setting id="VendorSurchargeInfo1">
<string />
</Setting>
<Setting id="VendorSurchargeInfo2">
<string />
</Setting>
</Policy>'
select T.c.query('string') as value, T.c.query('#id') as id from #PolicySettingsXml.nodes('/Policy/Setting') T(c)
I'm getting an error, "XQuery [query()]: Attribute may not appear outside of an element," I'd expect to see:
id | value
VendorInfo0 | fsdfdsfds
VendorInfo1 | <null>
VendorInfo2 | <null>
SendSurchargeAsSeparateTransaction | <null>
VendorSurchargeInfo1 | <null>
VendorSurchargeInfo2 | <null>

The first issue is related to syntax.
This:
from PSXML.x.nodes('/*:Policy/*:Setting') T(c);
should be:
from (VALUES(#PolicySettingsXml)) AS PSXML(x)
CROSS APPLY PSXML.x.nodes('/*:Policy/*:Setting') T(c);
For what you are doing you can use the value method for the id column.
select
id = T.c.value('(#id)[1]', 'varchar(100)'),
[value] = T.c.query('(*:string/text())[1]')
from (VALUES(#PolicySettingsXml)) AS PSXML(x)
CROSS APPLY PSXML.x.nodes('/*:Policy/*:Setting') T(c);
Note that I am cheating on the namespaces using the "all namespaces" syntax: * :Object. A google search for "sql server t-sql xml namespaces" will show you the way you're supposed to do it. I cheated due to time constraints.

Related

Postgres XML parsing - Calculate the sum of multiple nodes of the same name using XPath

I have an xml snippet as below:
<OpCodeLaborInfo JobStatus="F" UpSellFlag="N" JobNo="1" OpCode="02TTZ10K" OpCodeDesc="10K SERVICE">
<TechInfo ActualHrsWorked="2.50" CustTechRate="27.00" TechHrs="0.00" TechName="JEFF SELLERS" TechNo="4816" />
<TechInfo ActualHrsWorked="0.00" CustTechRate="27.00" TechHrs="0.70" TechName="JEFF SELLERS" TechNo="4816" />
<BillTimeRateHrs BillRate="129.97" />
<CCCStmts Correction="PERFORMED 10K SERVICE" Complaint="LUBE OIL FILTER CHANGE, TIRE ROTATION, PERFORM MULTI POINT" />
<CCCStmts Correction="X" Complaint="INSPECTION, INSPECT FILTERS AND RECOMMEND, INSPECT BRAKES," />
<CCCStmts Complaint="BELTS AND HOSES" />
<RoAmts DlrCost="18.90" PayType="Cust" AmtType="Job" TotalAmt="59.12" />
</OpCodeLaborInfo>
<OpCodeLaborInfo JobStatus="F" UpSellFlag="N" JobNo="2" OpCode="02TTZ10K" OpCodeDesc="10K SERVICE">
<TechInfo ActualHrsWorked="2.50" CustTechRate="27.00" TechHrs="1.00" TechName="JEFF SELLERS" TechNo="4816" />
<TechInfo ActualHrsWorked="0.00" CustTechRate="27.00" TechHrs="0.00" TechName="JEFF SELLERS" TechNo="4816" />
<BillTimeRateHrs BillRate="129.97" />
<CCCStmts Correction="PERFORMED 10K SERVICE" Complaint="LUBE OIL FILTER CHANGE, TIRE ROTATION, PERFORM MULTI POINT" />
<CCCStmts Correction="X" Complaint="INSPECTION, INSPECT FILTERS AND RECOMMEND, INSPECT BRAKES," />
<CCCStmts Complaint="BELTS AND HOSES" />
<RoAmts DlrCost="18.90" PayType="Cust" AmtType="Job" TotalAmt="59.12" />
</OpCodeLaborInfo>
I need to calculate the sum of the TechInfo/#TechHrs for each OpCodeLaborInfo. I tried the following:
unnest(xpath('sum(//dns:RepairOrder/dns:RoRecord/dns:Rolabor/dns:OpCodeLaborInfo/dns:TechInfo/#TechHrs[1])'::text,
data_detail.ro_data_xml,
ARRAY[ARRAY['dns'::text, 'http://www.starstandards.org/STAR'::text]]))::text::numeric AS lbrsoldhours
but this seems to return the sum of the Tech Hours inside both the OpCodeLaborInfo nodes. Could someone be able to tell me how I can tweak the xpath so as to get the desired result.
So basically I need :
Job
Tech Hrs
1
sum(0.00+0.70)
2
sum(1.00+0.00)
I would solve this using xmltable()
select d.job, sum(t.hours)
from data_detail d
cross join xmltable (
'/OpCodeLaborInfo/TechInfo'
passing d.ro_data_xml
columns hours numeric path '#TechHrs') as t
group by d.job;
Online example
The XPath is probably not correct as the XPath you have shown doesn't match your sample XML data. Your sample XML also doesn't contain a namespace, so I am not sure why you are passing one to xpath()
But if you need one, you can use something like this:
cross join xmltable (
xmlnamespaces ('http://www.starstandards.org/STAR' as dns),
'/dns:OpCodeLaborInfo/dns:TechInfo'
passing d.ro_data_xml
columns hours numeric path '#TechHrs') as t
xpath can also work fine:
SELECT job
, SUM((xpath('//#TechHrs', element))[1]::text::decimal)
FROM data_detail
, LATERAL (SELECT unnest(xpath('/OpCodeLaborInfo/TechInfo', ro_data_xml))) u(element)
GROUP BY job;
fiddle
(based on the fiddle of #a_horse_with_no_name)

Liquibase: how to load data with CURRENT_TIMESTAMP?

I am trying to update my project, going from
<liquibase.version>3.5.5</liquibase.version>
<liquibase-hibernate5.version>3.6</liquibase-hibernate5.version>
to
<liquibase.version>3.9.0</liquibase.version>
<liquibase-hibernate5.version>3.8</liquibase-hibernate5.version>
and I'm having some troubles while loading data from CSV files, which contain some columns having the current timestamp as value.
E.g my CSV file:
id;name;created;modified
1;Book A;now();now()
2;Book B;now();now()
The Book table is created with:
<createTable tableName="book">
<column name="id" type="bigint" autoIncrement="${autoIncrement}">
<constraints primaryKey="true" nullable="false"/>
</column>
<column name="name" type="varchar(255)">
<constraints nullable="false"/>
</column>
<column name="created" type="timestamp">
<constraints nullable="false"/>
</column>
<column name="modified" type="timestamp">
<constraints nullable="false"/>
</column>
</createTable>
And the data is loaded with:
<property name="now" value="current_timestamp" dbms="postgresql"/>
<changeSet id="20180508144233-1" author="developer">
<loadData catalogName="public"
encoding="UTF-8"
file="config/liquibase/books.csv"
schemaName="public"
separator=";"
quotchar="'"
tableName="book">
</loadData>
</changeSet>
With the previous version of Liquibase it was working fine, however after the update I am getting the following error:
2020-06-26 16:49:57 [project-Executor-1] [ERROR] liquibase.changelog.ChangeSet - Change Set config/liquibase/changelog/20180508144233_added_books_data.xml::20180508144233-1::developer failed. Error: liquibase.exception.DateParseException: Improper value in 'NOW' value: now(). 'NOW' must be followed by + or -, then numeric offset, then units (h{our{s}}, m{inute{s}}, d{ay{s}}, or y{ears}
Hibernate: select answerweig0_.id as id1_1_, answerweig0_.likelihood as likeliho2_1_, answerweig0_.question_type as question3_1_, answerweig0_.weight as weight4_1_ from answer_weight answerweig0_
2020-06-26 16:49:57 [project-Executor-1] [ERROR] i.g.j.c.l.AsyncSpringLiquibase - Liquibase could not start correctly, your database is NOT ready: Migration failed for change set config/liquibase/changelog/20180508144233_added_books_data.xml::20180508144233-1::developer:
Reason: liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.DateParseException: Improper value in 'NOW' value: now(). 'NOW' must be followed by + or -, then numeric offset, then units (h{our{s}}, m{inute{s}}, d{ay{s}}, or y{ears}
liquibase.exception.MigrationFailedException: Migration failed for change set config/liquibase/changelog/20180508144233_added_books_data.xml::20180508144233-1::developer:
Reason: liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.DateParseException: Improper value in 'NOW' value: now(). 'NOW' must be followed by + or -, then numeric offset, then units (h{our{s}}, m{inute{s}}, d{ay{s}}, or y{ears}
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:646)
at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:53)
at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:83)
at liquibase.Liquibase.update(Liquibase.java:202)
at liquibase.Liquibase.update(Liquibase.java:179)
at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:366)
at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:314)
at org.springframework.boot.autoconfigure.liquibase.DataSourceClosingSpringLiquibase.afterPropertiesSet(DataSourceClosingSpringLiquibase.java:46)
at io.github.jhipster.config.liquibase.AsyncSpringLiquibase.initDb(AsyncSpringLiquibase.java:118)
at io.github.jhipster.config.liquibase.AsyncSpringLiquibase.lambda$afterPropertiesSet$0(AsyncSpringLiquibase.java:93)
at io.github.jhipster.async.ExceptionHandlingAsyncTaskExecutor.lambda$createWrappedRunnable$1(ExceptionHandlingAsyncTaskExecutor.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
As a workaround, I was able to set the values for such columns by using a default value in the table creation changelog:
<createTable tableName="book">
<column name="id" type="bigint" autoIncrement="${autoIncrement}">
<constraints primaryKey="true" nullable="false"/>
</column>
<column name="name" type="varchar(255)">
<constraints nullable="false"/>
</column>
<column name="created" type="timestamp" defaultValueComputed="CURRENT_TIMESTAMP">
<constraints nullable="false"/>
</column>
<column name="modified" type="timestamp" defaultValueComputed="CURRENT_TIMESTAMP">
<constraints nullable="false"/>
</column>
</createTable>
and removing the corresponding columns from the CSV file:
id;name
1;Book A
2;Book B
However, I'm still looking for a way to keep the CURRENT_TIMESTAMP value in the CSV file.
Your workaround is pretty much correct. But depending on database you will have to add a way that your update_timestamp or modified timestamp will change based on every addition, subtraction or edit you have in this database row. Here is how it can be setup in PostgresSQL for example.
<changeSet id="{UNIQUE_ID}" author="{YOUR_NAME}">
<preConditions onFail="MARK_RAN">
<dbms type="postgresql"/>
</preConditions>
<createProcedure>
CREATE OR REPLACE FUNCTION refresh_updated_date_column()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated_date = now();
RETURN NEW;
END;
$$ language 'plpgsql';
</createProcedure>
<rollback>
DROP FUNCTION IF EXISTS refresh_updated_date_column;
</rollback>
</changeSet>`
This will be an additional changeset that you will add after your original changeset of creating the table and adding the required columns.
You need to add CURRENT_TIMESTAMP in your csv file like:
"id","date_created"
"11236,CURRENT_TIMESTAMP
And your changeset like:
<column header="date_modified"
name="date_modified"
type="DATE"/>
Note: Im using liquibase version 4.2.0+

SQL Server 2016 XML Shredding

Have been trying to figure this out for a while without success, read like 10 posts and some other examples and the MS help, not resonating, need to shred some xml data with the following format:
<ncf_report xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://cp.com/rules/client">
<admin>
<quoteback name="abcd">ABCD A</quoteback>
<product_group>Abcd ABcd Abcd</product_group>
<pnc_account>123456</pnc_account>
<pnc_account_name>ABC</pnc_account_name>
<product_reference>123456789</product_reference>
<report_type>ABCDE</report_type>
<status>ABCDE</status>
<ownership>ABCD</ownership>
<report_code>1234</report_code>
<report_description>Abcde/report_description>
<purpose>ABCDEFGH</purpose>
<date_request_ordered>05/05/2020</date_request_ordered>
<date_request_received>05/05/2020</date_request_received>
<date_request_completed>05/05/2020</date_request_completed>
<time_report_processed>1028</time_report_processed>
<multiple_scores_ordered>false</multiple_scores_ordered>
<vendor name="Abcd" address="Abcd" />
<report>
<sequence>0000000001</sequence>
<count>0000000001</count>
</report>
</admin>
<report>
<alerts_scoring>
<scoring>
<score status="Abcd">
<model_label>ABCD</model_label>
<score>123</score>
<rating_state>AB</rating_state>
<classification> ABCD </classification>
<reason_codes>
<code>12</code>
<description>ABCD</description>
</reason_codes>
<reason_codes>
<code>12</code>
<description>ABCD</description>
</reason_codes>
<reason_codes>
<code>12</code>
<description>ABCD ABCD ABCD</description>
</reason_codes>
<reason_codes>
<code>12</code>
<description>ABCD ABCD ABCD</description>
</reason_codes>
</score>
</scoring>
<general>ABCD ABCD ABCD ORIGINAL REPORT DATE: 12/12/2000</general>
<general>ABCD ABCD ABCD</general>
<general> ABCD ABCD ABCD</general>
<general narrativeCode="Abcd Abcd">ABCD ABCD ABCD</general>
<general narrativeCode=" Abcd Abcd">ABCD ABCD ABCD</general>
<general narrativeCode=" Abcd Abcd">ABCD ABCD ABCD</general>
</alerts_scoring>
<vendor_dataset>
<subjects>
<subject type="Abcd" relationship_to_data="Abcd">
<name type="Abcd">
<first>XXXX</first>
<middle>X</middle>
<last>XXXX</last>
</name>
<birth_date>01/01/1900</birth_date>
<ssn>999999999</ssn>
<address type="Abcd" ref="1" />
<address type="Abcd" ref="2" />
<address type="Abcd" ref="3" />
</subject>
</subjects>
<addresses>
<address id="1">
<street1>ABCD</street1>
<city>ABCD</city>
<state>AB</state>
<postalcode>12345</postalcode>
<zip4>1234</zip4>
<date_first_at_address>01/02/1900</date_first_at_address>
<date_last_at_address>01/02/1900</date_last_at_address>
</address>
<address id="2">
<house>123</house>
<street1>ABCDE</street1>
<city>ABCDE</city>
<state>AB</state>
<postalcode>12345</postalcode>
<zip4>1234</zip4>
<date_first_at_address>00/00/1900</date_first_at_address>
<date_last_at_address>00/00/1900</date_last_at_address>
</address>
<address id="3">
<street1>ABCDE</street1>
<city>ABCDE</city>
<state>AB</state>
<postalcode>12345</postalcode>
<zip4>1234</zip4>
<date_first_at_address>00/00/1900</date_first_at_address>
<date_last_at_address>00/00/1900</date_last_at_address>
</address>
</addresses>
</vendor_dataset>
<summary>
<date_oldest_trade>00/00/1900</date_oldest_trade>
<date_latest_trade>00/00/1900</date_latest_trade>
<date_latest_activity>00/00/1900</date_latest_activity>
<includes_bankruptcies flag="true" date="02/02/2009" />
<includes_other_records public_records="false" collection="true" consumer_statement="false" />
<credit_range high="123456" low="1234" number_trade_lines="12" />
**<account_status_counters>
<account type="current" description="Pays Account as Agreed" status="1">12</account>
<account type="current" description="Status Not Known" status=" ">7</account>
<account type="former" description="Pays/Paid 30-60 Days or Max 2 Payments Past Due" status="2">5</account>
<account type="former" description="Pays/Paid 60-90 Days or Max 3 Payments Past Due" status="3">4</account>
<account type="former" description="Bad Debt" status="9">6</account>
</account_status_counters>**
I currently going down the path of trying to use the xml procedure but I could not get to the finish line with openxml as well. Trying to extract data in highlighted at bottom of xml
EXEC sp_xml_preparedocument #hdoc OUTPUT, #CreditScoreXML
SELECT * FROM OPENXML(#hdoc, '/<ncf_report xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://cp.com/rules/client">/admin/summary/account_status_counters')
WITH
(
[Ref_Number] VARCHAR(10) 'product_reference',
[current_account_type] VARCHAR(10) './account/#type',
[current_account_type_description] VARCHAR(50) './account/#description',
[current_account_type_description] VARCHAR(1) './account/#status'
You can define the namespace for your XML using WITH XMLNAMESPACES statement, then you can extract the values you need with .value().
I don't understand exactly the information you are trying to extract, but this should put you on the right track (I only put the first row of your xml to save space, you should put the entire XML fragment in the #xml variable):
declare #xml xml set #xml='
<ncf_report xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://cp.com/rules/client">
...
'
;WITH XMLNAMESPACES ('http://cp.com/rules/client' as ns1)
select
#xml.value('(ns1:ncf_report/ns1:admin/ns1:product_reference)[1]', 'varchar(10)') as Ref_Number
,#xml.value('(ns1:ncf_report/ns1:report/ns1:summary/ns1:account_status_counters/ns1:account[#type="current" and #status ="1"]/#description)[1]', 'varchar(50)') as CurrentDescription
,#xml.value('(ns1:ncf_report/ns1:report/ns1:summary/ns1:account_status_counters/ns1:account[#type="current" and #status ="1"])[1]', 'int') as CurrentStatus
,#xml.value('(ns1:ncf_report/ns1:report/ns1:summary/ns1:account_status_counters/ns1:account[#type="current" and #status =" "]/#description)[1]', 'varchar(50)') as CurrentDescription_2
,#xml.value('(ns1:ncf_report/ns1:report/ns1:summary/ns1:account_status_counters/ns1:account[#type="current" and #status =" "])[1]', 'int') as CurrentStatus_2
This sample query would extract:

NHibernate inserting wrong Id

I am using MVC razor 5
I'm making managment page to add SelectList values (like Car models)
I have values in my car models table:
--------------------------------------
Id | Name | Deleted
1 | BMW | False
2 | Audi | False
3 | Ford | False
4 | BMW5 | False
5 | SEAT | False
--------------------------------------
When i add new value id becomes 130++
I have noticed that my main class last records Id is 130.
And when I insert a record in car model class Id is the nextval(Id) of main class
But these two tables does not share same Controller or so...
All sessions are closed by transaction.Commit() so session closes.
What may be wrong?
Nhibernate class mapping:
<class name="web_nt.Models.Class.CarModel" table="car_model" schema="public">
<id name="Id" column="id" type="int">
<generator class="native" />
</id>
<property name="model_name" />
<property name="deleted" />
</class>
It was generator class fault.
To use postgres sequence I needed to specify it (NHibernate does not always catch it itself)
<generator class="sequence">
<param name="sequence">car_model_id_sequence</param>
</generator>

Adding an entity key when no key is inferred from a view

I have a database view which joins across a number of tables in SQL Server 2005. It looks something like this:
SELECT
m1.MenuName AS menu_name, m2.MenuName AS sub_menu_name, p.ProductName, od.amount
FROM
dbo.tblMenus AS m1
FULL OUTER JOIN
dbo.tblMenus AS m2
FULL OUTER JOIN
dbo.tblProductsRelMenus AS pm ON m2.Id = pm.SubMenuId ON m1.Id = pm.MenuId
FULL OUTER JOIN
(SELECT
dbo.tblOrderDetails.ProductId, SUM(dbo.tblOrderDetails.Ammount) AS amount
FROM
dbo.tblOrderDetails
FULL OUTER JOIN
dbo.tblOrders AS o ON dbo.tblOrderDetails.OrderId = o.OrderId
WHERE (o.OrderDestroyed = 0)
GROUP BY dbo.tblOrderDetails.ProductId) AS od
RIGHT OUTER JOIN
dbo.tblProducts AS p ON od.ProductId = p.ProductId ON pm.ProductId = p.ProductId
When I try to create an ADO .Net entity data model it complains about not having a primary key in the SSDL secion. I then found this:
http://msdn.microsoft.com/en-us/library/dd163156.aspx
but I don't understand the part about a defining query. Surely I just want a column with unique numbers to define the key, or?
<EntityType Name="SoldItemsView">
<Key>
<PropertyRef Name="SoldItemsViewId" />
</Key>
<Property Name="SoldItemsView" Type="int" Nullable="false" />
<Property Name="menu_name" Type="nvarchar" MaxLength="100" />
<Property Name="sub_menu_name" Type="nvarchar" MaxLength="100" />
<Property Name="ProductName" Type="nvarchar" MaxLength="50" />
<Property Name="amount" Type="int" />
</EntityType>
But how do I populate this column with unique numbers?
Thanks,
Barry
You can use only columns from the view. To define an entity key you must select column (or set of columns) from your view which uniquely identifies record from the view.