Adding an XML attribute conditionally - scala

I need a way to add an XML attribute 'POSITON' to an XML element 'node' conditionally. Currently I'm doing the condition check first and then creating the node.
if (lvl == 2)
node = <node COLOR={ color } CREATED={ epochTimeMillis } ID={ idGen } POSITION={ position } LINK={ link } MODIFIED={ epochTimeMillis } STYLE="bubble" TEXT={ f.getName() }>
<edge COLOR={ color } STYLE={ style } WIDTH={ width }/>
</node>
else
node = <node COLOR={ color } CREATED={ epochTimeMillis } ID={ idGen } LINK={ link } MODIFIED={ epochTimeMillis } STYLE="bubble" TEXT={ f.getName() }>
<edge COLOR={ color } STYLE={ style } WIDTH={ width }/>
</node>
}

Using "null" is not a good practice, but in this case it would help you:
scala> <root ta={ if (true) "true" else null } fa={ if (false) "false" else null } />
res0: scala.xml.Elem = <root ta="true" ></root>

A slightly cleaner way to do the same thing #senia suggests is:
val posOpt = if (lvl2) Some(myPosition) else None
val xml = <mydata position={posOpt orNull}/>

One way is to create the snippet before:
val pos =
if (lvl == 2) {
"position = ..."
} else {
""
}
and to always insert it in the result.
This could by extended by using an Option with embedded map in combination with string interpolation.
val pos =
if (lvl == 2) {
Some(position)
} else {
None
}
with
pos.map(v => s"position = $v")

Related

convert html to json using rdd.map

I have html file which I want to parse in pySpark.
Example:
<MainStruct Rank="1">
<Struct Name="A">
<Struct Name="AA">
<Struct Name="AAA">
<Field Name="F1">Data</Field>
</Struct>
<Struct Name="ListPart">
<List Name="ListName">
<Struct Name="S1">
<Field Name="F1">AAA</Field>
<Field Name="F2">BBB</Field>
<Field Name="F3">CCC</Field>
</Struct>
<Struct Name="S1">
<Field Name="F1">XXX</Field>
<Field Name="F2">GGG</Field>
<Field Name="F3">BBB</Field>
</Struct>
</List>
</Struct>
</Struct>
</Struct>
</FullStudy>
rdd_html = spark.sparkContext.wholeTextFiles(path_to_XML, minPartitions=1000, use_unicode=True)
df_html = spark.createDataFrame(rdd_html,['filename', 'content'])
rdd_map = df_html.rdd.map(lambda x: xmltodict(x['content'],'mainstruct'))
df_map = spark.createDataFrame(rdd_map)
df_map.display()
but in my Notebook output I have problem with list elements. They are parsed inсorrectly.
>object
>AA:
>ListPart:
ListName: "[{S1={F1=AAA, F2=BBB, F3=CCC}}, {S1={F1=XXX, F2=GGG, F3=BBB}}]"
>AAA:
F1: "Data"
List element represents as one string line.
My function to parse it:
def xmltodict(content,first_tag=''):
#Content from xml File
content = re.sub('\n', '', content)
content = re.sub('\r', '', content)
content = re.sub('>\s+<', '><', content)
data = unicodedata.normalize('NFKD', content)
soup = BeautifulSoup(data, 'lxml')
body = soup.find('body')
if(first_tag.strip()!=''):
struct = body.find(first_tag)
else:
struct=body
return parser(struct)
def parser(struct):
struct_all = struct.findAll(True, recursive=False)
struct_dict = {}
for strc in struct_all:
tag = strc.name
tag_name_prop = strc.attrs['name']
if tag == 'struct':
d = parser(strc)
el = {tag_name_prop: d}
struct_dict.update(el)
elif tag == 'field':
v = strc.text
struct_dict[tag_name_prop] = v
elif tag == 'list':
l_elem = []
for child in strc.contents:
soap_child = BeautifulSoup(str(child), 'lxml').find('body')
l_elem.append(parser(soap_child))
el = {tag_name_prop: l_elem}
struct_dict.update(el)
with open (result.txt,'w') as file:
file.write(json.dumps(struct_dict))
return struct_dict
the result in txt file is that I want to receive:
"A": { "AA": {
"AAA": {"F1": "Data"},
"ListPart": {
"ListName": [
{
"S1": {"F1": "AAA",
"F2": "BBB",
"F3": "CCC"
}
},
{
"S1": { "F1": "XXX",
"F2": "GGG",
"F3": "BBB"
}}]
}}}
but in my notebook output I have problem with list elements. They are parsed inсorrectly.
>object
>AA:
>ListPart:
ListName: "[{S1={F1=AAA, F2=BBB, F3=CCC}}, {S1={F1=XXX, F2=GGG, F3=BBB}}]"
>AAA:
F1: "Data"
Why list represents as one string line? Why are there "=" symbols instead of ":"?
i simplified this issue to that:
def parseList(row):
d = {}
d['el1']='AAA'
l = [{'x1':'XA'},{'x1':'XB'}]
d['el2']=l
return Row(res=d)
rdd_html = spark.sparkContext.wholeTextFiles(path_to_file_test, minPartitions=1000, use_unicode=True)
df_html = spark.createDataFrame(rdd_html,['filename', 'content'])
rdd_map = df_html.rdd.map(parseList2)
df_map = spark.createDataFrame(rdd_map)
df_map.display()
in result i also have
>object
el2: "[{x1=XA}, {x1=XB}]"
el1: "AAA"
not that
>object
>el2
x1:"XA"
x1:"XB"
el1: "AAA"
I finally resolved my problem.
The reason was that i should define schema and use it.
df_map = spark.createDataFrame(rdd_map,schema)

How to return connection from a JDBC connection class using Scala's exception handling?

I am trying to create a Scala JDBC program where a connection to Hive is being made. To do this, I wrote the below code.
var HIVECON: Connection = null
def hiveConnection(): Connection = {
val conf = new Configuration()
conf.set("hadoop.security.authentication", "Kerberos")
// DEV System Properties
System.setProperty("java.security.krb5.kdc", "ip-address.ec2.internal");
System.setProperty("java.security.krb5.realm", "DEV.COM");
// DEV System Properties
// DEV loginUserFromKeytab
UserGroupInformation.loginUserFromKeytab("username#DEV.COM", "/home/username/username.keytab");
// DEV loginUserFromKeytab
try {
Class.forName("org.apache.hive.jdbc.HiveDriver")
if(HIVECON == null || HIVECON.isClosed)
HIVECON = DriverManager.getConnection("jdbc:hive2://ip-address.ec2.internal:10500/dbname;principal=hive/ip-address.ec2.internal#DEV.COM", "username","password")
else HIVECON
} catch {
case s:SQLException => s.printStackTrace()
case e:Exception => e.printStackTrace()
}
}
But the code gives a compilation error at these lines:
With the way I wrote, the catch statements are returning UNIT where my method is trying to return CONNECTION. Is there any way to handle the exception better ?
I would handle the Exception in a functional way.
If you do not care about specific Exception use Option:
var HIVECON: Option[Connection] = None
def hiveConnection(): Option[Connection] = {
...
try {
Class.forName("org.apache.hive.jdbc.HiveDriver")
if(HIVECON == None || HIVECON.get.isClosed)
HIVECON = Some(DriverManager.getConnection("jdbc:hive2://ip-address.ec2.internal:10500/dbname;principal=hive/ip-address.ec2.internal#DEV.COM", "username","password"))
HIVECON // return Some(Connection)
} catch {
case s:Exception =>
s.printStackTrace()
None
}
If you care for the Exception use Try:
var HIVECON: Connection = null
def hiveConnection(): Try[Connection] = {
...
Try {
Class.forName("org.apache.hive.jdbc.HiveDriver")
if(HIVECON == null || HIVECON.isClosed)
HIVECON = DriverManager.getConnection("jdbc:hive2://ip-address.ec2.internal:10500/dbname;principal=hive/ip-address.ec2.internal#DEV.COM", "username","password")
HIVECON // return Success(Connection)
}
In case of a Failure it returns Failure(Exception).
See here the Docs: https://docs.scala-lang.org/overviews/scala-book/functional-error-handling.html

copy rows with myBatis

I'm trying to copy rows with Oracle and myBatis with this statement:
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd" >
<mapper
...
<insert id="copy" >
INSERT INTO ${table}
<foreach item="key" collection="keys" index="index" open="(" separator="," close=")">
${key}
</foreach>
VALUES
<foreach item="key" collection="map" index="index" open="(" separator="," close=")">
${key}
</foreach>
</insert>
</mapper>
--
for ( Map.Entry<Object, Object> entry2 : map.entrySet()) {
String key2 = (String) entry2.getKey();
Object value = null;
if (entry2.getValue() == null)
value = "NULL";
else if (entry2.getValue() instanceof java.sql.Timestamp)
{
//...
}
else if (entry2.getValue() instanceof Integer || entry2.getValue() instanceof Long ||
entry2.getValue() instanceof Short || entry2.getValue() instanceof java.math.BigDecimal
){
value = entry2.getValue();
}
else if (entry2.getValue().getClass().getName().equals("oracle.sql.CLOB"))
{
Clob clob=(Clob)entry2.getValue();
value= (String)clob.getSubString((long)1, (int)clob.length());
}
else{
value = "'" + entry2.getValue() + "'";
}
map.put(key2, value);
columans_valores.add("'" + key2 + "'");
}
copyService(map, map.keySet(), "table");
--
It works with most types, but CLOB and BLOB get their address copied instead or gets shrunk to 4000 bytes depending on the code i use to copy them, how do i deal with them?
Finally got it working with some tweaks:
To inform myBatis about the type used, switch $ for #
Avoid quotes
To prevent Oracle errors when inserting nulls, remove them from the map instead of
trying to insert null values

Cannot resolve element with ID while signing a part of SOAP REQUEST X509

I had the following error while trying to sign a part of SOAP Request :
org.apache.xml.security.utils.resolver.ResourceResolverException: Cannot resolve element with ID _53ea293168db637b15e2d4d7894
at org.apache.xml.security.utils.resolver.implementations.ResolverFragment.engineResolve(ResolverFragment.java:86)
at org.apache.xml.security.utils.resolver.ResourceResolver.resolve(ResourceResolver.java:279)
at org.apache.xml.security.signature.Reference.getContentsBeforeTransformation(Reference.java:417)
at org.apache.xml.security.signature.Reference.dereferenceURIandPerformTransforms(Reference.java:597)
at org.apache.xml.security.signature.Reference.calculateDigest(Reference.java:689)
at org.apache.xml.security.signature.Reference.generateDigestValue(Reference.java:396)
at org.apache.xml.security.signature.Manifest.generateDigestValues(Manifest.java:206)
at org.apache.xml.security.signature.XMLSignature.sign(XMLSignature.java:595)
It comes from the resolution of the URI declared on the reference tag.
Here is the java code i'm using for signing via X509 :
KeyStore.PrivateKeyEntry pke = ISKeyStoreManager.getInstance().getPrivateKeyEntry(keyStoreAlias, keyAlias);
AlgorithmStrings algStrings = AlgorithmStrings.getAlgDSStrings( pke.getPrivateKey(), signatureAlgorithmString, digestAlgorithmString);
String resultantXPath = StringUtils.join(xpaths, '|');
Transforms transforms = new Transforms(originalDocument);
NodeList targetDocumentList = obtainNodesForXPath(originalDocument, resultantXPath, nc);
if(targetDocumentList != null && targetDocumentList.getLength() > 0)
{
if(targetDocumentList.item(0).hasAttributes()){
Node attrId = targetDocumentList.item(0).getAttributes().getNamedItem("Id");
if(attrId != null && !attrId.getNodeValue().equals("")){
uri = new StringBuilder().append('#').append(attrId.getNodeValue()).toString();
}
else{
((Element) targetDocumentList.item(0)).setAttribute("xmlns:wsu", "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd");
((Element) targetDocumentList.item(0)).setAttribute("wsu:Id", idForXmlObject);
}
}
else{
((Element) targetDocumentList.item(0)).setAttribute("xmlns:wsu", "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd");
((Element) targetDocumentList.item(0)).setAttribute("wsu:Id", idForXmlObject);
}
}else{
log.debug("Target not found in the original document with xpath: " + resultantXPath);
}
transforms.addTransform("http://www.w3.org/2000/09/xmldsig#enveloped-signature");
if (resultantXPath != null) {
log.debug("Instantiation XPATHContainer");
XPathContainer xpathC = new XPathContainer(originalDocument);
xpathC.setXPath(resultantXPath);
if ((ncMap != null) && (!ncMap.isEmpty())) {
for (Map.Entry<String,String> e : ncMap.entrySet()) {
log.debug("Adding namespace to XPATH Container: " + e.getKey() + " -> " + e.getValue());
xpathC.setXPathNamespaceContext(e.getKey(), e.getValue());
}
}
transforms.addTransform("http://www.w3.org/TR/1999/REC-xpath-19991116", xpathC.getElement());
}
log.debug("Instantiation Signature");
XMLSignature sig = new XMLSignature(originalDocument, null, algStrings.signatureAlgorithm, canonicalizationAlg);
sig.setFollowNestedManifests(true);
log.debug("Ajout des assertions de transformation");
sig.addDocument("", transforms, algStrings.digestMethod);
if (idAttrForSignature != null) {
sig.setId(idAttrForSignature);
}
log.debug("DOMToString: " + serializeDOMToString(originalDocument));
// signature node insertion
NodeList nodeList = obtainNodesForXPath(originalDocument, insertSignatureAtXPath, nc);
if(nodeList != null && nodeList.getLength() > 0){
Node nodeSignature = nodeList.item(0);
Node childNode = nodeSignature.getFirstChild();
if (childNode != null) {
if (addSignatureAsLastElement)
nodeSignature.appendChild(sig.getElement());
else
nodeSignature.insertBefore(sig.getElement(), childNode);
}
else nodeSignature.appendChild(sig.getElement());
}
else{
throw new ServiceException("INVALID_SIGNATURE_NODE_SELECTOR_XPATH");
}
// Public key insertion
//X509Data x509Data = getX509Data(includeCertChain, certificateData, originalDocument, pke);
//KeyInfoReference kir = new KeyInfoReference(x509Data.getDocument());
SecurityTokenReference str = new SecurityTokenReference(sig.getKeyInfo().getDocument());
str.setKeyIdentifier(ISKeyStoreAccessorUtil.getIaikCertificate(pke.getCertificate()));
sig.getKeyInfo().getElement().appendChild(str.getElement());
log.debug("DOMToString: " + serializeDOMToString(originalDocument));
//sig.getSignedInfo().addResourceResolver(new ResolverXPointer());
((Element)(sig.getSignedInfo().getElement().getElementsByTagNameNS("http://www.w3.org/2000/09/xmldsig#", "Reference").item(0))).setAttribute("URI", uri);
log.debug("DOMToString: " + serializeDOMToString(originalDocument));
//sig.addDocument(uri, trans);
// Signature generation
sig.sign(pke.getPrivateKey());
Do you have any proposition of workaround ? or another way to set URI attribute ?
Thank you for helping !
I found it.
I added InclusiveNamespaces so that the sign method can figure out that ID is on a specific namespace defined attribute.

Magento rest api getting error Request does not match any route

I have created a rest api web service but I am getting error on hitting the url
http://localhost:81/magento/api/rest/category/2
<magento_api>
<messages>
<error>
<data_item>
<code>404</code>
<message>Request does not match any route.</message>
</data_item>
</error>
</messages>
</magento_api>
My api2.xml is:-
<?xml version="1.0"?>
<config>
<api2>
<resource_groups>
<esoft_rescategories translate="title" module="Esoft_Restcategories">
<title>Esoft Restcategories API</title>
<sort_order>11</sort_order>
</esoft_rescategories>
</resource_groups>
<resources>
<esoft_restcategories translate="title" module="Esoft_Restcategories">
<group>esoft_rescategories</group>
<model>esoft_restcategories/api2_restapi</model>
<title>Categories</title>
<sort_order>11</sort_order>
<privileges>
<admin>
<create>1</create>
<!--<retrieve>1</retrieve>
<update>1</update>
<delete>1</delete>-->
</admin>
<guest>
<retrieve>1</retrieve>
<!--<create>1</create>
<update>1</update>
<delete>1</delete>-->
</guest>
</privileges>
<attributes>
<category_id>Category ID</category_id>
<name>Name</name>
<parent_id>Category Parent ID</parent_id>
<child_id>Category Child List</child_id>
<active>Active</active>
<level>Level</level>
<position>Position</position>
</attributes>
<routes>
<route_entity>
<route>/categories/:cat_id</route>
<action_type>entity</action_type>
</route_entity>
<route_collection>
<route>/categories</route>
<action_type>collection</action_type>
</route_collection>
</routes>
<versions>1</versions>
</esoft_restcategories>
</resources>
</api2>
</config>
My version file for guest is:-
<?php
class Esoft_Restcategories_Model_Api2_Restapi_Rest_Guest_V1 extends Esoft_Restapi_Model_Api2_Restapi {
/**
* Retrieve list of category list.
*
* #return array
*/
protected function _retrieveCollection()
{
$ruleId = $this->getRequest()->getParam('cat_id');
// $cat_mod = Mage::getModel('catalog/category')->load($ruleId)->toArray();
$cats = Mage::getModel('catalog/category')->load($ruleId);
$subcats = Mage::getModel('catalog/category')->load($ruleId)->getChildren();
$cur_category = array();
$node['category_id'] = $ruleId;
$node['name'] = $cats->getName();
$node['parent_id'] = $cats->getParentId();
$node['child_id'] = $subcats;
if($cats->getIsActive()){
$node['active'] = 1;
}else{
$node['active'] = 0;
}
$node['level'] = $cats->getLevel();
$node['position'] = $cats->getPosition();
$cur_category[] = $node;
// $subcats = Mage::getModel('catalog/category')->load($ruleId)->getAllChildren();
// $subcats = Mage::getModel('catalog/category')->load($ruleId)->getChildren();
if($subcats != '')
{
foreach(explode(',',$subcats) as $subCatid)
{
$_category = Mage::getModel('catalog/category')->load($subCatid);
$childcats = Mage::getModel('catalog/category')->load($subCatid)->getChildren();
$node['category_id'] = $subCatid;
$node['name'] = $_category->getName();
$node['parent_id'] = $_category->getParentId();
$node['child_id'] = $childcats;
if($_category->getIsActive()){
$node['active'] = 1;
}else{
$node['active'] = 0;
}
$node['level'] = $_category->getLevel();
$node['position'] = $_category->getPosition();
$cur_category[] = $node;
}
}
return $cur_category;
}
}
Please let me know how to fix this error.
Also let me know on what basis we define routes.
Simple answer for those searching for cause of this error:
Request does not match any route
Is that you are posting the wrong METHOD such as GET/POST/PUT/DELETE.
It could also be the path of the API url itself is wrong.
As per your description your url format is wrong (route miss matching in given url).
<route>/categories/:cat_id</route>
^^^^^^^^^
So you need to change the url like below
http://localhost:81/magento/api/rest/categories/2