Getting java.lang.UnsupportedOperationException for spark dataset - scala

I am trying to get 25 rows for each key like this:
import spark.implicits._
val record = file.map(rec=>{
var row = rec.date + "," + rec.registrar + "," + rec.agency + "," + rec.state + "," +
rec.district + "," + rec.subDistrict + "," + rec.pinCode + "," + rec.gender + "," +
rec.age + "," + rec.aadharGenerated + "," + rec.rejected + "," + rec.mobileNo + "," +
rec.email
(rec.state,row)
}).groupByKey(_._2).mapGroups((a,b)=>(a,b.toSet.take(25))).collect()
record.foreach(println)
I have tried the other solution but those solutions are not working.
Error stack trace:
Exception in thread "main" java.lang.UnsupportedOperationException: No Encoder found for scala.collection.immutable.Set[(String, String)]
- field (class: "scala.collection.immutable.Set", name: "_2")
- root class: "scala.Tuple2"
at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$serializerFor(ScalaReflection.scala:598)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:592)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:583)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.immutable.List.flatMap(List.scala:355)
at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$serializerFor(ScalaReflection.scala:583)
at org.apache.spark.sql.catalyst.ScalaReflection$.serializerFor(ScalaReflection.scala:425)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:61)
at org.apache.spark.sql.Encoders$.product(Encoders.scala:274)
at org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:47)
at KPI1.Top25$.main(Top25.scala:20)
at KPI1.Top25.main(Top25.scala)

Related

Find fit with custom equation in Matlab

I'm trying to find a fit with a custom equation, but I keep getting errors and don't know why. The custom equation is really big. I'm using what I found here.
This is the code (sorry about the length of the custom equation):
myfittype = fittype('a + (b*2*y) + (c*2*x+dsqrt(6)(2*xy)) + (esqrt(3)*(2*x^2+2*y^2-1)) + (fsqrt(6)(x^2-y^2)) + (gsqrt(8)(3*x^2*y-y^3)) + (hsqrt(8)(3*x^2*y+3*y^3-2*y)) + (ipsqrt(8)(x^3+3*x*y^2-2*x)) + (jpsqrt(8)(x^3-3*xy^2)) + (ksqrt(10)*(4*x^3*y-4*xy^3)+lsqrt(10)*(8*x^3*y+8*x*y^3-6*xy)) + (msqrt(5)*(6*x^4+12*x^2*y^2+6*y^4-6*x^2-6*y^2+1)) + (nsqrt(10)(4*x^4+4*x^2*y^2-3*x^2-4*x^2*y^2-4*y^4+3*y^2)) + (osqrt(10)(x^4-6*x^2*y^2+y^4)) + (psqrt(12)(5*x^4*y-10*x^2*y^3+y^5)) + (qsqrt(12)(15*x^4*y-12*x^2*y-5*y^5+4*y^3+10*x^2*y^3)) + (rsqrt(12)(10*x^4*y+20*x^2*y^3+10*y^5-12*x^2*y-12*y^3+3*y)) + (ssqrt(12)(10*x^5+20*x^3*y^2+10*x*y^4-12*x^3-12*x*y^2+3*x)) + (tsqrt(12)(5*x^5-10*x^3*y^2-4*x^3-15*x*y^4+12*xy^2)) + (usqrt(12)*(x^5-10*x^3*y^2+5*x*y^4))',... 'independent',{'x'},'dependent',{'y'},...
'coefficients',{'a','b','c','d','e','f','g','h','ip','jp','k','l','m','n','o','p','q','r','s','t','u'})
Here is the code formatted with line breaks (...) so that you can read it:
myfittype = fittype('a + (b*2*y) + (c*2*x+d*sqrt(6)*(2*x*y)) + ...
(e*sqrt(3)*(2*x^2+2*y^2-1)) + (f*sqrt(6)*(x^2-y^2)) + ...
(g*sqrt(8)*(3*x^2*y-y^3)) + (h*sqrt(8)*(3*x^2*y+3*y^3-2*y)) + ...
(ip*sqrt(8)*(x^3+3*x*y^2-2*x)) + (jp*sqrt(8)*(x^3-3*x*y^2)) + ...
(k*sqrt(10)*(4*x^3*y-4*x*y^3)+l*sqrt(10)*(8*x^3*y+8*x*y^3-6*x*y)) + ...
(m*sqrt(5)*(6*x^4+12*x^2*y^2+6*y^4-6*x^2-6*y^2+1)) + ...
(n*sqrt(10)*(4*x^4+4*x^2*y^2-3*x^2-4*x^2*y^2-4*y^4+3*y^2)) + ...
(o*sqrt(10)*(x^4-6*x^2*y^2+y^4)) + (p*sqrt(12)*(5*x^4*y-10*x^2*y^3+y^5)) + ...
(q*sqrt(12)*(15*x^4*y-12*x^2*y-5*y^5+4*y^3+10*x^2*y^3)) + ...
(r*sqrt(12)*(10*x^4*y+20*x^2*y^3+10*y^5-12*x^2*y-12*y^3+3*y)) + ...
(s*sqrt(12)*(10*x^5+20*x^3*y^2+10*x*y^4-12*x^3-12*x*y^2+3*x)) + ...
(t*sqrt(12)*(5*x^5-10*x^3*y^2-4*x^3-15*x*y^4+12*x*y^2)) + ...
(u*sqrt(12)*(x^5-10*x^3*y^2+5*x*y^4))', ...
'independent',{'x'},'dependent',{'y'}, ...
'coefficients', ...
{'a','b','c','d','e','f','g','h','ip','jp','k','l','m','n','o','p','q','r','s','t','u'})
The error says this expression is not a valid matlab expression
Hope you can help me, thanks.
I run your code :
The problem of your code is that
Coefficient d does not appear in the equation expression.

Project taking too long to build and run after adding complex query

I have integrated SQLite.swift framework in one of my swift project and everything was working fine until i added the below query. After adding the below query project is taking too long to build. I waited for 30 mins but still project is n't complied.
do
{
let stmt = try DB!.prepare ("SELECT e." + ENDPOINT_ID + " as _id, lk." + HUB_ID + ", e." + X_ENDPOINT_ID + ", e." + ENDPOINT_DESC + ", e." + ENDPOINT_TYPE_ID +", et." + ENDPOINT_STATUS_MIN + ", et." + ENDPOINT_STATUS_MAX + ", e." + ENDPOINT_STATUS + " FROM " + TABLE_ENDPOINT + " as e INNER JOIN " + TABLE_ENDPOINT_TYPE + " as et " +
" ON e." + ENDPOINT_TYPE_ID + " = et." + ENDPOINT_TYPE_ID +
" INNER JOIN " + TABLE_LINKING + " as lk " +
" ON e." + ENDPOINT_ID + " = lk." + ENDPOINT_ID +
" INNER JOIN " + TABLE_NODE + " as n " +
" ON lk." + NODE_ID + " = n." + NODE_ID +
" INNER JOIN " + TABLE_NODE_TYPE + " as nt " +
" ON n." + NODE_TYPE_ID + " = nt." + NODE_TYPE_ID +
" WHERE lk." + SECTION_ID + "=" + section_Id +
" AND nt." + NODE_CATEGORY + " = "S" " +
" ORDER BY e." + ENDPOINT_ID + " ASC")
let arr = Array(try stmt.run)
print("\(arr)")
return arr
} catch {
print("failed: \(error)")
return []
}
If i comment the above code and try to run the project it takes hardly a minute to run the project but after adding this code, it's taking hell lot of time. I have waited for almost 30 mins but still the project is not compiled nor it's throwing any error.
Thanks in advance for help
The Swift compiler has difficulties to deal with big string literals. See also this question for further hints.
I would recommend to split up the sql and to build a string variable in small steps:
var sql = "SELECT e." + ENDPOINT_ID
sql = sql + HUB_ID + ", e."
sql = sql + X_ENDPOINT_ID + ", e."
...
let stmt = try DB!.prepare(sql)
let query = (TABLE_ENDPOINT.select(TABLE_ENDPOINT[ENDPOINT_ID], TABLE_LINKING[HUB_ID], TABLE_ENDPOINT[ETCT_ENDPOINT_ID], TABLE_ENDPOINT[ENDPOINT_DESC], TABLE_ENDPOINT[ENDPOINT_TYPE_ID], TABLE_ENDPOINT_TYPE[ENDPOINT_STATUS_MIN], TABLE_ENDPOINT_TYPE[ENDPOINT_STATUS_MAX], TABLE_ENDPOINT[ENDPOINT_STATUS]).join(TABLE_ENDPOINT_TYPE, on: TABLE_ENDPOINT[ENDPOINT_TYPE_ID] == TABLE_ENDPOINT_TYPE[ENDPOINT_TYPE_ID]).join(TABLE_LINKING, on: TABLE_ENDPOINT[ENDPOINT_ID] == TABLE_LINKING[ENDPOINT_ID]).join(TABLE_NODE, on: TABLE_LINKING[NODE_ID] == TABLE_NODE[NODE_ID]).join(TABLE_NODE_TYPE, on: TABLE_NODE[NODE_TYPE_ID] == TABLE_NODE_TYPE[NODE_TYPE_ID])
.filter(TABLE_LINKING[SECTION_ID] == section_Id && TABLE_NODE_TYPE[NODE_CATEGORY] == "S")
.order(TABLE_ENDPOINT[ENDPOINT_ID].asc))
let arr = Array(try DB!.prepare(query))

create csv file from array of dictionaries in swift

I have an Array of dictionary like this :
The dictionaries are created :
let dateOfSurvey = String(NSDate())
let idOfSurvey = General.codeClasse + "-" + String(General.codePersonnel)
let dictionaryResults = ["dateOfSurvey": dateOfSurvey,
"ID": idOfSurvey,
"themaQuestionStep": questionResultThema,
"actionQuestionStep": questionResultAction,
"F01D1a": questionResult01,
"F02D2c": questionResult02,
"F03D3a": questionResult03,
"F04D4a": questionResult04,
"F05D1b": questionResult05,
"F06D2b": questionResult06,
"F07D3b": questionResult07,
"F08D4b": questionResult08,
"F09D1c": questionResult09,
"F10D2a": questionResult10,
"F11D3c": questionResult11,
"F12D4c": questionResult12]
then added to an array :
arrayOfResults.addObject(dictionaryResults)
I would like to send/export the result by mail and think a CSV file would be a good way.
Is there a simple way to create this csv from the array of dictionaries in swift ?
After no solution for array of dict I have to change the data in a list of arrays
dictionaryResults become arrayOfSurvey
let dateOfSurvey = String(NSDate())
let idOfSurvey = General.codeClasse + "-" + String(General.codePersonnel)
let arrayOfSurvey = [dateOfSurvey , idOfSurvey , questionResultThema , questionResultAction, questionResult01, questionResult02, questionResult03, questionResult04, questionResult05, questionResult06, questionResult07, questionResult08, questionResult09, questionResult10, questionResult11, questionResult12]
I lose the key-value but now be able to put the data in cvs
arrayOfResults.addObject(arrayOfSurvey)
let count = arrayOfResults.count
General.cvs = "dateOfSurvey" + "," + "ID" + "," + "themaQuestionStep" + "," + "actionQuestionStep" + "," + "F01D1a" + "," + "F02D2c" + "," + "F03D3a" + "," + "F04D4a" + "," + "F05D1b" + "," + "F06D2b" + "," + "F07D3b" + "," + "F08D4b" + "," + "F09D1c" + "," + "F10D2a" + "," + "F11D3c" + "," + "F12D4c"
for i in 0...count {
let arrayAtIndex = arrayOfResults.objectAtIndex(i)
let stringRepresentation = arrayAtIndex.componentsJoinedByString(",")
General.cvs += "\n" + stringRepresentation
}
let fileData = General.cvs.dataUsingEncoding(NSUTF8StringEncoding, allowLossyConversion: false)

Retargeting in Fb,Google

I have a campaign and pixel Id for Fb and Google both which looks something like this.
Can some one tell me how can I implement retargeting pixel in FB, google. What are the changes do I need to make in this dynamicaly generated Pixel script?
PixelScript = PixelScript + "<script>(function(){" + Environment.NewLine +
"window._fbds = window._fbds || {};" + Environment.NewLine +
"_fbds.pixelId = " + FB_Pixel + ";" + Environment.NewLine +
"var fbds = document.createElement(\"script\");" + Environment.NewLine +
"fbds.async = true;" + Environment.NewLine +
"fbds.src = \"//connect.facebook.net/en_US/fbds.js\";" + Environment.NewLine +
"var s = document.getElementsByTagName(\"script\")[0];" + Environment.NewLine +
"s.parentNode.insertBefore(fbds, s);" + Environment.NewLine +
"})();" + Environment.NewLine +
"window._fbq = window._fbq || [];" + Environment.NewLine +
"window._fbq.push([\"track\", \"PixelInitialized\", {}]);" + Environment.NewLine +
"</script>" + Environment.NewLine +
"<noscript><img height=\"1\" width=\"1\" border=\"0\" alt=\"\" style=\"display:none\" src=\"https://www.facebook.com/tr?id=" + FB_Pixel + "&ev=NoScript\" /></noscript>";
The given pixel is enough to drop cookies on the website. No need to add anything.

Management API - Create VM - InvalidXmlRequest - The request body XML was invalid or not correctly specified

I am getting error while creating virtual machine from Azure REST management API. Could someone please point me what wrong I am POSTing in below XML
ERROR --- <Error xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Code>InvalidXmlRequest</Code><Message>The request body's XML was invalid or not correctly specified.</Message></Error>
My POST string:
" <Deployment xmlns=\"http://schemas.microsoft.com/windowsazure\" xmlns:i=\"http://www.w3.org/2001/XMLSchema-instance\">" +
" <Name>SomeName</Name>" +
" <DeploymentSlot>Production</DeploymentSlot>" +
" <Label>SomeLabel</Label>" +
" <RoleList>" +
" <Role>" +
" <RoleName>SomeRoleName</RoleName>" +
" <RoleType>PersistentVMRole</RoleType>" +
" <ConfigurationSets>" +
" <ConfigurationSet i:type=\"WindowsProvisioningConfigurationSet\">" +
" <ConfigurationSetType>WindowsProvisioningConfiguration</ConfigurationSetType>" +
" <ComputerName>SomeCompName</ComputerName>" +
" <AdminPassword>Admin098</AdminPassword>" +
" <EnableAutomaticUpdates>true</EnableAutomaticUpdates>" +
" <ResetPasswordOnFirstLogon>false</ResetPasswordOnFirstLogon>" +
" </ConfigurationSet>" +
" <ConfigurationSet i:type=\"NetworkConfigurationSet\">" +
" <ConfigurationSetType>NetworkConfiguration</ConfigurationSetType>" +
" <InputEndpoints>" +
" <InputEndpoint>" +
" <LocalPort>3389</LocalPort>" +
" <Name>RemoteDesktop</Name>" +
" <Protocol>tcp</Protocol>" +
" </InputEndpoint>" +
" </InputEndpoints>" +
" </ConfigurationSet>" +
" </ConfigurationSets>" +
" <OSVirtualHardDisk>" +
" <DiskName>SomeDiskName-0-20121007173943</DiskName>" +
" <MediaLink>http://portalvhdsrx7q659n2n5j2.blob.core.windows.net/public/myvhd0008980.vhd</MediaLink>" +
" <SourceImageName>MSFT__Win2K8R2SP1-Datacenter-201208.01-en.us-30GB.vhd</SourceImageName>" +
" </OSVirtualHardDisk>" +
" <RoleSize>Small</RoleSize>" +
" </Role>" +
" </RoleList>" +
" </Deployment>"
Hi,
Added AdminUsername also. But still getting error. DateTime of request - 04-11 17:30:03.759
<Error xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Code>InvalidXmlRequest</Code><Message>The request body's XML was invalid or not correctly specified.</Message></Error>
POST string
<Deployment xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"> <Name>SomeName</Name> <DeploymentSlot>Production</DeploymentSlot> <Label>SomeLabel</Label> <RoleList> <Role> <RoleName>SomeRoleName</RoleName> <RoleType>PersistentVMRole</RoleType> <ConfigurationSets> <ConfigurationSet i:type="WindowsProvisioningConfigurationSet"> <ConfigurationSetType>WindowsProvisioningConfiguration</ConfigurationSetType> <ComputerName>SomeCompName</ComputerName> <AdminPassword>Admin098</AdminPassword> <AdminUsername>AwesomeAdmin</AdminUsername> <EnableAutomaticUpdates>true</EnableAutomaticUpdates> <ResetPasswordOnFirstLogon>false</ResetPasswordOnFirstLogon> </ConfigurationSet> <ConfigurationSet i:type="NetworkConfigurationSet"> <ConfigurationSetType>NetworkConfiguration</ConfigurationSetType> <InputEndpoints> <InputEndpoint> <LocalPort>3389</LocalPort> <Name>RemoteDesktop</Name> <Protocol>tcp</Protocol> </InputEndpoint> </InputEndpoints> </ConfigurationSet> </ConfigurationSets> <OSVirtualHardDisk> <DiskName>SomeDiskName-0-20121007173943</DiskName> <MediaLink>http://portalvhdsrx7q659n2n5j2.blob.core.windows.net/public/myvhd0008980.vhd</MediaLink> <SourceImageName>MSFT__Win2K8R2SP1-Datacenter-201208.01-en.us-30GB.vhd</SourceImageName> </OSVirtualHardDisk> <RoleSize>Small</RoleSize> </Role> </RoleList> </Deployment>
Try by adding AdminUsername node after AdminPassword node.
" <Deployment xmlns=\"http://schemas.microsoft.com/windowsazure\" xmlns:i=\"http://www.w3.org/2001/XMLSchema-instance\">" +
" <Name>SomeName</Name>" +
" <DeploymentSlot>Production</DeploymentSlot>" +
" <Label>SomeLabel</Label>" +
" <RoleList>" +
" <Role>" +
" <RoleName>SomeRoleName</RoleName>" +
" <RoleType>PersistentVMRole</RoleType>" +
" <ConfigurationSets>" +
" <ConfigurationSet i:type=\"WindowsProvisioningConfigurationSet\">" +
" <ConfigurationSetType>WindowsProvisioningConfiguration</ConfigurationSetType>" +
" <ComputerName>SomeCompName</ComputerName>" +
" <AdminPassword>Admin098</AdminPassword>" +
" <AdminUsername>AwesomeAdmin</AdminUsername>" +
" <EnableAutomaticUpdates>true</EnableAutomaticUpdates>" +
" <ResetPasswordOnFirstLogon>false</ResetPasswordOnFirstLogon>" +
" </ConfigurationSet>" +
" <ConfigurationSet i:type=\"NetworkConfigurationSet\">" +
" <ConfigurationSetType>NetworkConfiguration</ConfigurationSetType>" +
" <InputEndpoints>" +
" <InputEndpoint>" +
" <LocalPort>3389</LocalPort>" +
" <Name>RemoteDesktop</Name>" +
" <Protocol>tcp</Protocol>" +
" </InputEndpoint>" +
" </InputEndpoints>" +
" </ConfigurationSet>" +
" </ConfigurationSets>" +
" <OSVirtualHardDisk>" +
" <DiskName>SomeDiskName-0-20121007173943</DiskName>" +
" <MediaLink>http://portalvhdsrx7q659n2n5j2.blob.core.windows.net/public/myvhd0008980.vhd</MediaLink>" +
" <SourceImageName>MSFT__Win2K8R2SP1-Datacenter-201208.01-en.us-30GB.vhd</SourceImageName>" +
" </OSVirtualHardDisk>" +
" <RoleSize>Small</RoleSize>" +
" </Role>" +
" </RoleList>" +
" </Deployment>"