In Load Test - "The cast to value type 'System.Int32' failed because the materialized value is null" - entity-framework

when we are debugging below code with single user it is working fine. But while we tried to run Load test when user count goes more than 30 we are getting
"The cast to value type 'System.Int32' failed because the materialized value is null. Either the result type's generic parameter or the query must use a nullable type." exception.
This is code snippet where we are getting exception.
profile.AccessRights = (
from t in entities.Opportunities
join x in entities.OpportunityWorkloadGroups on t.OpportunityId equals x.OpportunityId into oppworkloads
from ow in oppworkloads.DefaultIfEmpty()
join m in entities.OpportunityWorkloads on ow.OpportunityWorkloadGroupId equals m.OpportunityWorkloadGroupId into oppworkloadmodules
from om in oppworkloadmodules.DefaultIfEmpty()
join y in entities.OpportunityUsers on om.OpportunityWorkloadId equals y.OpportunityWorkloadId into users
from ou in users.DefaultIfEmpty()
where (((t.LookupOpportunityStatu.Name.Equals(Constants.EstimationStatusCompletedName) || t.LookupOpportunityStatu.Name.Equals(Constants.OpportunityStatusDiscardedName)
|| t.LookupOpportunityStatu.Name.Equals(OSEConstants.OpportunityStatusCompletedWithDiscards)) && !t.IsRestricted) ? true : ou.User.Alias == null ? true : ou.User.Alias == alias)
select new Microsoft.OneEstimator.DataAccess.Entities.AccessRights()
{
MasterUserId = userId,
MasterUserRole = roleName,
OpportunityId = t.OpportunityId,
IsOpportunityOwner = t.OpportunityOwner == userId ? true : false,
IsRestricted = t.IsRestricted,
StatusName = t.LookupOpportunityStatu.Name,
OpportunityWorkloadId = ow.OpportunityWorkloads.Any(m => m.OpportunityWorkloadGroupId == ow.OpportunityWorkloadGroupId) ? ow.OpportunityWorkloads.FirstOrDefault(m => m.OpportunityWorkloadGroupId == ow.OpportunityWorkloadGroupId).OpportunityWorkloadId : 0,
EstimationId = ow.OpportunityWorkloadGroupId == null ? 0 : ow.IsActive == false ? 0 : ow.OpportunityWorkloadGroupId,
RoleName = ou.Role == null ? string.Empty : ou.User.Alias == alias ? ou.Role.RoleName : string.Empty,
HasRightsForWA = ow.WorkloadGroupVersion != null ? entities.WorkloadGroupUsers.Any(y => y.WorkloadGroupId == ow.WorkloadGroupVersion.WorkloadGroup.WorkloadGroupId && y.UserId == userId && y.Role.RoleName.Equals(OSEConstants.Estimator)) : true,
WorkstreamUserRoleName = ow.WorkloadGroupVersion != null ? entities.WorkloadGroupUsers.FirstOrDefault(y => y.WorkloadGroupId == ow.WorkloadGroupVersion.WorkloadGroup.WorkloadGroupId && y.UserId == userId).Role.RoleName : string.Empty
}).ToList();
I guess I'm doing something wrong with DefaultIfNull() not sure.

Related

Pyspark AssertionError: on should be Column or list of Column

Hi I have the below dataframes and when I join them I get AssertionError: on should be Column or list of Column. How do I get around this please as I cannot find any solution on google related to this?
Pages = sc.read.json("/Users/me/desktop/clickstream/Clicks/Page*.json.gz")
Pages_Dataset = Pages.select("SessionNumber", "PageLocation", "PageInstanceID")\
.withColumnRenamed("PageLocation", "URL")\
.withColumnRenamed("PageInstanceID", "CsaNumber")\
.withColumn("URL2", expr("CASE WHEN INSTR(URL, '=') > 0 THEN SUBSTR(URL,0,INSTR(URL, '=') -1) ELSE URL END"))\
.withColumn("URL2", expr("CASE WHEN INSTR(URL2, '?') > 0 THEN SUBSTR(URL2,0,INSTR(URL2, '?') -1) ELSE URL2 END"))\
.withColumn("URL2", expr("CASE WHEN INSTR(URL2, '#') > 0 THEN SUBSTR(URL2,0,INSTR(URL2, '#') -1) ELSE URL2 END"))\
.withColumn("URL3", expr("CASE WHEN INSTR(URL, 'prdcls=') > 0 THEN SUBSTR(URL,INSTR(URL, 'prdcls=')+7,2) ELSE '' END"))\
.withColumn("URL", concat("URL2", "URL3"))\
.select("SessionNumber", "URL", "CsaNumber").alias("a")\
.join(ConfiguredUrls.alias("b"), lower("a.URL") == lower("b.URL"), "left")\
.select("a.SessionNumber", "b.Product", "a.CsaNumber", "b.EndQuote", "a.URL")\
.withColumnRenamed("Product", "Session")\
.withColumn("Session", expr("CASE WHEN lower(URL) like 'https://mobilephones.com/deals/%' THEN 'Mobile Phones' ELSE Session END"))\
.withColumn("EndQuote", expr("CASE WHEN lower(URL) like 'https://mobilephones.com/deals/%' THEN 'Mobile Phones' ELSE EndQuote END"))\
.distinct()
Goals_Dataset = Goals.select("SessionNumber", "GoalName", "PageInstanceID", "EventTimestamp")\
.withColumnRenamed("EventTimestamp", "GoalDate")\
.withColumnRenamed("PageInstanceID", "CsaNumber")\
.select("SessionNumber", "GoalName", "CsaNumber", "GoalDate").alias("a")\
.join(ConfiguredGoals.alias("b"), lower("a.GoalName") == lower("b.GoalNameValue"), "left")\
.select("a.SessionNumber", coalesce("b.StartQuote", "b.EndQuote", "b.Switch").alias("Session"), "a.CsaNumber", "b.EndQuote")\
.distinct()
Session_Dataset = Pages_Dataset.select("SessionNumber", "Session", "CsaNumber", "EndQuote").alias("a")\
.join(Goals_Dataset.alias("b"), "a.SessionNumber" == "b.SessionNumber", "fullouter")\
.select(coalesce("a.SessionNumber", "b.SessionNumber").alias("SessionNumber"), coalesce("a.Session", "b.Session").alias("Session"), coalesce("a.CsaNumber", "b.CsaNumber").alias("CsaNumber"), coalesce("a.EndQuote", "b.EndQuote").alias("EndQuote"))\
.distinct()
#Error:
Session_Dataset = Pages_Dataset.select("SessionNumber", "Session", "CsaNumber", "EndQuote").alias("a")\
File "/usr/local/Cellar/apache-spark/3.2.1/libexec/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1343, in join
AssertionError: on should be Column or list of Column
"a.SessionNumber" == "b.SessionNumber" should be col("a.SessionNumber") == col("b.SessionNumber"), or just "SessionNumber"

TypeError: Cannot read property 'markets' of undefined (line 23)

TypeError: Cannot read property 'markets' of undefined (line 23)
I use the same database without touching anything for 6 months, it had never failed, today this error started to appear and I've tried to find the fault but I'm not able to find the fault when collecting the data.
Here is the basis from which I collect the values (you can access the data map by clicking on the link):
https://webhooks.mongodb-stitch.com/api/client/v2.0/app/oddsbf-dvyne/service/http/incoming_webhook/webhook0
Here is the script I use:
const response = UrlFetchApp.fetch("https://webhooks.mongodb-stitch.com/api/client/v2.0/app/oddsbf-dvyne/service/http/incoming_webhook/webhook0");
const mongo_matches = JSON.parse(response.getContentText());
const matches = mongo_matches.map(function(match) {
var over = '-'
var under = '-'
if(match['markets']['over_25'] !== undefined &&
match['markets']['over_25']['over']['odds'] !== undefined &&
(match['markets']['over_25']['over']['odds']['availableToBack'].length > 0 &&
match['markets']['over_25']['under']['odds']['availableToBack'].length > 0 )){
over = match['markets']['over_25']['over']['odds']['availableToBack'][0] !== undefined ? match['markets']['over_25']['over']['odds']['availableToBack'][0]['price']['$numberDouble'] : match['markets']['over_25']['over']['odds']['availableToLay'][0]['price']['$numberDouble']
under = match['markets']['over_25']['under']['odds']['availableToBack'][0] !== undefined ? match['markets']['over_25']['under']['odds']['availableToBack'][0]['price']['$numberDouble'] : match['markets']['over_25']['under']['odds']['availableToLay'][0]['price']['$numberDouble']
}
return [
match['markets']['marketStartTime'],
match['markets']['lastSaved'],
match['markets']['competition'],
match['markets']['homeTeam']['runnerName'],
match['markets']['awayTeam']['runnerName'],
match['markets']['homeTeam']['odds']['availableToBack'][0] !== undefined ? match['markets']['homeTeam']['odds']['availableToBack'][0]['price']['$numberDouble'] : match['markets']['homeTeam']['odds']['availableToLay'][0]['price']['$numberDouble'],
match['markets']['awayTeam']['odds']['availableToBack'][0] !== undefined ? match['markets']['awayTeam']['odds']['availableToBack'][0]['price']['$numberDouble'] : match['markets']['awayTeam']['odds']['availableToLay'][0]['price']['$numberDouble'],
match['markets']['draw']['odds']['availableToBack'][0] !== undefined ? match['markets']['draw']['odds']['availableToBack'][0]['price']['$numberDouble'] : match['draw']['markets']['odds']['availableToLay'][0]['price']['$numberDouble'],
under,
over,
];
});
Line 23 that is giving the error is this one:
match['markets']['draw']['odds']['availableToBack'][0] !== undefined ? match['markets']['draw']['odds']['availableToBack'][0]['price']['$numberDouble'] : match['draw']['markets']['odds']['availableToLay'][0]['price']['$numberDouble'],
Replace line 23 with this:
match['markets']['draw']['odds']['availableToBack'][0] !== undefined ? match['markets']['draw']['odds']['availableToBack'][0]['price']['$numberDouble'] : match['markets']['draw']['odds']['availableToLay'][0]['price']['$numberDouble'],
Update:
Logger.log(mongo_matches['draw'])
Logger.log(mongo_matches['matches'])
both return null. Meaning that you can't access an information from a null object.

ORA-06550: PLS-00103: Encountered the symbol "" with mybatis TypeHandler

I am using Typehandler to map a List<Dep> to oracle array of ... here is the setPArameter method in the handler :
public void setParameter(PreparedStatement ps, int i, List<Dep> parameter, JdbcType jdbcType)
throws SQLException {
Connection connection = ps.getConnection();
// StructDescriptor structDescriptor = StructDescriptor.createDescriptor("MEMS_ARR", connection);
Struct[] structs = null;
if(parameter != null && parameter.size() >0) {
structs = new Struct[parameter.size()];
for (int index = 0; index < parameter.size(); index++)
{
Dep dep = parameter.get(index);
Object[] params = new Object[7];
params[0] = dep.getOrder();
params[1] = dep.getIdTp;
params[2] = dep.getId();
params[3] = " ";
params[4] = " ";
params[5] = " ";
params[6] = " ";
// STRUCT struct = new STRUCT(structDescriptor, ps.getConnection(), params);
structs[index] = connection.createStruct("MEMS", params);
}
// ArrayDescriptor desc = ArrayDescriptor.createDescriptor("MEMS_ARR", ps.getConnection());
// ARRAY oracleArray = new ARRAY(desc, ps.getConnection(), structs);
}else {
parameter = new ArrayList<DependentDTO>();
structs= new Struct[0];
}
this.parameter = parameter;
Array oracleArray = ((OracleConnection) connection).createOracleArray("MEMS_ARR", structs);
ps.setArray(i, oracleArray);
}
and here is the MEMS type :
create or replace TYPE MEMS AS OBJECT
( MEM1 NUMBER(2,0),
MEM2 VARCHAR2(1),
MEM3 VARCHAR2(15),
MEM4 VARCHAR2(60),
MEM5 VARCHAR2(1),
MEM6 VARCHAR2(40),
MEM7 VARCHAR2(10)
);
and here is the portion of the xml mapping file that uses the Typehandler :
#{nat,javaType=String,jdbcType=VARCHAR,mode=IN}, --nat
**#{deps,javaType=List,jdbcType=ARRAY,mode=IN,jdbcTypeName=MEMS_ARR,typeHandler=com.my.package.MyHandler}, --mems**
#{res,javaType=String,jdbcType=VARCHAR,mode=OUT} --res
the error log is as follows :
Error querying database. Cause: java.sql.SQLException: ORA-06550: line 31, column 5: PLS-00103: Encountered the symbol "" when expecting one of the following: . ( ) , * # % & = - + < / > at in is mod remainder not rem => <an exponent (**)> <> or != or ~= >= <= <> and or like like2 like4 likec between || indicator multiset member submultiset The symbol "(" was substituted for "" to continue. ORA-06550: line 44, column 4: PLS-00103: Encountered the symbol ";" when expecting one of the following: . ( ) , * % & = - + < / > at in is mod remainder not rem => <an exponent (**)> <> or != or ~= >= <= <> and or like like2 like4 likec between || multiset ### The error may exist in file [E:\path\to\mapper\ADao.xml] ### The error may involve my.package.ADao.mthodToCall -Inline ### The error occurred while setting parameters ### SQL: {call MY_PROC( ... , --nat?, **--mems? --res**)}
As you can see in the logs, the mems is replaced by empty string or is merged with the next arg res ... the comma is not there
Also kindly note that I already debugged inside the mybatis code and realized that the mapping setParameter method is called and the input List is mapped correctly to the oracle array ... the issue happens at the time of real calling
The issue actually was that I simply missed one comma between two previous parameters ... but the error pointed to the wrong parameter to look at

Cannot resolve element with ID while signing a part of SOAP REQUEST X509

I had the following error while trying to sign a part of SOAP Request :
org.apache.xml.security.utils.resolver.ResourceResolverException: Cannot resolve element with ID _53ea293168db637b15e2d4d7894
at org.apache.xml.security.utils.resolver.implementations.ResolverFragment.engineResolve(ResolverFragment.java:86)
at org.apache.xml.security.utils.resolver.ResourceResolver.resolve(ResourceResolver.java:279)
at org.apache.xml.security.signature.Reference.getContentsBeforeTransformation(Reference.java:417)
at org.apache.xml.security.signature.Reference.dereferenceURIandPerformTransforms(Reference.java:597)
at org.apache.xml.security.signature.Reference.calculateDigest(Reference.java:689)
at org.apache.xml.security.signature.Reference.generateDigestValue(Reference.java:396)
at org.apache.xml.security.signature.Manifest.generateDigestValues(Manifest.java:206)
at org.apache.xml.security.signature.XMLSignature.sign(XMLSignature.java:595)
It comes from the resolution of the URI declared on the reference tag.
Here is the java code i'm using for signing via X509 :
KeyStore.PrivateKeyEntry pke = ISKeyStoreManager.getInstance().getPrivateKeyEntry(keyStoreAlias, keyAlias);
AlgorithmStrings algStrings = AlgorithmStrings.getAlgDSStrings( pke.getPrivateKey(), signatureAlgorithmString, digestAlgorithmString);
String resultantXPath = StringUtils.join(xpaths, '|');
Transforms transforms = new Transforms(originalDocument);
NodeList targetDocumentList = obtainNodesForXPath(originalDocument, resultantXPath, nc);
if(targetDocumentList != null && targetDocumentList.getLength() > 0)
{
if(targetDocumentList.item(0).hasAttributes()){
Node attrId = targetDocumentList.item(0).getAttributes().getNamedItem("Id");
if(attrId != null && !attrId.getNodeValue().equals("")){
uri = new StringBuilder().append('#').append(attrId.getNodeValue()).toString();
}
else{
((Element) targetDocumentList.item(0)).setAttribute("xmlns:wsu", "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd");
((Element) targetDocumentList.item(0)).setAttribute("wsu:Id", idForXmlObject);
}
}
else{
((Element) targetDocumentList.item(0)).setAttribute("xmlns:wsu", "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd");
((Element) targetDocumentList.item(0)).setAttribute("wsu:Id", idForXmlObject);
}
}else{
log.debug("Target not found in the original document with xpath: " + resultantXPath);
}
transforms.addTransform("http://www.w3.org/2000/09/xmldsig#enveloped-signature");
if (resultantXPath != null) {
log.debug("Instantiation XPATHContainer");
XPathContainer xpathC = new XPathContainer(originalDocument);
xpathC.setXPath(resultantXPath);
if ((ncMap != null) && (!ncMap.isEmpty())) {
for (Map.Entry<String,String> e : ncMap.entrySet()) {
log.debug("Adding namespace to XPATH Container: " + e.getKey() + " -> " + e.getValue());
xpathC.setXPathNamespaceContext(e.getKey(), e.getValue());
}
}
transforms.addTransform("http://www.w3.org/TR/1999/REC-xpath-19991116", xpathC.getElement());
}
log.debug("Instantiation Signature");
XMLSignature sig = new XMLSignature(originalDocument, null, algStrings.signatureAlgorithm, canonicalizationAlg);
sig.setFollowNestedManifests(true);
log.debug("Ajout des assertions de transformation");
sig.addDocument("", transforms, algStrings.digestMethod);
if (idAttrForSignature != null) {
sig.setId(idAttrForSignature);
}
log.debug("DOMToString: " + serializeDOMToString(originalDocument));
// signature node insertion
NodeList nodeList = obtainNodesForXPath(originalDocument, insertSignatureAtXPath, nc);
if(nodeList != null && nodeList.getLength() > 0){
Node nodeSignature = nodeList.item(0);
Node childNode = nodeSignature.getFirstChild();
if (childNode != null) {
if (addSignatureAsLastElement)
nodeSignature.appendChild(sig.getElement());
else
nodeSignature.insertBefore(sig.getElement(), childNode);
}
else nodeSignature.appendChild(sig.getElement());
}
else{
throw new ServiceException("INVALID_SIGNATURE_NODE_SELECTOR_XPATH");
}
// Public key insertion
//X509Data x509Data = getX509Data(includeCertChain, certificateData, originalDocument, pke);
//KeyInfoReference kir = new KeyInfoReference(x509Data.getDocument());
SecurityTokenReference str = new SecurityTokenReference(sig.getKeyInfo().getDocument());
str.setKeyIdentifier(ISKeyStoreAccessorUtil.getIaikCertificate(pke.getCertificate()));
sig.getKeyInfo().getElement().appendChild(str.getElement());
log.debug("DOMToString: " + serializeDOMToString(originalDocument));
//sig.getSignedInfo().addResourceResolver(new ResolverXPointer());
((Element)(sig.getSignedInfo().getElement().getElementsByTagNameNS("http://www.w3.org/2000/09/xmldsig#", "Reference").item(0))).setAttribute("URI", uri);
log.debug("DOMToString: " + serializeDOMToString(originalDocument));
//sig.addDocument(uri, trans);
// Signature generation
sig.sign(pke.getPrivateKey());
Do you have any proposition of workaround ? or another way to set URI attribute ?
Thank you for helping !
I found it.
I added InclusiveNamespaces so that the sign method can figure out that ID is on a specific namespace defined attribute.

Filter on Oid using spark-mongo connector

I would like to filter on the objectId of the mongo document from spark program. I have tried the following:
case class _id(oid: String)
val str_start: _id = _id((start.getMillis() / 1000).toHexString + "0000000000000000")
val str_end: _id = _id((end.getMillis() / 1000).toHexString + "0000000000000000")
val filteredDF = df.filter(
$"_timestamp".isNotNull
.and($"_timestamp".between(new Timestamp(start.getMillis()), new Timestamp(end.getMillis())))
.and($"_id").between(str_start, str_end)
or
val str_start = (start.getMillis() / 1000).toHexString + "0000000000000000"
val str_end = (end.getMillis() / 1000).toHexString + "0000000000000000"
val filteredDF = df.filter(
$"_timestamp".isNotNull
.and($"_timestamp".between(new Timestamp(start.getMillis()), new Timestamp(end.getMillis())))
.and($"_id.oid").between(str_start, str_end)
Both give me an error in analysis:
Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot
resolve '(((`_timestamp` IS NOT NULL) AND ((`_timestamp` >= TIMESTAMP('2017-
07-31 00:22:00.0')) AND (`_timestamp` <= TIMESTAMP('2017-08-01
00:22:00.0')))) AND `_id`)' due to data type mismatch: differing types in
'(((`_timestamp` IS NOT NULL) AND ((`_timestamp` >= TIMESTAMP('2017-07-31
00:22:00.0')) AND (`_timestamp` <= TIMESTAMP('2017-08-01 00:22:00.0')))) AND
`_id`)' (boolean and struct<oid:string>).;;
'Filter (((((isnotnull(_timestamp#40) && ((_timestamp#40 >=
1501449720000000) && (_timestamp#40 <= 1501536120000000))) && _id#38) >=
597e4df80000000000000000) && (((isnotnull(_timestamp#40) && ((_timestamp#40
>= 1501449720000000) && (_timestamp#40 <= 1501536120000000))) && _id#38) <=
597f9f780000000000000000))
How can I query on the oid?
Thanks
Nir
I think you are misplaced parenthesis: Should be something like
and($"_id.oid" between(str_start, str_end) )
(so that's why you have the error message:
(boolean and struct<oid:string>)