What is the corrent syntax for filtering on multiple columns in the Scala API? If I want to do something like this:
dataFrame.filter($"col01" === "something" && $"col02" === "something else")
or
dataFrame.filter($"col01" === "something" || $"col02" === "something else")
EDIT:
This is what my original code looks like. Everything comes in as a string.
df.select($"userID" as "user", $"itemID" as "item", $"quantity" cast("int"), $"price" cast("float"), $"discount" cast ("float"), sqlf.substring($"datetime", 0, 10) as "date", $"group")
.filter($"item" !== "" && $"group" !== "-1")
I think i see what the issue is. For some reason, spark does not allow two !='s in the same filter. Need to look at how filter is defined in Spark source code.
Now for your code to work, you can use this to do the filter
df.filter(col("item").notEqual("") && col("group").notEqual("-1"))
or use two filters in same statement
df.filter($"item" !== "").filter($"group" !== "-1").select(....)
This link here can help with different spark methods.
Related
I am trying to filter my algolia results by boolean and date. The problem I have is that the results found do not match the items shown to the user.
Here is what I have so far:
transformItems(items) {
return items.filter(item => {
return (item.enabled === 0 || item.time_of_use_unix < moment().unix()) ? null : item;
});
},
This shows the right items, but does not update my results found number (nbHits).
Is there maybe another way to achieve this?
I found a link which describes something similar, but I don't know how to use it in vue js.
https://www.algolia.com/doc/guides/managing-results/refine-results/filtering/how-to/filter-by-boolean/#applying-a-boolean-filter
If someone could help I would appreciate that.
Solved it with <ais-configure :filters="filters" />
{
"cars": {
"Nissan": {
"Sentra": {"doors":4, "transmission":"automatic"},
"Maxima": {"doors":4, "transmission":"automatic","colors":["b#lack","pin###k"]}
},
"Ford": {
"Taurus": {"doors":4, "transmission":"automatic"},
"Escort": {"doors":4, "transmission":"auto#matic"}
}
}
}
I have this JSON that I have read, and I want to remove every # symbol in every string that may exist. My problem is doing this function generic, so it could work on every schema that I may encounter and not only this schema as used in JSON above.
You could do something like this: Get all the fields from the schema, use fold with the DataFrame itself as an accumulator and, apply the function that you want
def replaceSymbol(df: DataFrame): DataFrame =
df.schema.fieldNames.foldLeft(df)((df, field) => df.withColumn(field, regexp_replace(col(field), "#", "")))
You might need to check if the column is String or not.
I have a column called Account Verification which return values either true or false. I formatted the value on table to Yes and No, but when filtering the column, I still have to search true or false to be able to filter the column. I made a custom filter condition but it did not work. Anyone has solution?
columns = [
{
headerName: 'Account Verification', field: 'accountVerified', filter: 'agTextColumnFilter',
// Cell renderer
valueFormatter: (data) => {
if (data.value === true) return 'Yes';
else return 'No';
},
// Custom filter
filterParams: {
condition: (searchTerm, cellValue) => {
if (searchTerm === 'Yes' || 'yes') {
return cellValue === true;
} else if (searchTerm === 'No' || 'no') {
return cellValue === true;
} else return cellValue === null;
}
}
}
]
"ag-grid": "^18.0.1"
I think the best way to implement this - is to use built in methods of ag-grid, like:
onFilterModified
onFilterChanged
And there provide the logic, which would be transform the filter value to true/false.
Or, use javascript to add event listener on apply button of the filter (if you are using apply button to apply your filter), which would call fucntion for transform your value.
Have no example in the moment, but I develped solution for filter validation (custom) in the same way.
Try one of these options:
Similar to to valueFormatter in your column definition, use a valueGetter also. There is also this method available in column definition (filterValueGetter(params) - Function or expression. Gets the value for filtering purposes.)
You can add a new field in your data model that you initialize to Yes/No based on the values of the original field that has true/false. Then in the column definitions, you can use this new field to display in the grid.
Alternatively, you can write a custom filter component. See the custom Angular filter example on the ag-grid site.
You can use TextFormatter within the filterParams to modify the filter value during filtering:
filterParams: { textFormatter: (filterValue) => 'yourvalueformatter' }
https://www.ag-grid.com/javascript-grid-filter-text/#text-formatter
I had the same issue and went through all the inbuilt filters.
Finally I was able to fix the issue by following the below steps
Steps :
Install the ag-grid-enterprise package. (npm install ag-grid-enterprise)
Set the filter to agSetColumnFilter. (filter: 'agSetColumnFilter')
Add filterParams and specify the same valueFormatter which you used in the column definition
filterParams: {
valueFormatter: (data) => {
if (data.value === true) return 'Yes';
else return 'No';
}
}
You can try adding this to your column definition:
filterValueGetter: params => params.data.accountVerified ? "Yes" :
"No"
I am trying to handle errors using findOne in meteor-mongo.
From this stackoverflow question, it appears that I should be able to handle errors by doing collection.findOne({query}, function(err, result){ <handleError> }, but doing so results in an errormessage:
"Match error: Failed Match.OneOf, Match.Maybe or Match.Optional validation"
The following code works:
export default createContainer((props) => {
let theID = props.params.theID;
Meteor.subscribe('thePubSub');
return {
x: theData.findOne({_id: theID}),
};
}, App);
The following code does not:
export default createContainer((props) => {
let theID = props.params.theID;
Meteor.subscribe('thePubSub');
return {
x: theData.findOne({_id: theID}, function(err,result){
if(!result){
return {}
};
}),
};
}, App);
What am I doing wrong and how should I be resolving this error? Is this a meteor specific error?
Any help is greatly appreciated!
What kind of error are you exactly trying to handle with your callback?
Meteor's findOne is different from node's mongodb driver's findOne that the post you link to uses.
The expected signature is:
collection.findOne([selector], [options])
There is no callback involved, since the method runs synchronously (but is reactive).
If you want to return a default value when the document is not found, you can simply use a JS logical OR:
// Provide an alternative value on the right that will be used
// if the left one is falsy.
theData.findOne({_id: theID}) || {};
A more rigorous approach would be to compare its type with
typeof queryResult === 'undefined'
Note that if theData collection is fed by the above subscription Meteor.subscribe('thePubSub'), I doubt Meteor will have time to populate the collection on the client by the time you query it…
So what I want to do is make findOne work more like it does in Meteor, but through the Mongo shell. In short I want to be able to do something like this db.collection.findOne("thisIsAnId") and have it lookup that id in that collection.
I tried loading a file that has this in it...
db.collection.findOne = function(query, fields, options){
if(typeof query === "string") {
return db.collection.originalFindOne({_id : query}, fields, options);
}
return db.collection.originalFindOne(query, fields, options);
}
Where originalFindOne would just chain to the default findOne, this didn't work at all. So after having no luck finding a way to override a default function I thought maybe I could create a new function like db.collection.simpleFindOne() or something, but I can't find a way to attach it to the mongo shell so that it will be available to any collection.
Anyone have some insight on how mongo internals work that could give me some help?
Try adding this snippet to one of your Mongo config files:
(function() {
// Duck-punch Mongo's `findOne` to work like Meteor's `findOne`
if (
typeof DBCollection !== "undefined" && DBCollection &&
DBCollection.prototype && typeof DBCollection.prototype.findOne === "function" &&
typeof ObjectId !== "undefined" && ObjectId
) {
var _findOne = DBCollection.prototype.findOne,
_slice = Array.prototype.slice;
DBCollection.prototype.findOne = function() {
var args = _slice.call(arguments);
if (args.length > 0 && (typeof args[0] === "string" || args[0] instanceof ObjectId)) {
args[0] = { _id: args[0] };
}
return _findOne.apply(this, args);
};
}
})();
I originally found the DBCollection object/class by typing db.myCollection.constructor into the Mongo shell. I then confirmed that findOne was defined on its prototype by verifying that (a) DBCollection was globally accessible, (b) that DBCollection.prototype existed, and (c) that typeof DBCollection.prototype.findOne === "function" (much like the snippet does).
Edit: Added a logic branch to also cover ObjectId-based IDs.