Generate list of values, based on matching keys - python-3.7

I have the below dictionary:
{'Closed': {'High': 33, 'Medium': 474, 'Low': 47, 'Critical': 6}, 'Impact Statement Pending': {'Low': 3, 'Medium': 1, 'Critical': 0, 'High': 0}, 'New': {'Low': 1, 'High': 2, 'Critical': 2, 'Medium': 2}, 'Remediation Plan Pending': {'Medium': 10, 'Low': 1, 'Critical': 1, 'High': 0}, 'Remedy in Progress': {'Medium': 36, 'Low': 18, 'High': 4, 'Critical': 1}}
How might I accomplish creating a list comprised of all values for a specified key? A list for all high values, or another list for all medium values?
The way I am currently accomplishing this doesn't seem like the best way. I've got a list of all severity levels, which I iterate over and compare such as shown below:
trace_list = ['High', 'Medium', 'Critical', 'Low']
total_status_dict = {'Closed': {'High': 33, 'Medium': 474, 'Low': 47, 'Critical': 6}, 'Impact Statement Pending': {'Low': 3, 'Medium': 1, 'Critical': 0, 'High': 0}, 'New': {'Low': 1, 'High': 2, 'Critical': 2, 'Medium': 2}, 'Remediation Plan Pending': {'Medium': 10, 'Low': 1, 'Critical': 1, 'High': 0}, 'Remedy in Progress': {'Medium': 36, 'Low': 18, 'High': 4, 'Critical': 1}}
for item in trace_labels:
y_values = []
for key, val in total_status_dict.items():
for ke in total_status_dict[key]:
if item is ke:
y_values.append(total_status_dict[key][ke])

Note: you are iterating over total_status_dict keys and appending results to a list. Remember that even if dictionaries are officially ordered in Python since 3.7 (see https://docs.python.org/3/whatsnew/3.7.html) you do not always control the Python version of the user. I would rather build a dict key -> item -> value, where key is Closed, Impact Statement Pending, ... and  item is one of the trace_labels than a dict key -> [values] where values is supposed to be ordered as in trace_labels.
Your code is not efficient because you iterate over trace_labels twice:
for item in trace_labels:
for ke intotal_status_dict[key]: if item is ke:`
How to iterate only once? Instead of building y_values lists one by one (with a whole iteration over total_status_dict each time), you can build several lists at once:
>>> trace_labels = ['High', 'Medium', 'Critical', 'Low']
>>> total_status_dict = {'Closed': {'High': 33, 'Medium': 474, 'Low': 47, 'Critical': 6}, 'Impact Statement Pending': {'Low': 3, 'Medium': 1, 'Critical': 0, 'High': 0}, 'New': {'Low': 1, 'High': 2, 'Critical': 2, 'Medium': 2}, 'Remediation Plan Pending': {'Medium': 10, 'Low': 1, 'Critical': 1, 'High': 0}, 'Remedy in Progress': {'Medium': 36, 'Low': 18, 'High': 4, 'Critical': 1}}
>>> y_values_by_label = {}
>>> for key, value_by_label in total_status_dict.items():
... for label, value in value_by_label.items(): # total_status_dict[key] is value_by_label
... y_values_by_label.setdefault(label, {})[key] = value
...
>>> y_values_by_label
{'High': {'Closed': 33, 'Impact Statement Pending': 0, 'New': 2, 'Remediation Plan Pending': 0, 'Remedy in Progress': 4}, 'Medium': {'Closed': 474, 'Impact Statement Pending': 1, 'New': 2, 'Remediation Plan Pending': 10, 'Remedy in Progress': 36}, 'Low': {'Closed': 47, 'Impact Statement Pending': 3, 'New': 1, 'Remediation Plan Pending': 1, 'Remedy in Progress': 18}, 'Critical': {'Closed': 6, 'Impact Statement Pending': 0, 'New': 2, 'Remediation Plan Pending': 1, 'Remedy in Progress': 1}}
setdefault(label, {}) creates a empty dict y_values_by_label[label] = {} if y_values_by_label does not have the key label.
If you want to turn this in a dict comprehension, you have to use your inefficient method:
>>> {label:{k:v for k, value_by_label in total_status_dict.items() for l, v in value_by_label.items() if l==label} for label in trace_labels}
{'High': {'Closed': 33, 'Impact Statement Pending': 0, 'New': 2, 'Remediation Plan Pending': 0, 'Remedy in Progress': 4}, 'Medium': {'Closed': 474, 'Impact Statement Pending': 1, 'New': 2, 'Remediation Plan Pending': 10, 'Remedy in Progress': 36}, 'Critical': {'Closed': 6, 'Impact Statement Pending': 0, 'New': 2, 'Remediation Plan Pending': 1, 'Remedy in Progress': 1}, 'Low': {'Closed': 47, 'Impact Statement Pending': 3, 'New': 1, 'Remediation Plan Pending': 1, 'Remedy in Progress': 18}}

Related

adding a hashed index on a nested field results in 0 matches in Mongodb

I have a collection with nested fields. Here is a sample document from note_concept_tree collection
{'_id': ObjectId('2d8e5189a4834bc4c1d8f02a'),
'text_key': 123
'concepts':[{'cui':'abc',...},
{'cui':'cde', ...}
]
}
When I search on an unindexed collection in Pymongo with db.note_concept_tree.find_one() I am getting documents. After indexing with:
db.note_concept_tree.create_index([("concepts.cui", "hashed")])
I am getting zero matches with db.note_concept_tree.find_one({"concepts.cui":"abc"}).
After dropping that index with db.note_concept_tree.drop_index('concepts.cui_hashed'), I am getting matches again.
Sorted indices work OK (db.note_concept_tree.create_index([("concepts.cui", "hashed")])), but I assume hashed are supposed to be faster.
What can go wrong and how to debug it?
update:
Hashed query explanation:
{'queryPlanner': {'plannerVersion': 1,
'namespace': 'notedb_test.note_concept_tree',
'indexFilterSet': False,
'parsedQuery': {'concepts.cui': {'$eq': 'C0032326'}},
'queryHash': '3D19D0D4',
'planCacheKey': 'E1B1AF79',
'winningPlan': {'stage': 'FETCH',
'filter': {'concepts.cui': {'$eq': 'C0032326'}},
'inputStage': {'stage': 'IXSCAN',
'keyPattern': {'concepts.cui': 'hashed'},
'indexName': 'concepts.cui_hashed',
'isMultiKey': False,
'isUnique': False,
'isSparse': False,
'isPartial': False,
'indexVersion': 2,
'direction': 'forward',
'indexBounds': {'concepts.cui': ['[-2609654341490927349, -2609654341490927349]']}}},
'rejectedPlans': []},
'executionStats': {'executionSuccess': True,
'nReturned': 0,
'executionTimeMillis': 0,
'totalKeysExamined': 0,
'totalDocsExamined': 0,
'executionStages': {'stage': 'FETCH',
'filter': {'concepts.cui': {'$eq': 'C0032326'}},
'nReturned': 0,
'executionTimeMillisEstimate': 0,
'works': 1,
'advanced': 0,
'needTime': 0,
'needYield': 0,
'saveState': 0,
'restoreState': 0,
'isEOF': 1,
'docsExamined': 0,
'alreadyHasObj': 0,
'inputStage': {'stage': 'IXSCAN',
'nReturned': 0,
'executionTimeMillisEstimate': 0,
'works': 1,
'advanced': 0,
'needTime': 0,
'needYield': 0,
'saveState': 0,
'restoreState': 0,
'isEOF': 1,
'keyPattern': {'concepts.cui': 'hashed'},
'indexName': 'concepts.cui_hashed',
'isMultiKey': False,
'isUnique': False,
'isSparse': False,
'isPartial': False,
'indexVersion': 2,
'direction': 'forward',
'indexBounds': {'concepts.cui': ['[-2609654341490927349, -2609654341490927349]']},
'keysExamined': 0,
'seeks': 1,
'dupsTested': 0,
'dupsDropped': 0}},
'allPlansExecution': []},
'serverInfo': {'host': 'xyz',
'port': 27017,
'version': '4.2.0',
'gitVersion': 'a4b751dcf51dd249c5865812b390cfd1c0129c30'},
'ok': 1.0}
Sorted query explanation:
{'queryPlanner': {'plannerVersion': 1,
'namespace': 'notedb_test.note_concept_tree',
'indexFilterSet': False,
'parsedQuery': {'concepts.cui': {'$eq': 'C0032326'}},
'queryHash': '3D19D0D4',
'planCacheKey': 'E1B1AF79',
'winningPlan': {'stage': 'FETCH',
'inputStage': {'stage': 'IXSCAN',
'keyPattern': {'concepts.cui': 1},
'indexName': 'concepts.cui_1',
'isMultiKey': True,
'multiKeyPaths': {'concepts.cui': ['concepts']},
'isUnique': False,
'isSparse': False,
'isPartial': False,
'indexVersion': 2,
'direction': 'forward',
'indexBounds': {'concepts.cui': ['["C0032326", "C0032326"]']}}},
'rejectedPlans': []},
'executionStats': {'executionSuccess': True,
'nReturned': 856,
'executionTimeMillis': 2,
'totalKeysExamined': 856,
'totalDocsExamined': 856,
'executionStages': {'stage': 'FETCH',
'nReturned': 856,
'executionTimeMillisEstimate': 0,
'works': 857,
'advanced': 856,
'needTime': 0,
'needYield': 0,
'saveState': 6,
'restoreState': 6,
'isEOF': 1,
'docsExamined': 856,
'alreadyHasObj': 0,
'inputStage': {'stage': 'IXSCAN',
'nReturned': 856,
'executionTimeMillisEstimate': 0,
'works': 857,
'advanced': 856,
'needTime': 0,
'needYield': 0,
'saveState': 6,
'restoreState': 6,
'isEOF': 1,
'keyPattern': {'concepts.cui': 1},
'indexName': 'concepts.cui_1',
'isMultiKey': True,
'multiKeyPaths': {'concepts.cui': ['concepts']},
'isUnique': False,
'isSparse': False,
'isPartial': False,
'indexVersion': 2,
'direction': 'forward',
'indexBounds': {'concepts.cui': ['["C0032326", "C0032326"]']},
'keysExamined': 856,
'seeks': 1,
'dupsTested': 856,
'dupsDropped': 0}},
'allPlansExecution': []},
'serverInfo': {'host': 'xyz',
'port': 27017,
'version': '4.2.0',
'gitVersion': 'a4b751dcf51dd249c5865812b390cfd1c0129c30'},
'ok': 1.0}

Retrieve entries from map based on one of the key values

I have reviewed similar posts but am still failing to line things up with what i'm after.
I want to retrieve all entries from the following map where the condition a particular condition is met.
The simple map
List<Map<String, dynamic>> _playList = [
{'songId': 1, 'setId': 1},
{'songId': 2, 'setId': 1},
{'songId': 3, 'setId': 1},
{'songId': 4, 'setId': 2},
{'songId': 5, 'setId': 3},
{'songId': 6, 'setId': 4},
{'songId': 6, 'setId': 5},
{'songId': 6, 'setId': 6},
{'songId': 6, 'setId': 7},
];
I am trying to use a method which will iterate through this map, find all entries for the int that was passed into the method and then store the results in a new array.
The method I have so far
Future<List<Song>> songFromCurrentSetlist (int setlistId) async {
_playList.forEach((_) => setlistId == _playList.???);
}
Clearly, a bit stuck.
Basically, if 1 is received for setlistId in the parameter list for the method, I am expecting to get back the values 1,2,3 for each of the songIds that exist for setlistId == 1.
Thank you,
Bob
You can create a list of songs and add new songs when your setId match.
Following Example Clear your Idea.
void main(){
List<Map<String, dynamic>> _playList = [
{'songId': 1, 'setId': 1},
{'songId': 2, 'setId': 1},
{'songId': 3, 'setId': 1},
{'songId': 4, 'setId': 2},
{'songId': 5, 'setId': 3},
{'songId': 6, 'setId': 4},
{'songId': 6, 'setId': 5},
{'songId': 6, 'setId': 6},
{'songId': 6, 'setId': 7},
];
List<int> songs = [];
songFromCurrentSetlist (int setlistId) async {
_playList.forEach((index){
if(index['setId'] == setlistId){
songs.add(index['songId']);
}
});
}
songFromCurrentSetlist(1);
print(songs.toList());
}
Instead of Map why you not using model class
Exp:
class Songs{
final int songId;
final int setId;
Songs(this.songId, this.setId);}
ListSongs> _playList
try using this

Spark RDDs: How to join value in a Map to a row in an RDD

I have a csv file that I am loading into Spark as an RDD with:
val home_rdd = sc.textFile("hdfs://path/to/home_data.csv")
val home_parsed = home_rdd.map(row => row.split(",").map(_.trim))
val home_header = home_parsed.first
val home_data = home_parsed.filter(_(0) != home_header(0))
home_data then is:
scala> home_data
res17: org.apache.spark.rdd.RDD[Array[String]] = MapPartitionsRDD[3] at filter at <console>:30
scala> home_data.take(3)
res20: Array[Array[String]] = Array(Array("7129300520", "20141013T000000", 221900, "3", "1", 1180, 5650, "1", 0, 0, 3, 7, 1180, 0, 1955, 0, "98178", 47.5112, -122.257, 1340, 5650), Array("6414100192", "20141209T000000", 538000, "3", "2.25", 2570, 7242, "2", 0, 0, 3, 7, 2170, 400, 1951, 1991, "98125", 47.721, -122.319, 1690, 7639), Array("5631500400", "20150225T000000", 180000, "2", "1", 770, 10000, "1", 0, 0, 3, 6, 770, 0, 1933, 0, "98028", 47.7379, -122.233, 2720, 8062))
I also have a csv of zipcodes to neighborhoods loaded as RDD then used to create a map that is a Map[String,String] with:
val zip_rdd = sc.textFile("hdfs://path/to/zipcodes.csv")
val zip_parsed = zip_rdd.map(row => row.split(",").map(_.trim))
val zip_header = zip_parsed.first
val zip_data = zip_parsed.filter(_(0) != zip_header(0))
val zip_map = zip_data.map(row => (row(0), row(1))).collectAsMap
val zip_ind = home_header.indexOf("zipcode") //to get the zipcode column in home_data
Where:
scala> zip_map.take(3)
res21: scala.collection.Map[String,String] = Map(98151 -> Seattle, 98052 -> Redmond, 98104 -> Seattle)
What I am trying to do next is iterate through home_data and use the zipcode value in each row (at zip_ind = 16) to fetch the neighborhood value from zip_map and append that value to the end of the row.
val zip_processed = home_data.map(row => row :+ zip_map.get(row(zip_ind)))
But each time it fetches from zip_map, something is failing and so it only appends None to the end of each row in home_data
scala> zip_processed.take(3)
res19: Array[Array[java.io.Serializable]] = Array(Array("7129300520", "20141013T000000", 221900, "3", "1", 1180, 5650, "1", 0, 0, 3, 7, 1180, 0, 1955, 0, "98178", 47.5112, -122.257, 1340, 5650, None), Array("6414100192", "20141209T000000", 538000, "3", "2.25", 2570, 7242, "2", 0, 0, 3, 7, 2170, 400, 1951, 1991, "98125", 47.721, -122.319, 1690, 7639, None), Array("5631500400", "20150225T000000", 180000, "2", "1", 770, 10000, "1", 0, 0, 3, 6, 770, 0, 1933, 0, "98028", 47.7379, -122.233, 2720, 8062, None))
I am trying to debug this, but am not sure why it's failing at zip_map.get(row(zip_ind)).
I am fairly green with Scala so maybe I am making some bad assumptions, but trying to figure out how to better understand what is happening in the map function.
Map.get() returns None when there is no match. You can use getOrElse to append the Map value with a fall-back:
val home_data = sc.parallelize(Array(
Array("7129300520", "20141013T000000", 221900, "3", "1", 1180, 5650, "1", 0, 0, 3, 7, 1180, 0, 1955, 0, "98178", 47.5112, -122.257, 1340, 5650),
Array("6414100192", "20141209T000000", 538000, "3", "2.25", 2570, 7242, "2", 0, 0, 3, 7, 2170, 400, 1951, 1991, "98125", 47.721, -122.319, 1690, 7639),
Array("5631500400", "20150225T000000", 180000, "2", "1", 770, 10000, "1", 0, 0, 3, 6, 770, 0, 1933, 0, "98028", 47.7379, -122.233, 2720, 8062)
))
val zip_ind = 16
val zip_map: Map[String, String] = Map("98178" -> "A", "98028" -> "B")
val zip_processed = home_data.map(row => row :+ zip_map.getOrElse(row(zip_ind).toString, "N/A"))
zip_processed.collect
// res1: Array[Array[Any]] = Array(
// Array(7129300520, 20141013T000000, 221900, 3, 1, 1180, 5650, 1, 0, 0, 3, 7, 1180, 0, 1955, 0, 98178, 47.5112, -122.257, 1340, 5650, A),
// Array(6414100192, 20141209T000000, 538000, 3, 2.25, 2570, 7242, 2, 0, 0, 3, 7, 2170, 400, 1951, 1991, 98125, 47.721, -122.319, 1690, 7639, N/A),
// Array(5631500400, 20150225T000000, 180000, 2, 1, 770, 10000, 1, 0, 0, 3, 6, 770, 0, 1933, 0, 98028, 47.7379, -122.233, 2720, 8062, B)
// )

Formatting output scala lists

Currently I can search for a value, say SK5 and it will return all values higher than SK5. However the format it is returned in as shown below;
List((SK6,List(2, 7, 5, 9, 1, 9, 8, 4, 1, 7, 3, 7, 0, 8, 4, 5, 9, 2,
4, 4, 8, 7, 9, 2, 2, 7, 9, 1, 6, 9)), (SK4,List(2, 9, 5, 7, 0, 8, 6,
6, 7, 9, 0, 1, 3, 1, 6, 0, 0, 1, 3, 8, 5, 4, 0, 9, 7, 1, 4, 5, 2, 8)),
(SK8,List(2, 8, 8, 3, 1, 1, 0, 8, 5, 9, 0, 3, 1, 6, 8, 7, 9, 6, 7, 7,
0, 9, 5, 2, 5, 0, 2, 1, 8, 6)), (SK9,List(7, 1, 8, 8, 4, 4, 2, 2, 7,
4, 0, 6, 9, 5, 5, 4, 9, 1, 8, 6, 3, 4, 8, 2, 7, 9, 7, 2, 6, 6)),
(SK5,List(2, 6, 8, 0, 3, 5, 5, 2, 5, 9, 4, 5, 3, 5, 7, 8, 8, 2, 5, 9,
3, 8, 6, 7, 8, 7, 4, 1, 2, 3)))
What I want is SK6 - 9, SK4 - 8 etc etc
All bundled together, How would I split this up and only show the last number in the list? I thought I had already filtered this out however apparently not.
Below is my code. Mapdata is saved as Map(String, List[Int])
//functionality to find the last tail element, the "Current" stock price
def findLast(list:List[Int]) = list.last
//8 - Show Stocks Higher Than (W) THIS ONE THIS ONE THIS ONE
def handleEight(): Boolean = {
mnuShowPointsForStockHigher(higherThan2)
true
}
//Returns a list value
def mnuShowPointsForStockHigher(stock: (String) => List[(String, List[Int])]) = {
print("Enter Stock > ")
val data = stock(readLine)
println(s"${data}")
//println(s"${data._1}: ${data._2.last}")
}
def higherThan2(stock: String): List[(String, List[Int])] = {
mapdata.toList.sortWith(_._2.last > _._2.last).takeWhile(row => row._2.last > mapdata.get(stock).map(findLast(_)).getOrElse(0))
}
If you are trying to get the last value in each list, Best option will be map + last. Sorry, don't need to use flatten. My bad .. Should be something line: list.map(x => x.last)

Visualize mongoDb with dcJs (Meteor)

I'm interested to visualize Meteor data (mongoDb) with dcJs. However, I can't find much information about this topic.
I recreated http://www.codeproject.com/Articles/697043/Making-Dashboards-with-Dc-js-Part-2-Graphing using Meteor.
Without calling mongodb,
var data = [
{_id: "iD1", date: "12/27/2012", http_404: 2, http_200: 190, http_302: 100},
{_id: "iD2", date: "12/28/2012", http_404: 2, http_200: 10, http_302: 100},
{_id: "iD3", date: "12/29/2012", http_404: 1, http_200: 300, http_302: 200},
{_id: "iD4", date: "12/30/2012", http_404: 2, http_200: 90, http_302: 0},
{_id: "iD5", date: "12/31/2012", http_404: 2, http_200: 90, http_302: 0},
{_id: "iD6", date: "01/01/2013", http_404: 2, http_200: 90, http_302: 0},
{_id: "iD7", date: "01/02/2013", http_404: 1, http_200: 10, http_302: 1},
{_id: "iD8", date: "01/03/2013", http_404: 2, http_200: 90, http_302: 0},
{_id: "iD9", date: "01/04/2013", http_404: 2, http_200: 90, http_302: 0},
{_id: "iD10", date: "01/05/2013", http_404: 2, http_200: 90, http_302: 0},
{_id: "iD11", date: "01/06/2013", http_404: 2, http_200: 200, http_302: 1},
{_id: "iD12", date: "01/07/2013", http_404: 1, http_200: 200, http_302: 100}
];
The result is working fine.
By inserting and calling data from mongodb, failed.
Not sure what's missing here.
Hope to get advice. Thanks!
I suspect the error is fired from the following code:
var minDate = dateDim.bottom(1)[0].date;
var maxDate = dateDim.top(1)[0].date;
It's probable that your template rendered callback is being fired before the data has been loaded from the database.
Probably the easiest way to fix it would be to put an if statement:
if (Dichas.find().count() !== 0) {
... all of the code that depends on there being data
}