I want to check values in one table against a mapping table.
Data Table:
ID
Color
Object
001
Green
Grass
002
Green
Tree
003
Green
Sky
004
Green
Apple
005
Red
Apple
006
Red
Poppy
007
Red
Water
Allowed Mappings
MappingID
MappingKey
MappingValue
M001
Green
Grass
M002
Green
Tree
M003
Green
Apple
M004
Red
Apple
M005
Red
Poppy
The expected output is the entries that are not foreseen in the mapping table.
Output:
ID
Color
Object
003
Green
Sky
007
Red
Water
Here are the dataframe definitions so you can get started more easily:
data_df = spark.createDataFrame(
[
("001", "Green", "Grass"),
("002", "Green", "Tree"),
("003", "Green", "Sky"),
("004", "Green", "Apple"),
("005", "Red", "Apple"),
("006", "Red", "Poppy"),
("007", "Red", "Water")
],
["ID", "Color", "Object"],
)
mapping_df = spark.createDataFrame(
[
("M001", "Green", "Grass"),
("M002", "Green", "Tree"),
("M003", "Green", "Apple"),
("M004", "Red", "Apple"),
("M005", "Red", "Poppy")
],
["MappingID", "MappingKey", "MappingValue"],
)
You can use left_anti join.
join_on=[
data_df.Color == mapping_df.MappingKey,
data_df.Object == mapping_df.MappingValue
]
df = data_df.join(mapping_df, on=join_on, how='left_anti')
Related
A complex JSON file looks like below.
{"Activename": "ZenerDiode", "composition": " BACKWARDS\n3 Zener voltages and some\n1 Zener voltages and some\n2 sounds electronic circuits gradual junction (silicon)\n Zener effect\n1 supply rails, Diced\n3 higher voltage and to provide, reference\n4 Paste\n4 electronic circuits Stock Or Broth, More If Needed For Thinning\n Several Dashes Worcestershire\n1/2 teaspoon Sugar\n4 whole Carrots, Peeled And Diced\n2 whole Turnips, Peeled And Diced\n2 Tablespoons Minced Fresh Parsley\n Mashed Potatoes\n5 pounds Russet Potatoes (peeled)\n1 package (8 Ounce) Cream Cheese, Softened\n1 stick Butter, Softened\n1/2 cup Heavy Cream\n1 teaspoon Seasoned Salt\n Salt And Pepper, to taste", "url": "https://en.wikipedia.org/wiki/Zener_diode", "image": "https://en.wikipedia.org/wiki/Zener_diode.jpg", "reverseTime": "PT3H", "Yield": "8", "datePublished": "2013-01-14", "upTime": "PT15M", "description": "Yesterday was cold and windy and shivery and frigid, and to psychologically withstand such things, I made Sunday Night Stew a..."}
{"Activename": "Coil Capacitor", "composition": "2 Tablespoons Butter\n2 whole Large Onions, Halved And Sliced Thin\n1/4 cup Broth\n7 dashes Worcestershire Sauce\n Splash Of Red Or White Wine\n1/2 cup Grated Gruyere Cheese (can Use Swiss)\n Kosher Salt\n24 whole White Or Crimini Mushrooms, Washed And Stems Removed\n Minced Parsley", "url": "https://en.wikipedia.org/wiki/Zener_diode", "image": "https://en.wikipedia.org/wiki/Zener_diodejpg", "reverseTime": "PT30M", "Yield": "8", "datePublished": "2010-11-23", "upTime": "PT20M", "description": "Important note: this recipe has absolutely nothing to do with Thanksgiving. I'm so glad I got that out. I feel cleansed! ..."}
{"Activename": "Metal Inductor", "composition": " STEW\n3 Tablespoons Olive Oil\n1 Tablespoon Butter\n2 pounds electronic circuits Stew Meat (chuck Roast Cut Into Chunks)\n Salt And Pepper\n1 whole Medium Onion, Diced\n3 cloves Garlic, Minced\n4 ounces, weight Tomato Paste\n4 cups Low electronic circuits Stock Or Broth, More If Needed For Thinning\n Several Dashes Worcestershire\n1/2 teaspoon Sugar\n4 whole Carrots, Peeled And Diced\n2 whole Turnips, Peeled And Diced\n2 Tablespoons Minced Fresh Parsley\n Mashed Potatoes\n5 pounds Russet Potatoes (peeled)\n1 package (8 Ounce) Cream Cheese, Softened\n1 stick Butter, Softened\n1/2 cup Heavy Cream\n1 teaspoon Seasoned Salt\n Salt And Pepper, to taste", "url": "https://en.wikipedia.org/wiki/Zener_diode", "image": "https://en.wikipedia.org/wiki/Zener_diode.jpg", "reverseTime": "PT3H", "Yield": "8", "datePublished": "2013-01-14", "upTime": "PT15M", "description": "Yesterday was cold and windy and shivery and frigid, and to psychologically withstand such things, I made Sunday Night Stew a..."}
i have created a Dataframe out of it.
when i tried to filter only those rows from dataframe whose composition contain electronic circuits like
ds = df_concat.filter(lower("composition").like("%electronic circuits%"))
but it is not giving any records.
Please suggest.
Thank you very much.
Based on your sample data , I can see the records are getting filtered
Data Preparation
js = [
{"Activename": "ZenerDiode", "composition": " BACKWARDS\n3 Zener voltages and some\n1 Zener voltages and some\n2 sounds electronic circuits gradual junction (silicon)\n Zener effect\n1 supply rails, Diced\n3 higher voltage and to provide, reference\n4 Paste\n4 electronic circuits Stock Or Broth, More If Needed For Thinning\n Several Dashes Worcestershire\n1/2 teaspoon Sugar\n4 whole Carrots, Peeled And Diced\n2 whole Turnips, Peeled And Diced\n2 Tablespoons Minced Fresh Parsley\n Mashed Potatoes\n5 pounds Russet Potatoes (peeled)\n1 package (8 Ounce) Cream Cheese, Softened\n1 stick Butter, Softened\n1/2 cup Heavy Cream\n1 teaspoon Seasoned Salt\n Salt And Pepper, to taste", "url": "https://en.wikipedia.org/wiki/Zener_diode", "image": "https://en.wikipedia.org/wiki/Zener_diode.jpg", "reverseTime": "PT3H", "Yield": "8", "datePublished": "2013-01-14", "upTime": "PT15M", "description": "Yesterday was cold and windy and shivery and frigid, and to psychologically withstand such things, I made Sunday Night Stew a..."}
,{"Activename": "Coil Capacitor", "composition": "2 Tablespoons Butter\n2 whole Large Onions, Halved And Sliced Thin\n1/4 cup Broth\n7 dashes Worcestershire Sauce\n Splash Of Red Or White Wine\n1/2 cup Grated Gruyere Cheese (can Use Swiss)\n Kosher Salt\n24 whole White Or Crimini Mushrooms, Washed And Stems Removed\n Minced Parsley", "url": "https://en.wikipedia.org/wiki/Zener_diode", "image": "https://en.wikipedia.org/wiki/Zener_diodejpg", "reverseTime": "PT30M", "Yield": "8", "datePublished": "2010-11-23", "upTime": "PT20M", "description": "Important note: this recipe has absolutely nothing to do with Thanksgiving. I'm so glad I got that out. I feel cleansed! ..."}
,{"Activename": "Metal Inductor", "composition": " STEW\n3 Tablespoons Olive Oil\n1 Tablespoon Butter\n2 pounds electronic circuits Stew Meat (chuck Roast Cut Into Chunks)\n Salt And Pepper\n1 whole Medium Onion, Diced\n3 cloves Garlic, Minced\n4 ounces, weight Tomato Paste\n4 cups Low electronic circuits Stock Or Broth, More If Needed For Thinning\n Several Dashes Worcestershire\n1/2 teaspoon Sugar\n4 whole Carrots, Peeled And Diced\n2 whole Turnips, Peeled And Diced\n2 Tablespoons Minced Fresh Parsley\n Mashed Potatoes\n5 pounds Russet Potatoes (peeled)\n1 package (8 Ounce) Cream Cheese, Softened\n1 stick Butter, Softened\n1/2 cup Heavy Cream\n1 teaspoon Seasoned Salt\n Salt And Pepper, to taste", "url": "https://en.wikipedia.org/wiki/Zener_diode", "image": "https://en.wikipedia.org/wiki/Zener_diode.jpg", "reverseTime": "PT3H", "Yield": "8", "datePublished": "2013-01-14", "upTime": "PT15M", "description": "Yesterday was cold and windy and shivery and frigid, and to psychologically withstand such things, I made Sunday Night Stew a..."}
]
jsonRDD = sc.parallelize(js)
sparkDF = sql.read.json(jsonRDD)
sparkDF.show()
+--------------+-----+--------------------+-------------+--------------------+--------------------+-----------+------+--------------------+
| Activename|Yield| composition|datePublished| description| image|reverseTime|upTime| url|
+--------------+-----+--------------------+-------------+--------------------+--------------------+-----------+------+--------------------+
| ZenerDiode| 8| BACKWARDS
3 Zene...| 2013-01-14|Yesterday was col...|https://en.wikipe...| PT3H| PT15M|https://en.wikipe...|
|Coil Capacitor| 8|2 Tablespoons But...| 2010-11-23|Important note: t...|https://en.wikipe...| PT30M| PT20M|https://en.wikipe...|
|Metal Inductor| 8| STEW
3 Tablespoo...| 2013-01-14|Yesterday was col...|https://en.wikipe...| PT3H| PT15M|https://en.wikipe...|
+--------------+-----+--------------------+-------------+--------------------+--------------------+-----------+------+--------------------+
Filter
sparkDF.filter(F.col('composition').like('%electronic circuits%')).show()
+--------------+-----+--------------------+-------------+--------------------+--------------------+-----------+------+--------------------+
| Activename|Yield| composition|datePublished| description| image|reverseTime|upTime| url|
+--------------+-----+--------------------+-------------+--------------------+--------------------+-----------+------+--------------------+
| ZenerDiode| 8| BACKWARDS
3 Zene...| 2013-01-14|Yesterday was col...|https://en.wikipe...| PT3H| PT15M|https://en.wikipe...|
|Metal Inductor| 8| STEW
3 Tablespoo...| 2013-01-14|Yesterday was col...|https://en.wikipe...| PT3H| PT15M|https://en.wikipe...|
+--------------+-----+--------------------+-------------+--------------------+--------------------+-----------+------+--------------------+
The attribute table of the .shp file has the following format:
street_name start_node end_node
street_1 A B
street_1 B C
street_2 B D
How could I create links using the start and end nodes and then assign to each link the street name associated with its start and end node. For example, the link with start node A and end node B should get the name "street_1" and the street with start node B and end node D should get the name "street_2".
I used foreach gis:feature-list-of to link the nodes of the dataset but this way I cannot name the links based on their start and end nodes because some of the nodes are shared between street segments.
Many thanks.
Edit:
The columns of the attribute table I am interested in are name1, startNode, and endNode. I have already connected the nodes using the code below and I now have a fully connected road network. I am unsure how I could integrate your code so as the links between the nodes will get the name associated with the combination of nodes that form that link.
foreach gis:feature-list-of roads-dataset [ vector-feature ->
foreach gis:vertex-lists-of vector-feature [ vertex ->
let previous-turtle nobody
foreach vertex [point ->
let location gis:location-of point
if not empty? location
[
let x item 0 location
let y item 1 location
let current-node one-of (turtles-on patch x y) with [ xcor = x and ycor = y ]
if current-node = nobody [
create-nodes 1 [
setxy x y
set size 0.2
set shape "circle"
set color black
set hidden? true
set name gis:property-value vector-feature "name1"
set current-node self
]
]
ask current-node [
if is-turtle? previous-turtle [
create-link-with previous-turtle
]
set previous-turtle self
]
]
]
]
]
Are you saying that your nodes are now correctly named in your model? If that's the case, here's a simplified version of an approach that may work for you. I will note that this is not a very efficient way to go about this as it loops through your links and your attribute table, so if you have many many links it will take a while. To start out with, since I don't have your shapefile, I have made a version of your links example:
extensions [csv]
globals [ whole-file ]
turtles-own [ node ]
links-own [ name ]
to setup
ca
reset-ticks
let names [ "A" "B" "C" "D" ]
let n 0
crt 4 [
setxy random 30 - 15 random 30 - 15
set node item n names
set n n + 1
]
ask turtles with [ node = "A" ] [
create-links-to turtles with [node = "B" ]
]
ask turtles with [ node = "B" ] [
create-links-to turtles with [ node = "C" or node = "D" ]
]
end
That just builds four turtles, with links as indicated in your example shapefile attribute table. I'm using a file called "node_example.csv" that looks like this:
street_name start_node end_node
1 street_1 A B
2 street_1 B C
3 street_2 B D
with four columns, where the first is the observation number.
Essentially, the approach is to iterate through the list and pull out the names of the nodes, from end1 to end2 and vice versa (since both-ends would pull them in a random order), and compare them to each start_node and end_node combination in the table. If they match, assign the street_name from that row to the link with the match:
to link-name
set whole-file csv:from-file "node_example.csv"
foreach sort links [
[ i ] ->
show i
let way-1 list ( [node] of [end1] of i ) ( [node] of [end2] of i )
let way-2 list ( [node] of [end2] of i ) ( [node] of [end1] of i )
foreach whole-file [
[j] ->
if sublist j 2 4 = way-1 or sublist j 2 4 = way-2 [
ask i [
set name item 1 j
]
]
]
]
ask links [
print name
print (word [node] of end1 [node] of end2 )
]
end
Obviously, this is predicated on your nodes being named in your model (in this example, the variable used is node)- if that's not the case this won't work.
Edit 1
Ok, I played around a little using your shapefile. This is not perfect yet, and I can't work on it any more for a while, but maybe it will get you started. Using this setup:
extensions [gis]
breed [ nodes node ]
globals [ roads-dataset ]
turtles-own [ name line-start line-end]
links-own [ lname ]
My idea is to assign the start and end node names to each point along the line feature, so that the links can check against the feature list. More specific notes in comments, but I've basically modified your gis-feature-node code to do that. Play around with it a bit (takes a while to run) and you'll see there are gaps that I haven't quite figured out- maybe you can make progress.
to gis-feature-node
set roads-dataset gis:load-dataset "road_links.shp"
foreach gis:feature-list-of roads-dataset [ vector-feature ->
; First, grab the names of the starting and ending node for the current
; vector feature in order to assign common names to all nodes within
; the feature
let first-vertex gis:property-value vector-feature "startNode"
let last-vertex gis:property-value vector-feature "endNode"
foreach gis:vertex-lists-of vector-feature [ vertex ->
let previous-turtle nobody
foreach vertex [ point ->
let location gis:location-of point
if not empty? location
[
let x item 0 location
let y item 1 location
let current-node one-of (turtles-on patch x y) with [ xcor = x and ycor = y ]
if current-node = nobody [
create-nodes 1 [
setxy x y
set size 0.05
set shape "circle"
set color white
set hidden? false
set name gis:property-value vector-feature "name1"
; Here you assign the first-vertex and last-vertex of the entire line
; to each node
set line-start first-vertex
set line-end last-vertex
set current-node self
]
]
ask current-node [
if is-turtle? previous-turtle [
create-link-with previous-turtle
]
set previous-turtle self
]
]
]
]
]
ask links [
;; Here is a major slowdown- reiterate through the entire roads-dataset
; and, if the names in "startNode" and "endNode" match, assign the
; value from "name1" to the link currently being created.
let way-1 list [line-start] of end1 [line-end] of end2
let way-2 list [line-end] of end1 [line-start] of end2
foreach gis:feature-list-of roads-dataset [ vector-feature-sub ->
let vector-start gis:property-value vector-feature-sub "startNode"
let vector-end gis:property-value vector-feature-sub "endNode"
let start-end list vector-start vector-end
if way-1 = start-end or way-2 = start-end [
set lname gis:property-value vector-feature-sub "name1"
]
]
]
ask links with [ lname = "Hamilton Place" ] [
set color red
set thickness 0.2
]
ask links with [ lname = "Whitcomb Street" ] [
set color yellow
set thickness 0.2
]
end
EDIT 2
The code below is tested and works - problem sorted.
ask links [
set is-road? true
;; Here is a major slowdown- reiterate through the entire roads-dataset
; and, if the names in "startNode" and "endNode" match, assign the
; value from "name1" to the link currently being created.
let way-1 list [line-start] of end1 [line-end] of end2
let way-2 list [line-end] of end1 [line-start] of end2
let way-3 list [ line-start ] of end1 [ line-end ] of end1
let way-4 list [ line-start ] of end2 [ line-end ] of end2
foreach gis:feature-list-of roads-dataset [ vector-feature-sub ->
let vector-start gis:property-value vector-feature-sub "startNode"
let vector-end gis:property-value vector-feature-sub "endNode"
let start-end list vector-start vector-end
let end-start list vector-end vector-start
if way-1 = start-end or way-2 = start-end or way-3 = start-end or way-4 = start-end [
set lname gis:property-value vector-feature-sub "name1"
]
]
]
In NetLogo, suppose there are two kinds (breeds) of turtles: AAA and BBB.
breed [ AAA ]
breed [ BBB ]
AAA-own [ vvv ]
BBB-own [ vvv ]
Suppose I am iterating over AAA, and when an AAA finds a BBB nearby, it steals 10% of vvv from the BBB individual. If there is a global variable called dummy, the following code may work:
to function-name
let QQQ one-of BBB in-radius 1
ask QQQ [
set dummy vvv * 0.1
set vvv vvv - dummy
]
set vvv vvv + dummy
end
Is there any way to do the similar thing without using the global variable, dummy?
Use myself for the turtle who asked.
Suppose I have a data file with x y z column, which looks like:
-3.063052922487259 -3.141592741012573 401.3000000000000
-3.063052922487259 -3.063052922487259 1.290000000000000
-3.063052922487259 -2.984513103961945 0.920000000000000
-2.984513103961945 -3.141592741012573 0.100000000000000
-2.984513103961945 -3.063052922487259 10.80000000000000
-2.984513103961945 -2.984513103961945 1001.290000000000
-2.905973285436630 -2.984513103961945 514.4000000000000
-2.905973285436630 -2.905973285436630 131.0300000000000
-2.905973285436630 -2.827433466911316 129.3300000000000
The range of the values within the z column will define the color of the data points. For example, on the z column, if the value is between 0.0 and 0.3, the color of data points will be set as blue; if between 0.3 and 1, the color of data points will be set as orange; if between 400 and 1000, the color of data points will be set as navy.
So I write some code like this:
set xrange [0:15]
set yrange [0:-15]
set zrange [0:1400]
set cbrange [0.001:1400]
set palette defined ( 0 "goldenrod", 0.3 "blue", 1 "orange", 2 "cyan", 4 "yellow", 10 "green", 20 "pink", 50 'gold', 100 'purple', 400 'navy', 1000 "red")
set palette maxcolors 11
unset key
unset surface
splot "DATA.dat" using 1:2:3 with image
Which does not work. Any help?
Further update:
I really want to explain clearer why does not work, but Stack Overflow does not allow me to further explain, because they require me to have 10 reputation points in order to post an image result. So I can not post my result due to lacking of reputation. But I do modify my data set, so you can see the xy-data is equidistance now.
So I just describe the problem by words, instead of image, which is that the color box is wrong. According to my code, between 400 and 1000 should be navy (just one color). But the color box on the image shows that between 400 and 1000, there are 5 different navy colors, from shallow navy to deep navy. How can I only have one navy color between 400 and 1000 please?
The maxcolors option doesn't work properly in your case because it only discretizes the underlying color gradient. You can use the test palette command to see how your actual palette looks like:
set palette defined ( 0 "goldenrod", 0.3 "blue", 1 "orange", 2 "cyan", 4 "yellow", 10 "green", 20 "pink", 50 'gold', 100 'purple', 400 'navy', 1000 "red")
set palette maxcolors 11
test palette
You must also keep in mind, that the numbers used in the palette definition aren't absolute values on the cb-axis, but the values (in your case from 0 to 1000) are mapped to the actual cbrange (0.001 to 1400).
In order to get regions with constant color value, you do the following:
set palette defined (0 "goldenrod", \
0 "blue", 0.3 "blue", \
0.3 "orange", 1 "orange", \
1 "cyan", 2 "cyan", \
2 "yellow", 4 "yellow", \
4 "green", 10 "green", \
10 "pink", 20 "pink", \
20 "gold", 50 "gold", \
50 "purple", 400 "purple", \
400 "navy", 1000 "navy", \
1000 "red", 1400 "red")
test palette
For a project, I'm developping a simulation in NetLogo dealing with rabies diseases in dogs and humans. I have some turtles-humans with dogs that can be vaccinated or not. At the beginning I create a dog with rabie and, in according to the fase (1 or 2) of the disease, it can spread the disease to other dogs with a probability. At the end the dog can die either for paralysis (if a probability is higher than 75%) or for other complications. Here's the code:
http://pastebin.com/esR75G3T
In the end you can see that a dog not dying for paralysis will die after some days (between 4 or 6). In other words when the days_infected are equal to end-life.
To check if everything is ok at the beginning I tried to set that NONE of the dog is vaccinated so everyone is supposed to get the disease. In fact when the dog is in phase 2 it will bite anyone. The problem is that if I delete the last line of the code, everything works and some dogs die of paralysis and the other remain alive. If I enable also the last line to let the other dogs die too, nothing works...no dog is infected. why?
This is not a problem with your code: this is a problem with the dynamics of your model. What's happening is that your initial sick dog dies before actually infecting another dog. This is why removing the if (days_infected = end-life) [die] "fixes" the problem.
When I tried your model with a huge population (e.g., 5000 people) so that encounters are more frequent, the infection does spread. You could also increase the probability of infection, or increase the duration of the "furious" phase, I guess.
Another unrelated suggestion, if I may: you should have distinct persons and dogs breeds. Trying to cram everything inside regular turtles makes your code much more complicated than it should be. The way I would approach this would be to create a link from the person to her dog, and then use tie so that the dog is automatically moved when you move the person.
Edit:
OK, here is a version of your code slightly modified to use breeds:
globals [
total_dogs_infected
total_dogs
dead_humans
dead_dogs
]
breed [ persons person ]
persons-own [
sick?
]
breed [ dogs dog ]
dogs-own [
sick?
vaccinated?
rabies_phase
days_infected
end-incubator
end-furious
end-life
]
to setup
clear-all
initialize-globals
setup-turtles
reset-ticks
end
to initialize-globals
set dead_humans 0
set dead_dogs 0
set total_dogs_infected 0
end
to setup-turtles
set-default-shape persons "person"
set-default-shape dogs "wolf"
create-persons people [
setxy random-xcor random-ycor
set size 1.5
set sick? false
ifelse random 100 < 43 [
set color green
hatch-dogs 1 [
set color brown
set heading 115 fd 1
create-link-from myself [ tie ]
set days_infected 0
set vaccinated? (random 100 > %_not_vaccinated)
if not vaccinated? [ set color orange ]
]
]
[
set color blue ;umano sano senza cane
]
]
set total_dogs count dogs
ask one-of dogs [ get_sick ]
end
to get_sick
set sick? true
set color white
set rabies_phase 1
set end-incubator 14 + random 57
set end-furious (end-incubator + random 5)
set end-life (end-furious + 4 + random 2)
set total_dogs_infected total_dogs_infected + 1
end
to go
move
infect
get-older-sick-dog
tick
end
to move
ask persons [
rt random 180
lt random 180
fd 1
]
end
to infect
ask dogs with [ sick? ] [
if (rabies_phase = 1 and (random 100) <= 2) or rabies_phase = 2 [
ask other dogs-here with [ not sick? and not vaccinated? ] [ get_sick ]
]
]
end
to get-older-sick-dog
ask dogs with [ sick? ] [
set days_infected days_infected + 1
;the incubator phase ends after at least 14 days + random(57) and then we have phase 2 (furious)
if (days_infected = end-incubator) [ set rabies_phase 2 ]
;when the main furious phase finishes we have 75% of probability that a secondary furious phase continues for other 4 - 6 days until death ;or we have a probability of 25% that the disease end in paralysis with a fast death
if (days_infected = end-furious and (random 100 > 75)) [
set dead_dogs dead_dogs + 1
die
]
if (days_infected = end-life) [
die
]
]
end
; These last reporters are not used,
; they just illustrate how to get the
; dog from the owner or vice-versa:
to-report my-dog ; person reporter
report one-of out-link-neighbors
end
to-report has-dog? ; person reporter
report any? out-link-neighbors
end
to-report my-owner ; dog reporter
report one-of in-link-neighbors
end
Not only does it simplify some expressions (e.g., ask dogs with [ sick? ] instead of ask turtles with [ has_dog? and sick_dog? ]), it opens up all sorts of possibilities: a dog could run away from its owner, the owner could die without the dog dying, an owner could have two dogs, etc.