Reinforcement learning in Netlogo: Error: No urn specified - netlogo

I'm totally new to NetLogo, and am trying to create an agent-based reinforcement learning (RL) model. I have recreated a toy model to get help on.
Here, one agent is doing RL by interacting with two agents, one of whom is unreliable (and so interactions with this agent ARE NOT reinforced), and another which is reliable (and so interactions with this agent ARE reinforced).
But I keep getting the error: Extension exception: No urn specified
So, clearly, I need to figure out how to specify urns!
The project folder is available for you to checkout at: http://bit.ly/1m8TAxq
Bellow is the minimal, complete, verify code is as follows (though you will need the URN extension in the linked dropbox folder above to do so).
Any help would be tremendously appreciated.
extensions [ urn ] ;; the urn extension that contains the RL functions
breed [ agents agent ]
globals [ ;; Global variables
A1 ;; urn-ball name of agent 1 (RED)
A2 ;; urn-ball name of agent 2 (GREEN)
]
agents-own [ ;; Values assigned to each agent
main-urn ;; Name of the Polya urn used in RL
interactionPartner ;; Variable tracking who the learner is interacting with
knowledgeState ;; Variable tracking whether the learner knows the truth after each interaction
]
to setup ;; Setup the stage
clear-all
reset-ticks
setup-agents ;; Sets up the agents
end
to have-agents-setup-urns ;; Setting up agent polya urns for reinforcement learning (by observer)
ask agent 0 [
set main-urn urn:make-polya-urn initial-weight (list A1 A2)
;; There are two types of balls in the urn: A1, A2 corresponding to the two possible agents with whom the chooser may interact
]
end
to setup-agents ;; Setting up the agents in the population (by observer)
create-agents 3 [
setxy (-6 * cos ( who * 360 / population-size)) (12 * sin (who * 360 / population-size))
set shape "circle"
set size 4
]
ask agent 0 [ ;; This is the agent doing the reinforcement learning (BLUE)
set color [0 0 255 175]
]
ask agent 1 [ ;; This is the unreliable agent (RED)
set knowledgeState 0
set color [255 0 0 175]
]
ask agent 2 [ ;; This is the reliable agent (GREEN)
set knowledgeState 1
set color [0 255 0 175]
]
end
to step ;; The step function
ask turtles [do-game-play]
tick
end
to go ;; The go function simply calls step continuously
step
end
to do-game-play ;; The game play function (by observer)
ask agent 0 [
set interactionPartner urn:draw-ball main-urn ;; Draw from the agent urn to figure out with whom to interact.
if (interactionPartner = A1) [
set knowledgeState ( 0 ) ;; If the interactionPartner is agent 1 (who is unreliable), then set agent 0 knowledge state to 0.
]
if (interactionPartner = A2) [
set knowledgeState ( 1 ) ;; If the interactionPartner is agent 2 (who is reliable), then set agent 0 knowledge state to 1, and reinforce.
urn:reinforce main-urn A2 1
]
]
end

Related

To do a netlogo model with several patch layers (think floors of a building) do I need to go to Netlogo 3D?

I have a netlogo application in mind which involves multiple non interacting layers. Think floors of a building. Would I need to go to netlogo 3D or is there a suggested way to handle in regular netlogo?
This was an interesting enough question that I decided to make a simple sample.
The method I use is to have a global variable to track the number-of-floors we'll have in our building, as well as the active-floor that we are currently working with. Then all our agents, walls and workers, have a floor-num that tracks which they are on. We have a set-active-floor procedure that handles switching our currently active floor that we want to see and work with, making the patches a certain color (active-color) if they have a wall agent present and swapping which workers are hidden?. In this way, most of the magic happens in the setup-floor and set-active-floor procedures, and the real work of our model in go can be pretty typical NetLogo code asking the active-workers to do whatever we want.
While the model is running you can call set-active-floor n to any value 0 to 4 to change the current workers and walls. I also included a show-all procedure that'll un-hide all the walls and workers and let you see where they are at; you'll need to run that with the model stopped.
globals [colors active-color number-of-floors active-floor]
turtles-own [floor-num]
breed [walls wall]
breed [workers worker]
to setup
clear-all
set colors (list red blue green orange violet)
set number-of-floors 5
foreach (range 0 number-of-floors) setup-floor
reset-ticks
set-active-floor 0
end
to setup-floor [num]
set active-floor num
set active-color (item num colors)
ask patches [ set pcolor black ]
; make some random walls
create-walls (count patches * 0.2) [
set floor-num num
set hidden? true
set size 0.5
setxy random-pxcor random-pycor
set pcolor active-color
]
; only one wall per patch
ask patches [
let patch-walls floor-walls-here
let wall-count count patch-walls
if (wall-count > 1) [
ask n-of (wall-count - 1) patch-walls [ die ]
]
]
; make some workes
create-workers 10 [
set floor-num num
set hidden? true
set shape "person"
set color active-color - 2
move-to one-of patches with [pcolor != active-color]
]
end
to go
; this only "runs" the active floor, but you could run all of them
; using the same `foreach (range 0 number-of-floors) ...` code as
; in the `setup` procedure.
set-active-floor active-floor
ask active-workers [
; this code can be about the same as you'd write in a normal model...
move-to one-of patches with [pcolor != active-color]
]
tick
end
to set-active-floor [num]
set active-floor num
set active-color (item num colors)
; after the `pcolor` is set, we can use that to determine if a wall
; exists or not for an `active-worker`, we don't have to check the
; floor number at all while we do our work.
ask walls [ set hidden? true ]
ask patches [
set pcolor ifelse-value no-walls-here [ black ] [ active-color ]
]
ask workers [
set hidden? floor-num != active-floor
]
end
to-report active-workers
report workers with [floor-num = active-floor]
end
to-report floor-walls-here
report walls-here with [floor-num = active-floor]
end
to-report no-walls-here
report (count floor-walls-here = 0)
end
to show-all
foreach (range 0 number-of-floors) [ num ->
ask patches [
set pcolor ifelse-value any? walls-here [ red - 3 ] [ black ]
]
ask turtles with [floor-num = num] [
set color item num colors
set hidden? false
]
]
end
Finally, if things got much more complicated than this, I would probably choose to move to NetLogo 3D instead.

NetLogo: link an agent to another, ask things to the agent and other things to the linked agent

I'm working on extending NetLogo Robotic Factory with a mechanism where there can be only one robot per machine and where the machine takes several ticks to perform the task.
To model this, I'm creating a link between the robot and the machine. And there can only be one link per machine.
It is the first time I'm using NetLogo. So I'm not used to switching contexts. I'm trying to find out how I can give orders to both the robot and the machine when they are linked.
to move
ask robots [
if waiting = 0 [
if can-move? 1 [forward 1 face destination]
set energy energy - 1
]
if waiting > 0 [
set waiting waiting - 1
]
]
end
to set-robots-destination
ask robots [
(ifelse
; ---------------
; Step1: forges
; ---------------
[ breed ] of destination = forges [
ifelse any? forges-here [
if [count links] of one-of forges-here = 0 [
create-link-with one-of forges-here
set color of one-of forges-here red
set waiting 5
if waiting = 0 [
set costs costs + 1
set destination one-of forges
set [color] of one-of materials-here blue
ask my-links [ die ]
]
]
]
[ move ]
]
...
As you can see, repeating one-of forge there create several issues. First of all the syntax to change its color is incorrect. Then, it is unclear if the color will actually turn blue by the end or if the robot will already have left.
I would prefer to find a way to switch context with the linked forge and give it orders directly. Is this possible?

NetLogo: Issues when restricting number of users charging at a certain patch

I am working on a project to model the impact of charging electric cars on the grid and modeling/simulating the driving and charging habits of the car users. I'm getting an issue in my code that unable to resolve yet.
Each location has a limited number of charging ports. For example, WORK has a total of 2 TERMINALS, so only 2 adopters can charge there simultaneously (first-come-first-serve basis). What I want to do is when 2 adopters arrive at WORK, they start charging (if required, i.e. "charging-status" = true). Any additional adopters wait until a port is available there. The adopters who finish charging should vacate the charging port for those in the wait-list, even if they don't leave.
Here's part of my effort (code) that I did:
to go
...
charge-car ; sets the charging-status based on state-of-charge.
ask adopters
[
if charging? and not marked?
[
ifelse remaining-ports != 0
[
set remaining-ports max list (remaining-ports - 1) 0
set marked? true
]
[set occupied? true]
]
if marked? and not charging?
[
set remaining-ports min list (remaining-ports + 1) terminals
set marked? false
set occupied? false
]
]
ask adopters with [charging? and marked?]
[
set color green
let battery0 battery
let charging-speed0 charging-speed
let battery1 max list 0 ( battery + charging-speed0 )
set battery min list battery1 battery-capacity
let charged min list ( battery - battery0 ) charging-speed0
set charge-demand charge-demand + charged
set soc battery / battery-capacity
set range-left battery / discharge-patch
]
tick
end
Now, the issue is this: there are multiple location on the map with charging ports. This code gives different results at some locations, even though it is the same algorithm for all locations and agents. For example, if both ports are occupied at certain locations, the "occupied?" will be true for some locations and not all of the ones with all ports engaged. I mean to say, this is showing quite a random response.
Can anyone please help me with this? Is there another way to do what I want to do? Also, please let me know if you need more info to understand my situation.
Thank you!
Edit:
This is my code for to go
to go
...
ask adopters
[
if patch-here = current-loc ; choose next target only when reached at a destination (current location)
[
choose-target
set nearest-station min-one-of patches with [location = "charging-station"][distance myself]
] ; choose target based on start time and current location
; go to target only when NOT at the arbitrary target location
if target != [0 0]
[
let dist-to-targ distance-between current-loc target
let dist-to-station distance-between current-loc nearest-station
ifelse dist-to-targ > range-left and dist-to-station < range-left
[go-to-station nearest-station]
[go-to-target]
]
if charging = "Charge Car Now"
[charge-car]
...
]
where, charge-car is
to charge-car
if patch-here = current-loc and charging-point
[
ifelse soc < 1
[
if charge-power = 1
[
set charging-speed 1 / 12
set charging-status true
]
if charge-power = 2
[
set charging-speed 6.6 / 12
set charging-status true
]
]
[
set charging-status false
set color blue
]
]
end
and go-to-target is
to go-to-target
ifelse patch-here != target
[
; move towards destination and use fuel
face target
; set marked? false
set color blue
ifelse distance target <= speed
[set speed1 0.3 * distance target] ; decrease speed as target gets nearer
[set speed1 speed]
forward speed1
set moving? true
set charging-status false
if marked?
[
set rem-term min list (rem-term + 1) terminals
type patch-here type "Updated ports" print rem-term
set marked? false
set occupied? false
]
]
[
move-to target
if target != [0 0]
[set dist-trav distance-between current-loc target]
set current-loc target
set moving? false
set dwell dwell-acq day-ind time-ind position [location] of target places ; calculate dwell time based on arrival time at target
ifelse dwell < 0
[
set dwell 288 - (ticks mod 288) ; spend rest of the time till 24:00 at that location
set dwell-flag 1
]
[set dwell-flag 0]
if current-loc = target
[
set arrival-time (ticks mod 288)
set start-time (dwell + arrival-time) mod 288
set target [0 0]
set battery battery - (discharge-patch * dist-trav) ; discharge based on distance traveled per tick
set soc battery / battery-capacity
set range-left battery / discharge-patch
if battery < 0
[set battery 0]
if soc < 0
[set soc 0]
]
]
end
where, rem-term is same as remaining-ports and charging-status is same as charging?.
I tried adding the same code in the go-to-target function, since charging-status changes there first, but that didn't show any change in the results I'm getting.
I can't see anything obviously wrong with your code. This sort of thing usually happens because you have multiple ask turtles blocks, and you work out the intention in the first block but don't do the behaviour until the second block. In your case, I can see you updating the ports count in the first block, so that doesn't directly apply.
However, I wonder if you're doing something similar with your if statements, that turtles are going through different blocks than you expect and the relevant code is missing from the extract that you pulled out. The easiest way to diagnose this type of problem is with print statements. See below for one possibility.
ask adopters
[ if charging? and not marked?
[ ifelse remaining-ports > 0
[ type patch-here print remaining-ports
set remaining-ports remaining-ports - 1
set marked? true
type patch-here type "Updated ports" print remaining-ports
]
[ set occupied? true ]
]
if marked? and not charging?
[ set remaining-ports min list (remaining-ports + 1) terminals
set marked? false
set occupied? false
]
]
Note that I also changed your code for testing/updating number of remaining ports for clarity.
On your question about lists, there is no problem adding a turtle to a list (eg set queue lput self queue) but if you want more detail than that, please ask a separate question. I strongly recommend that you do not make any attempt to introduce queues for your ports until you have the existing code working properly.

Counting turtles after a chunk of code has executed

In my model currently, I have a monitor on the Interface that counts the total number of turtles (deer in my case) every tick. Is it possible to have another monitor that counts the number of deer after a certain line of code is executed? For example, here's a snippet of code:
to catch-fawns-source
let fawn-hunting-rate (fawn-harvest-rate-source)
if any? fawns-on source-patches
[ ask fawns-on source-patches [
if random-float 1.0001 < (fawn-hunting-rate)
[ set harvest (harvest + 1)
set fawn-harvest (fawn-harvest + 1)
set source-harvest (source-harvest + 1)
die ] ]
]
end
In this case, this is where I have fawns being harvested. This code is the same for my male and female adult and juvenile deer. So is there a way I could track my population specifically after that snippet of code (and the other identical ones for the juveniles and adults) is executed?
I'm not sure if I would need any sort of Netlogo extension (if there are any applicable to this) or if I could somehow add in another global variable and lines of code that accomplishes this task.
Thanks for all of your help as always!
I think you can get away with another global variable that you just update however often you want. For a very simple example, consider this setup:
globals [ most-recent-pop-check ]
to setup
ca
crt 10
set most-recent-pop-check count turtles
reset-ticks
end
Here, most-recent-pop-check will only be updated when needed, then you can just set a monitor to display that variable. In this example, the value will only change every 25 ticks- see comments for more detail:
to go
ask turtles [
; Turtles may die
if random-float 1 < 0.25 [
die
]
; Throw in some density-dependence to control population size
if random-float 1 < ( 0.5 * ( 1 - ( count turtles / 500 ) ) ) [
hatch random 2 + 1
]
]
; If the ticks are not 0, and if the remainder after dividing
; the ticks by 0 is 0, update the most-recent-pop-check
; variable to reflect the current turtle population.
if ticks != 0 and ticks mod 25 = 0 [
set most-recent-pop-check count turtles
]
tick
end
Of course, in your case rather than having the update occur based on the number of ticks, just have it occur whenever you run the code chunk you're after.

Running a model several times with BehaviorSpace

From BehaviorSpace, I would like to run a model 100 times by varying two variables as follows:
["x-area" 1 ] and ["y-area" [1 1 100]].
A model is built as follows:
create landscape
add-turtles
go-simulation
By using the two variables, I would like that my 100 models run as follows:
;;; Model 1 ;;
"x-area" = 1
"y-area" = 1
clear-all
create landscape
add-turtles
go-simulation
;;; Model 2 ;;
"x-area" = 1
"y-area" = 2
add-turtles
go-simulation
;;; Model 3 ;;
"x-area" = 1
"y-area" = 3
add-turtles
go-simulation
....
;;; Model 100 ;;
"x-area" = 1
"y-area" = 100
add-turtles
go-simulation
To do this, I built 100 experiments and this method worked. Is there a faster way to run automatically 100 models without doing 100 experiments ? I tried to build 1 experiment like this
But I have this error message:
OF expected input to be a turtle agentset or patch agentset or turtle or patch but got NOBODY instead.
org.nlogo.nvm.ArgumentTypeException: OF expected input to be a turtle agentset or patch agentset or turtle or patch but got NOBODY instead.
at org.nlogo.prim._asm_proceduremovewithinpolygon_ifelse_86.perform(:4)
at org.nlogo.nvm.Context.runExclusive(Context.java:119)
at org.nlogo.nvm.ExclusiveJob.run(ExclusiveJob.java:57)
at org.nlogo.nvm.Context.runExclusiveJob(Context.java:162)
at org.nlogo.prim._asm_procedurestartsimulation_ask_69.perform(:1)
at org.nlogo.nvm.Context.stepConcurrent(Context.java:91)
at org.nlogo.nvm.ConcurrentJob.step(ConcurrentJob.java:82)
at org.nlogo.job.JobThread.org$nlogo$job$JobThread$$runPrimaryJobs(JobThread.scala:143)
at org.nlogo.job.JobThread$$anonfun$run$1.apply$mcV$sp(JobThread.scala:78)
at org.nlogo.job.JobThread$$anonfun$run$1.apply(JobThread.scala:76)
at org.nlogo.job.JobThread$$anonfun$run$1.apply(JobThread.scala:76)
at scala.util.control.Exception$Catch.apply(Exception.scala:88)
at org.nlogo.util.Exceptions$.handling(Exceptions.scala:41)
at org.nlogo.job.JobThread.run(JobThread.scala:75)
The problem is that my models continue to run with this error. So it is difficult to see where is the problem. Given the following message:
"at org.nlogo.prim._asm_proceduremovewithinpolygon_ifelse_86.perform(:4)" in the error message,
maybe that the problem is in the procedure "move-within-polygon".
Here is my procedure "move-within-polygon" for a given color of polygons:
if [pcolor] of patch-here = green [
set list-angles-in-green item 0 table-angles
loop [
let angle-in-green one-of list-angles-in-green
ifelse [pxcor] of (patch-right-and-ahead angle-in-green 1) = max-pxcor or [pycor] of (patch-right-and-ahead angle-in-green 1) = max-pycor [
print "die"
die
stop ]
[ ifelse (patch-right-and-ahead angle-in-green 1) != nobody and [pcolor] of (patch-right-and-ahead angle-in-green 1) = green [
print "move"
move-to patch-right-and-ahead angle-in-green 1
[ if not any? neighbors with [pcolor = green] [
print "no neighbors"
move-to patch-here
stop ] ] ] ] ]
Thanks for your help.
If a turtle ends up on the edge of the world, then patch-right-and-ahead angle-in-green 1 might point outside the world (depending on what angle-in-green is), so [pxcor] of in [pxcor] of (patch-right-and-ahead angle-in-green 1) = max-pxcor would ask for the coordinate of nobody. The same thing could happen for pycor later in the same line.
Question: Can a turtle ever get to the edge of the world in your model? It looks to me like the code that you displayed could lead to that result.
If so, then one way to prevent the error would be to replace
[pxcor] of (patch-right-and-ahead angle-in-green 1) = max-pxcor
with
xcor = max-pxcor
That's true when a turtle reaches the last column of patches. If you also want turtles to die when they're close to the edge, you could use both of these expressions in the if-else:
(xcor = max-pxcor) or ([pxcor] of (patch-right-and-ahead angle-in-green 1) = max-pxcor)
(I added parentheses simply for clarity.)
However, I wonder whether this would serve the same purpose:
xcor = max-pxcor or xcor = max-pxcor - 1
If any of these methods are right for your application, then you can obviously do the same thing for the y coordinates.
Your experiment setup appears correct to me, except that you should remove the "Stop condition" of TRUE, because if the stop condition is always true, your runs will never run the go commands even once.
The error you're getting is coming from code that you haven't shown us, so I can't help you there. You'll need to show us the code in which the error is occurring.
Also, at the time the error occurs, what are the values of x-area and y-area? And does the same error occur if you set x-area and y-area to the same values outside BehaviorSpace? If so, then the error doesn't really have anything to do with BehaviorSpace.
Finally, a note on terminology: there is only one model here, not 100, and only one experiment here, not 100. You're attempting to run one experiment on your model, and that experiment consists of 100 model runs. Using the standard terminology will help you communicate your issue clearly.