As the question states I am looking for a good explanation/example for reinforcement learning in pybrain as the documentation on this confuses me no end, I can get it to work but I don't understand how to apply it to other things.
Thanks
Tom
Unfortunately, pybrain's documentation for rl classes is disappointing. I have found this blog quite useful.
In summary, you need to identify the following components (for the implementation details follow the tutorial on the link):
an environment: env = Environment(...)
a task --> task = Task(env)
a controller, which is a module (like a table) to keep your action-value information --> controller = Module(...)
a learner --> learner = SARSA() --> you may also add an Explorer to the learner. The default is epsilon-greedy with epsilon = 0.3, decay = 0.9999.
an agent to integrate controller and learner --> agent = Agent(controller, learner)
An experiment to integrate the task and the agent and do actual iterations --> experiment = Experiment(task, agent)
Each of the capitalized classes should be replaced with corresponding class from PyBrain.Then you simply run a do-while cycle to perform the iterations and learn. Note that there are several options to be set by the user, and in real-world problems you most likely need to write sub-classes to generalize the basic classes of pybrain, but the steps will be the same as here.
Related
I'm new to Autosar.
I was asked to implement Autosar interface for hand written code, create RTE and ARXML files, and send Arxml file to other group.
I have two runnables in my code.
Let's say for example interface variables (inputs) X and Y, and we have output as Z:
here my code:
global X;
global Y;
global Z;
Runnable_Step()
{
Z = X + Y;
}
Runnable_Init()
{
X =0 ;
Y =0;
Z =0;
}
I guess, my question where and how to start?
we have Autosar DaVinci Developer and Autosar DaVinci configurator.
Your question is quite general in nature and can not be answered in a few lines. However, I try to give a general answer which may help other people to orient themselves in AUTOSAR development with Vector DaVinci.
Generally in DaVinci you have the following work split between tools:
DaVinci Developer is used for modeling Application Component(s), Interfaces, Data Types etc. You also can run the RTE generator for generating "Contract Phase" artifacts.
DaVinci Configurator is used for integrating/configuring the ECU (Electronic Controll Unit). In an Application Developer's scenario it is typically used for generating the RTE that will actually be built into the final image.
In practice, the transition between these two tools often is fluid: You enter some information in Configurator (for example signals in the CAN Stack), then switch over to Developer for mapping the signals to your component instance's ports, then switch back to Configurator in order to configure and generate the production RTE.
You should start with thinking about how the AUTOSAR model for your task will look like. This includes the following aspects:
Which Data Types will be used?
Which Interfaces will be used (SenderReceiverInterface, ClientServerInterface)
Which Software Components do you need? Add the required Port Prototypes, using the Port Interfaces you defined in the previous step
Which resources does you SW Component need from the RTE? You add these to the "Internal Behavior" of the Component. In particular, these are Runnable Entities, but there are also Read/Write dependencies to the ports that need to be considered.
Integrate your new Software Component into a top-level composition.
Creation/Editing of these elements is done in Developer. Based on the work done so far you should be able to generate the "contract phase" RTE header files and C-language implementation stubs. You then can proceed to implement your runnable entities by adding code to the provided stubs.
Next, you will probably need to switch over to Configurator in order to configure the ECU your application should run in. If your work is based on an existing ECU, you will just add to the existing model. If you do not have an ECU project yet, you need to create one and typically populate it with the communication aspects of the model. This can be done either by importing a communciation extract or a CANdb file (Vector's proprietary CAN configuration format).
Switching back to Developer, you may deploy your Software Component onto the ECU, and also map its ports to the signals known by the ECU.
Finally, return to Configurator, optionally configure Basic Software and generate the RTE. It may be a lengthy process to get all BSW modules and parameters right in order to pass the RTE generator's validation process.
Once you managed to generate the RTE, you start the build process, and off you go!
Be sure to consult the AUTOSAR documents at https://www.autosar.org/
This is my sample network and idea of trying how to make a new routing instead of just the shortest path (the path i want to follow is via the pink arrows)
What am I missing here to make my predefined function work?
Fixing the basics
As explained here, you can't instantiate a Java List, because it is an interface. You can however instantiate any implementing Class of a List, for example an ArrayList.
With this in mind your code will look like this:
List<Path> myPath = new ArrayList<Path>();
myPath.add(path14);
myPath.add(path8);
myPath.add(path);
myPath.add(path1);
myPath.add(path4);
myPath.add(path13);
return myPath;
So far for the basics.
Where to go from here
To get it to consider your actual source and destination for the route planning, define both as input parameters of type ILocation in the properties of the function.
Now comes the really tricky part: writing your own or importing a routing algorithm that can give you that list of paths automatically based on criteria that you define. This is however a topic too broad for this question. The basic steps will be:
Create a graph that represents your AnyLogic path network
Solve the graph routing problem with a solving algorithm (eg. Dijkstra Algorithm), using the graph, the startpoint and the endpoint
Convert the solution you get from the solver back again to an ArrayList that you can work with in AnyLogic
You can do these steps on your own, eg. by implementing the Dijkstra Algorithm yourself, or you import into AnyLogic one of the available graph solving Java packages like JUNG or Graphhopper. In this article I explain step by step how to do so with JUNG.
In the implementation of a flow models that function with Modelica Standard Library DynamicPipe (or a similar model that builds from PartialTwoPortFlow) there are examples of flow models that take place in an environment with heat transfer that requires wall properties (e.g., heatPorts.T and/or heatPorts.Q_flow) in order to calculate the pressure drop.
For example, a pressure drop model may need to calculate a new visocisty or Prandtl number based on the medium pressure and the wall temperature to capture cooling/heating effects, etc.
The heat transfer model obtains properties of the medium via passing the "states" however there is no existing connection in DynamicPipe or PartialTwoPortFlow that goes the other way.
I've tried numerous variations of ideas and have had no success, including creating a new PartialTwoPortFlow that contains all the heat transfer calls that exist in DynamicPipe.
I hesitate to post this question as I am surprised I am having so much difficulty with this and would not be surprised to find a straight forward solution. Nevertheless I need this ability and curious if others have already solved this issue as I am running short on ideas.
So my questions is:
What is a proper/efficient means of passing the heatPorts.T values to the flowModel?
For those familiar with the MSL Fluids library and more specifically the Pipe models provided, this answer should (hopefully) make sense.
Aside:
It seems the dynamic pipe could be improved a little bit by not restricting the heat transfer area to the perimeter x lengths and instead introduce a parameter (e.g., heatTransferArea) that would permit the user to define it and default to perimeter x lengths. See below
parameter SI.Area heatTransferArea = perimeter*lengths "Total heat transfer area";
HeatTransfer heatTransfer(
...
final surfaceAreas=heatTransferArea , //perimeter*lengths <- replaced
...
End Aside:
In order to communicate heatPorts.T to the flowModel and to not have errors when I checked each of the models I had to do the following:
Make an "input" in the flowModel for Ts_w. Not parameter (take a look at how mediums.state is passed)! Might have to do some finagling with it like "diameters" (see DetailedPipeFlow) to make it be used how you think it's going to be used.
Duplicate PartialTwoPortFlow and add the final Ts_w = Ts_wFM to flowModel. Additionally define the variable SI.Temperature[nFM+1] Ts_wFM in PartialTwoPortFlow and establish definitions similar to statesFM in the equation section.
This will require adding a HeatPorts model to be added.
Duplicate DynamicPipe and change the extension to the new PartialTwoPortFlow. Set use_HeatTransfer to true (as I've set it up this has to be true now for this to work which isn't ideal but manageable). Might be good to make it a final parameter so it can't be changed.
Don't forget to connect heatPorts to the heatports added in step 2.
I believe that this capture a quick version of how I was able to get the wall temperature passed to the flowModel. Perhaps there is a more elegant way but I though this was pretty serviceable. I now simply have one more Partial model and one more pipe model called PartialTwoPort_wTemp and GenericDynamicPipe (I also incorporated my surfaceArea correction in the new pipe).
I am trying to run Omnet++ and matlab software in parallel and want them to communicate. When Omnet++ is running, I want to update the position of the node and for that I want to edit the .ned and .int files with matlab results continuously. During simulation I want to generate the result file using the updated files. I want just to update the position and don't want to add or delete any node. Please suggest me a way for proceeding?
matlab_loop
{
matlab_writes_position_in_ned_file;
delay(100ms);
}
omnet_loop
{
omnet_loads_ned_and_simulates;
//sca and vec should update;
delay(100ms);
}
Thank you.
NED and Ini files are read only during initialization of the model. You can't "read" them again after the simulation started. On the other hand, you are free to modify your parameters and create/delete modules using OMNeT++'s C++ API. What you want to achieve is basicaly: set your node position based on some calculations carried out by matlab code. The proper way to do it:
Generate C code from your matlab code.
Link that code to your OMNeT++ model
Create a new mobility model (assuming you are using INET) that is using the matlab code
What you are looking for seems to be more of a project rather than a question/problem which can be solved in Q&A site like stackoverflow.
Unfortunately, I have little understanding of matlab and V-REP to provide you a satisfactory answer. However, it seems that you will need to play around with APIs in lower levels.
As an example of coupling different simulation tools to form a simulation framework in case of need consider reading this paper and this
Also note the answer given by #Rudi. He seems to know what he is talking about.
need some help on this.
Currently I am doing a project on computer vision that requires me to train a new model to detect a certain object.
In this case, I am using the system provided by P. Felzenszwalb, D. McAllester, D. Ramaman and his team => Discriminatively trained deformable part models which is implemented in Matlab.
Project webpage: http://www.cs.uchicago.edu/~pff/latent/.
However I have no idea how to direct the system to use my dataset(a collection of images and annotation) which is different from the the PASCAL datasets so as to train a new model.
By directing, I meant a line of code that allows me to change the dataset the system reads from, for training a model.
E.g.
% directory for caching models, intermediate data, and results
cachedir = ['/var/tmp/rbg/YOURPATH/' VOCyear '/'];
I tried looking at their Readme and documentation guides but they do not make any mention. Do correct me if I am wrong.
Let me know if I have not made my problem clear enough.
I tried looking at some files such as global.m but no go.
Your help is much appreciated and thanks in advance!
You can try to read pascal.m in the DPM package(voc-release5), there are similar code working on VOC2007/2010 dataset.
There are plenty of parts that need to be adapted to achieve this. For example the voc_config has to be adapted in order to read from your files.
The same with the pascal_train.m function. Depending on the images and the way you parse them, this may require quite some time to adapt this function.
Other functions to consider:
imreadx
pascal_test
pascaleval