Rheological models are usually build using three (or four) basics elements, which are :
The spring (existing in Modelica.Mechanics.Translational.Components for example). Its equation is f = c * (s_rel - s_rel0);
The damper (dashpot) (also existing in Modelica.Mechanics.Translational.Components). Its equation is f = d * v_rel; for a linear damper, an could be easily modified to model a non-linear damper : f = d * v_rel^(1/n);
The slider, not existing (as far as I know) in this library... It's equation is abs(f)<= flim. Unfortunately, I don't really understand how I could write the corresponding Modelica model...
I think this model should extend Modelica.Mechanics.Translational.Interfaces.PartialCompliant, but the problem is that f (the force measured between flange_b and flange_a) should be modified only when it's greater than flim...
If the slider extends PartialCompliant, it means that it already follows the equations flange_b.f = f; and flange_a.f = -f;
Adding the equation f = if abs(f)>flim then sign(f)*flim else f; gives me an error "An independent subset of the model has imbalanced number of equations and variables", which I couldn't really explain, even if I understand that if abs(f)<=flim, the equation f = f is useless...
Actually, the slider element doesn't generate a new force (just like the spring does, depending on its strain, or just like the damper does, depending on its strain rate). The force is an input for the slider element, which is sometime modified (when this force becomes greater than the limit allowed by the element). That's why I don't really understand if I should define this force as an input or an output....
If you have any suggestion, I would greatly appreciate it ! Thanks
After the first two comments, I decided to add a picture that, I hope, will help you to understand the behaviour I'm trying to model.
On the left, you can see the four elements used to develop rheological models :
a : the spring
b : the linear damper (dashpot)
c : the non-linear damper
d : the slider
On the right, you can see the behaviour I'm trying to reproduce : a and b are two associations with springs and c and d are respectively the expected stress / strain curves. I'm trying to model the same behaviour, except that I'm thinking in terms of force and not stress. As i said in the comment to Marco's answer, the curve a reminds me the behaviour of a diode :
if the force applied to the component is less than the sliding limit, there is no relative displacement between the two flanges
if the force becomes greater than the sliding limit, the force transmitted by the system equals the limit and there is relative displacement between flanges
I can't be sure, but I suspect what you are really trying to model here is Coulomb friction (i.e. a constant force that always opposes the direction of motion). If so, there is already a component in the Modelica Standard Library, called MassWithStopAndFriction, that models that (and several other flavors of friction). The wrinkle is that it is bundled with inertia.
If you don't want the inertia effect it might be possible to set the inertia to zero. I suspect that could cause a singularity. One way you might be able to avoid the singularity is to "evaluate" the parameter (at least that is what it is called in Dymola when you set the Evaluate flat to be true in the command line). No promises whether that will work since it is model and tool dependent whether such a simplification can be properly handled.
If Coulomb friction is what you want and you really don't want inertia and the approach above doesn't work, let me know and I think I can create a simple model that will work (so long as you don't have inertia).
A few considerations:
- The force is not an input and neither an output, but it is just a relation that you add into the component in order to define how the force will be propagated between the two translational flanges of the component. When you deal with acausal connectors I think it is better to think about the degrees of freedom of your component instead of inputs and outputs. In this case you have two connectors and independently at which one of the two frames you will recieve informations about the force, the equation you implement will define how that information will be propagated to the other frame.
- I tested this:
model slider
extends
Modelica.Mechanics.Translational.Interfaces.PartialCompliantWithRelativeStates;
parameter Real flim = 1;
equation
f = if abs(f)>flim then sign(f)*flim else f;
end slider;
on Dymola and it works. It is correct modelica code so it should be work also in OpenModelica, I can't think of a reason why it should be seen as an unbalance mathematical model.
I hope this helps,
Marco
Related
I'm new to reinforcement learning at all, so I may be wrong.
My questions are:
Is the Q-Learning equation ( Q(s, a) = r + y * max(Q(s', a')) ) used in DQN only for computing a loss function?
Is the equation recurrent? Assume I use DQN for, say, playing Atari Breakout, the number of possible states is very large (assuming the state is single game's frame), so it's not efficient to create a matrix of all the Q-Values. The equation should update the Q-Value of given [state, action] pair, so what will it do in case of DQN? Will it call itself recursively? If it will, the quation can't be calculated, because the recurrention won't ever stop.
I've already tried to find what I want and I've seen many tutorials, but almost everyone doesn't show the background, just implements it using Python library like Keras.
Thanks in advance and I apologise if something sounds dumb, I just don't get that.
Is the Q-Learning equation ( Q(s, a) = r + y * max(Q(s', a')) ) used in DQN only for computing a loss function?
Yes, generally that equation is only used to define our losses. More specifically, it is rearranged a bit; that equation is what we expect to hold, but it generally does not yet precisely hold during training. We subtract the right-hand side from the left-hand side to compute a (temporal-difference) error, and that error is used in the loss function.
Is the equation recurrent? Assume I use DQN for, say, playing Atari Breakout, the number of possible states is very large (assuming the state is single game's frame), so it's not efficient to create a matrix of all the Q-Values. The equation should update the Q-Value of given [state, action] pair, so what will it do in case of DQN? Will it call itself recursively? If it will, the quation can't be calculated, because the recurrention won't ever stop.
Indeed the space of state-action pairs is much too large to enumerate them all in a matrix/table. In other words, we can't use Tabular RL. This is precisely why we use a Neural Network in DQN though. You can view Q(s, a) as a function. In the tabular case, Q(s, a) is simply a function that uses s and a to index into a table/matrix of values.
In the case of DQN and other Deep RL approaches, we use a Neural Network to approximate such a "function". We use s (and potentially a, though not really in the case of DQN) to create features based on that state (and action). In the case of DQN and Atari games, we simply take a stack of raw images/pixels as features. These are then used as inputs for the Neural Network. At the other end of the NN, DQN provides Q-values as outputs. In the case of DQN, multiple outputs are provided; one for every action a. So, in conclusion, when you read Q(s, a) you should think "the output corresponding to a when we plug the features/images/pixels of s as inputs into our network".
Further question from comments:
I think I still don't get the idea... Let's say we did one iteration through the network with state S and we got following output [A = 0.8, B = 0.1, C = 0.1] (where A, B and C are possible actions). We also got a reward R = 1 and set the y (a.k.a. gamma) to 0.95 . Now, how can we put these variables into the loss function formula https://imgur.com/a/2wTj7Yn? I don't understand what's the prediction if the DQN outputs which action to take? Also, what's the target Q? Could you post the formula with placed variables, please?
First a small correction: DQN does not output which action to take. Given inputs (a state s), it provides one output value per action a, which can be interpreted as an estimate of the Q(s, a) value for the input state s and the action a corresponding to that particular output. These values are typically used afterwards to determine which action to take (for example by selecting the action corresponding to the maximum Q value), so in some sense the action can be derived from the outputs of DQN, but DQN does not directly provide actions to take as outputs.
Anyway, let's consider the example situation. The loss function from the image is:
loss = (r + gamma max_a' Q-hat(s', a') - Q(s, a))^2
Note that there's a small mistake in the image, it has the old state s in the Q-hat instead of the new state s'. s' in there is correct.
In this formula:
r is the observed reward
gamma is (typically) a constant value
Q(s, a) is one of the output values from our Neural Network that we get when we provide it with s as input. Specifically, it is the output value corresponding to the action a that we have executed. So, in your example, if we chose to execute action A in state s, we have Q(s, A) = 0.8.
s' is the state we happen to end up in after having executed action a in state s.
Q-hat(s', a') (which we compute once for every possible subsequent action a') is, again, one of the output values from our Neural Network. This time, it's a value we get when we provide s' as input (instead of s), and again it will be the output value corresponding to action a'.
The Q-hat instead of Q there is because, in DQN, we typically actually use two different Neural Networks. Q-values are computed using the same Neural Network that we also modify by training. Q-hat-values are computed using a different "Target Network". This Target Network is typically a "slower-moving" version of the first network. It is constructed by occasionally (e.g. once every 10K steps) copying the other Network, and leaving its weights frozen in between those copy operations.
Firstly, the Q function is used both in the loss function and for the policy. Actual output of your Q function and the 'ideal' one is used to calculate a loss. Taking the highest value of the output of the Q function for all possible actions in a state is your policy.
Secondly, no, it's not recurrent. The equation is actually slightly different to what you have posted (perhaps a mathematician can correct me on this). It is actually Q(s, a) := r + y * max(Q(s', a')). Note the colon before the equals sign. This is called the assignment operator and means that we update the left side of the equation so that it is equal to the right side once (not recurrently). You can think of it as being the same as the assignment operator in most programming languages (x = x + 1 doesn't cause any problems).
The Q values will propagate through the network as you keep performing updates anyway, but it can take a while.
I am trying to reduce dimensionality of a training set using PCA.
I have come across two approaches.
[V,U,eigen]=pca(train_x);
eigen_sum=0;
for lamda=1:length(eigen)
eigen_sum=eigen_sum+eigen(lamda,1);
if(eigen_sum/sum(eigen)>=0.90)
break;
end
end
train_x=train_x*V(:, 1:lamda);
Here, I simply use the eigenvalue matrix to reconstruct the training set with lower amount of features determined by principal components describing 90% of original set.
The alternate method that I found is almost exactly the same, save the last line, which changes to:
train_x=U(:,1:lamda);
In other words, we take the training set as the principal component representation of the original training set up to some feature lamda.
Both of these methods seem to yield similar results (out of sample test error), but there is difference, however minuscule it may be.
My question is, which one is the right method?
The answer depends on your data, and what you want to do.
Using your variable names. Generally speaking is easy to expect that the outputs of pca maintain
U = train_x * V
But this is only true if your data is normalized, specifically if you already removed the mean from each component. If not, then what one can expect is
U = train_x * V - mean(train_x * V)
And in that regard, weather you want to remove or maintain the mean of your data before processing it, depends on your application.
It's also worth noting that even if you remove the mean before processing, there might be some small difference, but it will be around floating point precision error
((train_x * V) - U) ./ U ~~ 1.0e-15
And this error can be safely ignored
first a little background. I'm a psychology student so my background in coding isn't on par with you guys :-)
My problem is as follow and the most important observation is that curve fitting with 2 different programs gives completly different results for my parameters, altough my graphs stay the same. The main program we have used to fit my longitudinal data is kaleidagraph and this should be seen as kinda the 'golden standard', the program I'm trying to modify is matlab.
I was trying to be smart and wrote some code (a lot at least for me) and the goal of that code was the following:
1. Taking an individual longitudinal datafile
2. curve fitting this data on a non-parametric model using lsqcurvefit
3. obtaining figures and the points where f' and f'' are zero
This all worked well (woohoo :-)) but when I started comparing the function parameters both programs generate there is a huge difference. The kaleidagraph program stays close to it's original starting values. Matlab wanders off and sometimes gets larger by a factor 1000. The graphs stay however more or less the same in both situations and both fit the data well. However it would be lovely if I would know how to make the matlab curve fitting more 'conservative' and more located near it's original starting values.
validFitPersons = true(nbValidPersons,1);
for i=1:nbValidPersons
personalData = data{validPersons(i),3};
personalData = personalData(personalData(:,1)>=minAge,:);
% Fit a specific model for all valid persons
try
opts = optimoptions(#lsqcurvefit, 'Algorithm', 'levenberg-marquardt');
[personalParams,personalRes,personalResidual] = lsqcurvefit(heightModel,initialValues,personalData(:,1),personalData(:,2),[],[],opts);
catch
x=1;
end
Above is a the part of the code i've written to fit the datafiles into a specific model.
Below is an example of a non-parametric model i use with its function parameters.
elseif strcmpi(model,'jpa2')
% y = a.*(1-1/(1+(b_1(t+e))^c_1+(b_2(t+e))^c_2+(b_3(t+e))^c_3))
heightModel = #(params,ages) abs(params(1).*(1-1./(1+(params(2).* (ages+params(8) )).^params(5) +(params(3).* (ages+params(8) )).^params(6) +(params(4) .*(ages+params(8) )).^params(7) )));
modelStrings = {'a','b1','b2','b3','c1','c2','c3','e'};
% Define initial values
if strcmpi('male',gender)
initialValues = [176.76 0.339 0.1199 0.0764 0.42287 2.818 18.52 0.4363];
else
initialValues = [161.92 0.4173 0.1354 0.090 0.540 2.87 14.281 0.3701];
end
I've tried to mimick the curve fitting process in kaleidagraph as good as possible. There I've found they use the levenberg-marquardt algorithm which I've selected. However results still vary and I don't have any more clues about how I can change this.
Some extra adjustments:
The idea for this code was the following:
I'm trying to compare different fitting models (they are designed for this purpose). So what I do is I have 5 models with different parameters and different starting values ( the second part of my code) and next I have the general curve fitting file. Since there are different models it would be interesting if I could put restrictions into how far my starting values could wander off.
Anyone any idea how this could be done?
Anybody willing to help a psychology student?
Cheers
This is a common issue when dealing with non-linear models.
If I were, you, I would try to check if you can remove some parameters from the model in order to simplify it.
If you really want to keep your solution not too far from the initial point, you can use upper bounds and lower bounds for each variable:
x = lsqcurvefit(fun,x0,xdata,ydata,lb,ub)
defines a set of lower and upper bounds on the design variables in x so that the solution is always in the range lb ≤ x ≤ ub.
Cheers
You state:
I'm trying to compare different fitting models (they are designed for
this purpose). So what I do is I have 5 models with different
parameters and different starting values ( the second part of my code)
and next I have the general curve fitting file.
You will presumably compare the statistics from fits with different models, to see whether reductions in the fitting error are unlikely to be due to chance. You may want to rely on that comparison to pick the model that not only fits your data suitably but is also simplest (which is often referred to as the principle of parsimony).
The problem is really with the model you have shown resulting in correlated parameters and therefore overfitting, as mentioned by #David. Again, this should be resolved when you compare different models and find that some do just as well (statistically speaking) even though they involve fewer parameters.
edit
To drive the point home regarding the problem with the choice of model, here are (1) results of a trial fit using simulated data (2) the correlation matrix of the parameters in graphical form:
Note that absolute values of the correlation close to 1 indicate strongly correlated parameters, which is highly undesirable. Note also that the trend in the data is practically linear over a long portion of the dataset, which implies that 2 parameters might suffice over that stretch, so using 8 parameters to describe it seems like overkill.
I'm trying to learn Modelica and I have constructed a very simple model using the multibody library. the model consists of a world object and a body (mass) connected to to beams which are then connected to 2 extended PartialOneFrame_a classes (see below) which I modified to create a constant force in one axis. essentially all this group of objects does is fall under gravity and spin around due to the two forces acting at a longituidnal offset from the body center creating a couple about the cg.
model Constant_Force
extends Modelica.Mechanics.MultiBody.Interfaces.PartialOneFrame_a;
parameter Real force = 1.0;
equation
frame_a.f = {0.0,0.0,force};
frame_a.t = {0.0,0.0,0.0};
end Constant_Force;
I next wanted see if I could create a very simple aerodynamic force component which I would connect to the end of one the rotating 'arms'. My idea was to follow the example of the Constant_force model above and for my first simple cut generate forces based on the local frame velocities. This where my problem arose - I tried to compute the velocity using der(frame_a.r_0) which I was then going to transform to local frame using resolve2 function but adding the der(...) line caused the model to not work properly - it would 'successfully' simulate (using OpenModelica) but the v11b vector (see below) would be all zeros, so would der(frame_a.r_0) that appeared for plot plotting - not only that, all the other component behaviors were now also just zero constantly - (frame_a.r_0, w_a etc of the body).
model Aerosurf
extends Modelica.Mechanics.MultiBody.Interfaces.PartialOneFrame_a;
import Modelica.Mechanics.MultiBody.Frames;
import Modelica.SIunits;
//Real[3] v11b;
SIunits.Velocity v11b[3];
//initial equation
// v11b={0.0,0.0,0.0};
algorithm
//v11b:=der(frame_a.r_0);
equation
v11b=der(frame_a.r_0);
frame_a.f = {0.0,0.0,0.0};
frame_a.t = {0.0,0.0,0.0};
end Aerosurf;
I tried a number of ways just to simply compute the velocities (you will see from the commented lines) so i could plot to check correct behavior but to no avail. I used algorithm or equation approach - I did acheive some different (but also incorrect behaviours) with the different approaches.
Any tips? I must be missing something fundamental here, it seems the frame component does not inherently carry the velocity vector so I must have to compute it??
The simplest way is to use the Modelica.Mechanics.MultiBody.Sensors.AbsoluteVelocity block from MSL and connect it to your MB frame, then just use the variable of the output connector in your equation.
Just starting with Modelica and having trouble understanding how it works.
In the below 'method' of the model, qInflow and qOutflow are used in the second line to evaluate der(h), but they have not received a value yet! (they were not defined in the 'data' of the method)? In what order is the code executed.
equation
assert(minV >= 0, "minV must be greater or equal to zero");
der(h)=(qInflow - qOutflow)/area;
qInflow=if time > 150 then 3*flowLevel else flowLevel;
qOutflow=Functions.LimitValue(minV, maxV, -flowGain*outCtr);
error=ref - h;
der(x)=error/T;
outCtr=K*(error + x);
end FlatTank;
From http://www.mathcore.com/resources/documents/ie_tank_system.pdf
This is an understandable point of confusion when coming from languages and systems that utilize imperative semantics. But Modelica doesn't work like that.
When working with Modelica it is important to understand that an equation section contains equations, not assignments. Consider this, if I gave you the following equations:
x + y = 3;
x + 2*y = 5;
If you understand that this is a mathematical context, you can then determine that x must have a value of 1 and y must have a value of 2. In other words, you have to solve a system of simultaneous equations. You'll note that the left hand side of these equations are not variables (in general), they are expressions. An equation is simply a relationship that equates one expression, on the left hand side, with another expression, on the right hand side. Furthermore, this relationship is always true and so order is irrelevant.
This is quite different from imperative programming languages with imperative semantics. But it is also very powerful because you can state these relationships (linear systems of equations, non-linear systems of equations, implicit equations, etc) and the compiler will work out the most efficient way to solve them.
Getting back to your example, when you look at the code in your question you are interpreting those equations as assignment statements. This notion is reinforced because they just happen to have variables on the left hand sides. But they are really equations. In an equation based system, you do not worry about whether a given variable has been assigned to previously. Instead, the requirement is simply that for every variable there exists (somewhere) an equation and that there are no extra equations. In other words, you should have the same number of variables as unknowns and that the system of equations has a unique solution. That is all that Modelica requires.
Now, Modelica supports the kind of imperative semantics you are used to. But they are only to be used in special cases because they constrain the interpretation of the mathematical behavior in such a way that it interferes with the symbolic manipulation that allows Modelica compilers to generate really fast code. So it is more than a question of style. You should use equations if at all possible and algorithms in Modelica should only be used as a last resort.
One last note. Some people may be wondering "Are you telling me that these equations will be put into some giant system of equations and solved by matrix inversion or Newton-Raphson or something? Why make it so complicated when it could obviously be solved in a much easier way!" But it will not be solved as a giant system of equations. If it can be solved as a simple set of assignments it will. That is one (among many) of the different symbolic manipulation techniques that will be applied. In fact, this is a key point about Modelica...you don't need to worry about optimizing the solution method, the tool will take care of that. And more importantly, if you connect components in such a way that a simultaneous system does arise, you don't need to worry about that either. Modelica tools can handle such "algebraic loops" for you, they will optimize it to find the most computationally efficient formulation and won't depend on you reformulating your model for those cases.
Does that help?
You cannot know the execution order of the equations in a Modelica model until you run a Modelica tool on it (you can re-order any equation in the source model and get the same result). And then the order is only true for this tool with the settings you used.
This was the order chosen by the OpenModelica compiler (omc +s +simCodeTarget=Dump model.mo):
error = ref - h;
outCtr = K * (error + x);
der(x) = DIVISION(error, T, #SHARED_LITERAL_2(String#);
qOutflow = LimitValue(minV, maxV, (-flowGain) * outCtr);
qInflow = if time > 150.0 then 3.0 * flowLevel else flowLevel;
der(h) = DIVISION(qInflow - qOutflow, area, #SHARED_LITERAL_3(String#);
This example was a little boring because the left and right sides of no equation changed place (h = error - ref would be viable if h was not chosen as a state variable, etc).