I want to know if a model can be inversed in modelica. (here inverse means: if in causal statement y= x +a; x and a are input and y is output; but if I want to find 'x' as output and 'y' and 'a' as input, the model is called reversed/inversed model) For example, if I have compressor with input air port and output air port, and port has variables associated with it are pressure(P), temperature(T) and mass flow rate(mdot). I have simple steady state model containing three equations as follow:
OutPort.mdot = InPort.mdot
OutPort.P = rc * InPort.P
OutPort.T = InPort.T * (1 + rc[ (gamma-1)/gamma) - 1][/sup] / eta);
Here, rc, gamma and eta are compression ratio, ratio of specific heat capacitites and efficiency of compressor respectively.
I want to know, if I know values of : gamma, eta, OutPort.mdot, OutPort.P and OutPort.T and InPort.P and InPort.T, can I find the value of rc.
Can I find values of rc and how should be the model of compressor with above equation in Modelica. As far as I know, there are some variables designated as parameters which can not be changed during simulation. How the modelica model should be with above equations
Thanks
Yes, this should not be a problem as long as you make sure that rc is not a parameter, but a normal variable, and you supply the appropriate number of known quantities to achieve a balanced system (roughly, number of unknowns matches number of equations).
E.g. in your case if you know/supply OutPort.P and InPort.P, rc is already determined from eq 2. Then, in the third equation, there are no unknowns left, so either the temperature values are consistent with the equation or you (preferably) leave one temperature value undetermined.
In addition if you only want to compute the parameter rc during steady-state initialization i.e. that nothing changes with time that is also possible:
...
parameter Real rc(fixed=false);
initial equation
Inport.mdot=12; // Or something else indirectly determining rc.
The fixed=false means that rc is indirectly determined from the initialization. However, if the model is not completely stationary it will only find the correct rc during the initialization and then use that afterwards.
Related
I need modelica to solve an equation system for a variable only once at initialization. After that the variable 'turns' into a parameter and does not change any more. Is there any way to accomplish this?
As background information: I implemented a modelica model for a simple pump which has the input parameters maximum volume flow rate, pressure loss of the system at maximum flow rate, total pipe length and surface roughness. Now I need to calculate the corresponding (mean) hydraulic diameter of the pipes so that I can estimate the pressure loss at variable volume flow rate during the normal simulation. I'm using the Colebrook-White-Approach so I need to solve an equation system.
The code looks like this. The prefix var_ indicates its a variable, param_indicates it's a known parameter. I need var_d.
// calculation of velocity and reynolds number
var_w_max = param_Q_max/(Pi/4*var_d^2);
var_Re_max = var_w_max*var_d/param_my;
// Colebrook-White approach
1/sqrt(var_lambda_max) = -2*log10(2.51/(var_Re_max*var_lambda_max)+param_k/(3.71*var_d));
param_p_loss = var_lambda_max*param_l/var_d*param_rho_h2o*var_w_max^2/2;
If you want to compute a parameter based on values at the start and then freeze it you can use an initial equation.
E.g. if you want to compute param_p_loss and param_k based on the last two equations you do:
parameter Real param_p_loss(fixed=false);
parameter Real param_k(fixed=false);
initial equation
1/sqrt(var_lambda_max) = -2*log10(2.51/(var_Re_max*var_lambda_max)+param_k/(3.71*var_d));
param_p_loss = var_lambda_max*param_l/var_d*param_rho_h2o*var_w_max^2/2;
equation
...
The fixed=false mean that the parameter needs to be solved initially.
You can in fact solve for a parameter value during initialization. The clue lies in the modifier fixed=false.
Below is a simple example of a pressure drop where you solve for a hydraulic diameter during initialization to obtain a desired nominal mass flow.
model SolveParameter
parameter Modelica.SIunits.Diameter dh(fixed=false, start=0.1)
"Hydraulic diameter. Start attribute is guess value";
parameter Real k=0.06 "Roughness, pipe length etc. combined";
parameter Modelica.SIunits.MassFlowRate m_flow_nominal=2
"Nominal mass flow rate";
parameter Modelica.SIunits.PressureDifference dp=1e5
"Differential pressure (boundary condition)";
Modelica.SIunits.MassFlowRate m_flow "Time varying mass flow rate";
initial equation
m_flow = m_flow_nominal;
equation
m_flow = dh*k*sqrt(dp);
end SolveParameter;
If the diameter is a parameter within an instatiated class (pipe model) you can apply the fixed=false when you instantiate the model, i.e.
Modelica.Fluid.Pipes.DynamicPipe pipe(diameter(fixed=false));
Best regards,
Rene Just Nielsen
I'm new to reinforcement learning at all, so I may be wrong.
My questions are:
Is the Q-Learning equation ( Q(s, a) = r + y * max(Q(s', a')) ) used in DQN only for computing a loss function?
Is the equation recurrent? Assume I use DQN for, say, playing Atari Breakout, the number of possible states is very large (assuming the state is single game's frame), so it's not efficient to create a matrix of all the Q-Values. The equation should update the Q-Value of given [state, action] pair, so what will it do in case of DQN? Will it call itself recursively? If it will, the quation can't be calculated, because the recurrention won't ever stop.
I've already tried to find what I want and I've seen many tutorials, but almost everyone doesn't show the background, just implements it using Python library like Keras.
Thanks in advance and I apologise if something sounds dumb, I just don't get that.
Is the Q-Learning equation ( Q(s, a) = r + y * max(Q(s', a')) ) used in DQN only for computing a loss function?
Yes, generally that equation is only used to define our losses. More specifically, it is rearranged a bit; that equation is what we expect to hold, but it generally does not yet precisely hold during training. We subtract the right-hand side from the left-hand side to compute a (temporal-difference) error, and that error is used in the loss function.
Is the equation recurrent? Assume I use DQN for, say, playing Atari Breakout, the number of possible states is very large (assuming the state is single game's frame), so it's not efficient to create a matrix of all the Q-Values. The equation should update the Q-Value of given [state, action] pair, so what will it do in case of DQN? Will it call itself recursively? If it will, the quation can't be calculated, because the recurrention won't ever stop.
Indeed the space of state-action pairs is much too large to enumerate them all in a matrix/table. In other words, we can't use Tabular RL. This is precisely why we use a Neural Network in DQN though. You can view Q(s, a) as a function. In the tabular case, Q(s, a) is simply a function that uses s and a to index into a table/matrix of values.
In the case of DQN and other Deep RL approaches, we use a Neural Network to approximate such a "function". We use s (and potentially a, though not really in the case of DQN) to create features based on that state (and action). In the case of DQN and Atari games, we simply take a stack of raw images/pixels as features. These are then used as inputs for the Neural Network. At the other end of the NN, DQN provides Q-values as outputs. In the case of DQN, multiple outputs are provided; one for every action a. So, in conclusion, when you read Q(s, a) you should think "the output corresponding to a when we plug the features/images/pixels of s as inputs into our network".
Further question from comments:
I think I still don't get the idea... Let's say we did one iteration through the network with state S and we got following output [A = 0.8, B = 0.1, C = 0.1] (where A, B and C are possible actions). We also got a reward R = 1 and set the y (a.k.a. gamma) to 0.95 . Now, how can we put these variables into the loss function formula https://imgur.com/a/2wTj7Yn? I don't understand what's the prediction if the DQN outputs which action to take? Also, what's the target Q? Could you post the formula with placed variables, please?
First a small correction: DQN does not output which action to take. Given inputs (a state s), it provides one output value per action a, which can be interpreted as an estimate of the Q(s, a) value for the input state s and the action a corresponding to that particular output. These values are typically used afterwards to determine which action to take (for example by selecting the action corresponding to the maximum Q value), so in some sense the action can be derived from the outputs of DQN, but DQN does not directly provide actions to take as outputs.
Anyway, let's consider the example situation. The loss function from the image is:
loss = (r + gamma max_a' Q-hat(s', a') - Q(s, a))^2
Note that there's a small mistake in the image, it has the old state s in the Q-hat instead of the new state s'. s' in there is correct.
In this formula:
r is the observed reward
gamma is (typically) a constant value
Q(s, a) is one of the output values from our Neural Network that we get when we provide it with s as input. Specifically, it is the output value corresponding to the action a that we have executed. So, in your example, if we chose to execute action A in state s, we have Q(s, A) = 0.8.
s' is the state we happen to end up in after having executed action a in state s.
Q-hat(s', a') (which we compute once for every possible subsequent action a') is, again, one of the output values from our Neural Network. This time, it's a value we get when we provide s' as input (instead of s), and again it will be the output value corresponding to action a'.
The Q-hat instead of Q there is because, in DQN, we typically actually use two different Neural Networks. Q-values are computed using the same Neural Network that we also modify by training. Q-hat-values are computed using a different "Target Network". This Target Network is typically a "slower-moving" version of the first network. It is constructed by occasionally (e.g. once every 10K steps) copying the other Network, and leaving its weights frozen in between those copy operations.
Firstly, the Q function is used both in the loss function and for the policy. Actual output of your Q function and the 'ideal' one is used to calculate a loss. Taking the highest value of the output of the Q function for all possible actions in a state is your policy.
Secondly, no, it's not recurrent. The equation is actually slightly different to what you have posted (perhaps a mathematician can correct me on this). It is actually Q(s, a) := r + y * max(Q(s', a')). Note the colon before the equals sign. This is called the assignment operator and means that we update the left side of the equation so that it is equal to the right side once (not recurrently). You can think of it as being the same as the assignment operator in most programming languages (x = x + 1 doesn't cause any problems).
The Q values will propagate through the network as you keep performing updates anyway, but it can take a while.
I have one "Thermal Mass" block in Simulink, which represents a thermal mass, which is the ability of a material or combination of materials to store internal energy. In this standard block of Simulink, the initial temperature must be entered. Only one signal can be connected to the block. The source code of the block looks like following:
component mass
% Thermal Mass
% The block represents a thermal mass, which is the ability of a material
% or combination of materials to store internal energy. The property is
% characterized by mass of the material and its specific heat.
%
% The block has one thermal conserving port.
% The block positive direction is from its port towards the block. This
% means that the heat flow is positive if it flows into the block.
% Copyright 2005-2013 The MathWorks, Inc.
nodes
M = foundation.thermal.thermal; % :top
end
parameters
mass = { 1, 'kg' }; % Mass
sp_heat = { 447, 'J/(kg*K)' }; % Specific heat
end
variables
Q = { 0, 'J/s' }; % Heat flow
end
variables(Conversion=absolute)
T = { 300, 'K' }; % Temperature
end
function setup
% Parameter range checking
if mass <= 0
pm_error('simscape:GreaterThanZero','Mass')
end
if sp_heat <= 0
pm_error('simscape:GreaterThanZero','Specific heat')
end
end
branches
Q : M.Q -> *;
end
equations
T == M.T;
Q == mass * sp_heat * T.der;
assert(T>0, 'Temperature must be greater than absolute zero')
end
end
I would like to build another component, whose initial temperature can come from another block, so that it can be also calculated somewhere else. So, one input parameter and everything else should be the same. I am new to Simulink and don't know much about the domains. Any idea, how this can be done?
Thank you!
Parameters entered on a Simulink block are usually utilized for initial values and tuning of block behavior. While newer versions of Simulink will allow you to tune some parameters during simulation, others will be locked down and un-modifiable. This may mean that you need to first execute a model to calculate the initial value for your Thermal Mass, and then start up a second simulation using that temperature as an initial value.
I believe the Simulink help on how to control block parameters will be useful. Depending on the specific design of your model, different techniques found here may be more or less applicable, but generally speaking I know of 2 easy and simple ways to accomplish modifying a mask value.
Set the value to a variable in your Matlab base workspace.
Place the block inside a Masked subsystem. The mask can be used to define a variable that accessible to all the blocks inside it.
This is not possible, while you can execute some pre-processing to determine initial temperature you can not have this as an input from other blocks.
The workaround described by Jared is probably what you're looking for.
It's actually pretty rare to need to do this, if you tell us why you'r looking to set this up, we may be able to help.
Objective
Currently I am trying to create an uncertain system based on a family of statespace models using ucover. For this I am basing my script on the documentation "Modeling a Family of Responses as an Uncertain System" which shows the technique for creating an uncertain system based on a single-input-single-output system (SISO) explicitly but makes it clear that this is fully useable for MIMO systems as well.
Technical details
Specifically it is stated with the documentation of ucover that it supports MIMO systems:
USYS = ucover(PARRAY,PNOM,ORD1,ORD2,UTYPE) returns an uncertain
system USYS with nominal value PNOM and whose range of behaviors
includes all LTI responses in the LTI array PARRAY. PNOM and PARRAY
can be SS, TF, ZPK, or FRD models. USYS is of class UFRD if PNOM
is an FRD model and of class USS otherwise.
ORD1 and ORD2 specify the order (number of states) of each diagonal
entry of W1 and W2. If PNOM has NU inputs and NY outputs, ORD1 and ORD2
should be vectors of length:
UTYPE ORD1 ORD2
InputMult NU-by-1 NU-by-1
OutputMult NY-by-1 NY-by-1
Additive NY-by-1 NU-by-1
In my case I am using both 2 inputs and 2 outputs so both ORD1 adn ORD2 should be 2 by 1. I am using 8 as the number of states used by W1 and W2 (just because, I will try adjusting that once this issue is sorted).
The Attempt
Based on the SISO example I have attempted to create a MIMO example, this is shown below
noInputs=2;
noOutputs=2;
noOfStates=4;
Anom=rand(noOfStates,noOfStates);
Bnom=rand(noOfStates,noInputs);
Cnom=rand(noOutputs,noOfStates);
Dnom=rand(noOutputs,noInputs);
Pnom=ss(Anom, Bnom, Cnom, Dnom);
p1 = Pnom*tf(1,[.06 1]); % extra lag
p2 = Pnom*tf([-.02 1],[.02 1]); % time delay
p3 = Pnom*tf(50^2,[1 2*.1*50 50^2]);
Parray = stack(1,p1,p2,p3);
Parrayg = frd(Parray,logspace(-1,3,60));
[P,Info] = ucover(Parrayg,Pnom,[8 8]',[8 8]','InputMult');
Wt = Info.W1;
bodemag((Pnom-Parray)/Pnom,'b--',Wt,'r'); grid
title('Relative Gaps vs. Magnitude of Wt')
The problem
Unlike the image in the documentation my uncertain model (when put through a bode plot) only shows a response on the lead diagonal. See the screenshot for what I mean:
Where blue is the individual models and red is the uncertain model
Question
How can I create an uncertain system based on a family of MIMO statespace models that correctly covers responses between all inputs and outputs?
If you use [8,8]' as your uncertainty order structure ord1,ord2, matlab will try to have two diagonal blocks in your uncertainty block each.
However matlab only supports diagonal weighting functions (due to some complications about nonconvex search) and what you are plotting is the diagonal weighting that will multiply the 2x2 full block LTI dynamic uncertainty. W1 affects the rows and W2 affects the columns of the uncertainty.
Hence you should check the samples of that uncertainty multiplied by the weights and then the plant. Then you can compare it with the uncertain model stack. Notice that your off-diagonal entries are practically zero (<1e-10) hence almost decoupled. But W1, W2 search looks for the H-infinity norm hence you don't get to see perfect covering at each block of the Bode plot. It combines the rows/columns of the required minimum uncertainty amount (see the examples on the help file). That's why you see 1 plot per each weight in the demos.
If you would like to model the each uncertainty affecting each block separately then you need to form a new augmented LFT such that the uncertainty is four 1x1(scalar) LTI dynamic uncertainty on the diagonal then you can have four entries in ord1 and ord2.
Since this is a MIMO system, you shouldn't compare things element-by-element. You are using the input-multiplicative form, so the uncertain system being created is of the form
Pnom*(I + W1*Delta*W2), where Delta is any stable (2-by-2, in this case) system, with ||Delta|| <= 1. So, to verify that the produced uncertain model "covers" your array of system, you should think of the equation
Parray = Pnom*(I + W1*Delta*W2)
and solve for Delta. Plot it (with SIGMA, say), and you will see that it is less than 1 in magnitude, for all frequencies. The Matlab code would be (multiply everything listed below, in order - my mulitplication symbol is not showing up in the posted answer...)
sigma(inv(W1)*inv(Pnom)*(Parrayg-Pnom)*inv(W2))
Now, using the syntax you specified, you are using weights W1 and W2 of the following form:
W1 = [W1_11 0;
0 W1_22]
and
W2 = [W2_11 0;
0 W2_22]
where you've specified 8th-order fits for all nonzero entries. Certainly for your example, this is overkill (although on a richer problem, it might be fine).
I would try much simpler, like
ucover(Parrag,Pnom,3,[],'InputMult')
That syntax will make an uncertain model of the form
Pnom*(I + w1*Delta)
where w1 is a scalar, 3rd order system. You could still see the covering by plotting SIGMA(Delta), namely
sigma((1/w1)*inv(Pnom)*(Parrayg-Pnom))
I hope that helps.
In order to create discrete or continuous time uncertain systems you can use uss associated with ureal.
Quick example
Define an uncertain propeller radius
% Propeller radius (m)
rp = ureal('rp',13.4e-2,'Range',[0.08 0.16]);
Define uncertain continuous time system
tenzo_unc = uss(A,Bw,Clocal,D,'statename',states,'inputname',inputs,'outputname',outputsLocal);
Simulate step response:
N = 5;
% Prende alcuni campioni del sistema incerto e calcola bound su incertezze
for i=1:1:N
sys{i} = usample(tenzo_unc);
step(sys{i})
hold on
cprintf('text','.');
end
Complete example
Quadcopter uncertain linearized model control with LQR. Code is available here
Step response
Closed Loop Step response
<script src="https://gist.github.com/GiovanniBalestrieri/f90a20780eb2496e730c8b74cf49dd0f.js"></script>
NB:
If you don't have the utility cprintf, include this script in your folder and use it.
Rheological models are usually build using three (or four) basics elements, which are :
The spring (existing in Modelica.Mechanics.Translational.Components for example). Its equation is f = c * (s_rel - s_rel0);
The damper (dashpot) (also existing in Modelica.Mechanics.Translational.Components). Its equation is f = d * v_rel; for a linear damper, an could be easily modified to model a non-linear damper : f = d * v_rel^(1/n);
The slider, not existing (as far as I know) in this library... It's equation is abs(f)<= flim. Unfortunately, I don't really understand how I could write the corresponding Modelica model...
I think this model should extend Modelica.Mechanics.Translational.Interfaces.PartialCompliant, but the problem is that f (the force measured between flange_b and flange_a) should be modified only when it's greater than flim...
If the slider extends PartialCompliant, it means that it already follows the equations flange_b.f = f; and flange_a.f = -f;
Adding the equation f = if abs(f)>flim then sign(f)*flim else f; gives me an error "An independent subset of the model has imbalanced number of equations and variables", which I couldn't really explain, even if I understand that if abs(f)<=flim, the equation f = f is useless...
Actually, the slider element doesn't generate a new force (just like the spring does, depending on its strain, or just like the damper does, depending on its strain rate). The force is an input for the slider element, which is sometime modified (when this force becomes greater than the limit allowed by the element). That's why I don't really understand if I should define this force as an input or an output....
If you have any suggestion, I would greatly appreciate it ! Thanks
After the first two comments, I decided to add a picture that, I hope, will help you to understand the behaviour I'm trying to model.
On the left, you can see the four elements used to develop rheological models :
a : the spring
b : the linear damper (dashpot)
c : the non-linear damper
d : the slider
On the right, you can see the behaviour I'm trying to reproduce : a and b are two associations with springs and c and d are respectively the expected stress / strain curves. I'm trying to model the same behaviour, except that I'm thinking in terms of force and not stress. As i said in the comment to Marco's answer, the curve a reminds me the behaviour of a diode :
if the force applied to the component is less than the sliding limit, there is no relative displacement between the two flanges
if the force becomes greater than the sliding limit, the force transmitted by the system equals the limit and there is relative displacement between flanges
I can't be sure, but I suspect what you are really trying to model here is Coulomb friction (i.e. a constant force that always opposes the direction of motion). If so, there is already a component in the Modelica Standard Library, called MassWithStopAndFriction, that models that (and several other flavors of friction). The wrinkle is that it is bundled with inertia.
If you don't want the inertia effect it might be possible to set the inertia to zero. I suspect that could cause a singularity. One way you might be able to avoid the singularity is to "evaluate" the parameter (at least that is what it is called in Dymola when you set the Evaluate flat to be true in the command line). No promises whether that will work since it is model and tool dependent whether such a simplification can be properly handled.
If Coulomb friction is what you want and you really don't want inertia and the approach above doesn't work, let me know and I think I can create a simple model that will work (so long as you don't have inertia).
A few considerations:
- The force is not an input and neither an output, but it is just a relation that you add into the component in order to define how the force will be propagated between the two translational flanges of the component. When you deal with acausal connectors I think it is better to think about the degrees of freedom of your component instead of inputs and outputs. In this case you have two connectors and independently at which one of the two frames you will recieve informations about the force, the equation you implement will define how that information will be propagated to the other frame.
- I tested this:
model slider
extends
Modelica.Mechanics.Translational.Interfaces.PartialCompliantWithRelativeStates;
parameter Real flim = 1;
equation
f = if abs(f)>flim then sign(f)*flim else f;
end slider;
on Dymola and it works. It is correct modelica code so it should be work also in OpenModelica, I can't think of a reason why it should be seen as an unbalance mathematical model.
I hope this helps,
Marco