I have successfully used annotation(derivative) in Modelica functions. Now I have reached a point where I think I need to use zeroDerivative or noDerivative, but from the specification I just do not understand what is the difference, and when to use what.
https://specification.modelica.org/v3.4/Ch12.html#declaring-derivatives-of-functions
It seems zeroDerivative is for time-constant parameters??
Does somebody have a simple example?
Use zeroDerivative to refer to inputs that are non-varying, i.e. parameters or constant values.
Use noDerivative for signals that do not have a derivative value. For example if an input signal comes from an external function.
The important case for noDerivative is when the input is "redundant".
As an example consider the computation of density for some media in MSL:
The density computation is found in Modelica.Media.R134a.R134a_ph.density_ph (note this does not contain any derivative in itself):
algorithm
d := rho_props_ph(
p,
h,
derivsOf_ph(
p,
h,
getPhase_ph(p, h)));
where the top function called is:
function rho_props_ph
"Density as function of pressure and specific enthalpy"
extends Modelica.Icons.Function;
input SI.Pressure p "Pressure";
input SI.SpecificEnthalpy h "Specific enthalpy";
input Common.InverseDerivatives_rhoT derivs
"Record for the calculation of rho_ph_der";
output SI.Density d "Density";
algorithm
d := derivs.rho;
annotation (
derivative(noDerivative=derivs) = rho_ph_der ...);
end rho_props_ph;
So the derivs-argument is sort of redundant and is given by p and h; and we don't need to differentiate it again. If you send in a derivs-argument that isn't given in this way may give unpredictable result, but describing this in detail would be too complicated. (There was some idea of noDerivative=something - but even just specifying it turned out to be too complicated.)
For zeroDerivative the corresponding requirement is that the arguments have zero derivative; that is straightforward to verify and if non-zero we cannot use the specific derivative (it is possible to specify multiple derivatives and use another derivative one for that case).
Related
I'm wondering if it's possible to define a natural variable n in TI-Nspire CAS. For example I'd like to write:
You can't define your own natural variables. However, Nspire has the following special variables you can use:
#n0...#n255: Restricted to natural numbers
#c0...#c255: Restricted to real numbers
You can replace the original variables with them by hand or for convience just put |x=#n0 and y=#n1 at the end of line.
Example: You are calculating fourier coefficients and know that variable k will only get real numbers from Σ operation. Replacing k with #n1 will simpilfy the function.
Picture
(Calculator needs to be in RAD mode if you want to try)
The answer is no. Variables in NSpire store a value. A variable has no type. Solve might return #n1 in a result to indicate an arbitrary natural number, but you can tell solve to look for integer solutions only.
I'm new to reinforcement learning at all, so I may be wrong.
My questions are:
Is the Q-Learning equation ( Q(s, a) = r + y * max(Q(s', a')) ) used in DQN only for computing a loss function?
Is the equation recurrent? Assume I use DQN for, say, playing Atari Breakout, the number of possible states is very large (assuming the state is single game's frame), so it's not efficient to create a matrix of all the Q-Values. The equation should update the Q-Value of given [state, action] pair, so what will it do in case of DQN? Will it call itself recursively? If it will, the quation can't be calculated, because the recurrention won't ever stop.
I've already tried to find what I want and I've seen many tutorials, but almost everyone doesn't show the background, just implements it using Python library like Keras.
Thanks in advance and I apologise if something sounds dumb, I just don't get that.
Is the Q-Learning equation ( Q(s, a) = r + y * max(Q(s', a')) ) used in DQN only for computing a loss function?
Yes, generally that equation is only used to define our losses. More specifically, it is rearranged a bit; that equation is what we expect to hold, but it generally does not yet precisely hold during training. We subtract the right-hand side from the left-hand side to compute a (temporal-difference) error, and that error is used in the loss function.
Is the equation recurrent? Assume I use DQN for, say, playing Atari Breakout, the number of possible states is very large (assuming the state is single game's frame), so it's not efficient to create a matrix of all the Q-Values. The equation should update the Q-Value of given [state, action] pair, so what will it do in case of DQN? Will it call itself recursively? If it will, the quation can't be calculated, because the recurrention won't ever stop.
Indeed the space of state-action pairs is much too large to enumerate them all in a matrix/table. In other words, we can't use Tabular RL. This is precisely why we use a Neural Network in DQN though. You can view Q(s, a) as a function. In the tabular case, Q(s, a) is simply a function that uses s and a to index into a table/matrix of values.
In the case of DQN and other Deep RL approaches, we use a Neural Network to approximate such a "function". We use s (and potentially a, though not really in the case of DQN) to create features based on that state (and action). In the case of DQN and Atari games, we simply take a stack of raw images/pixels as features. These are then used as inputs for the Neural Network. At the other end of the NN, DQN provides Q-values as outputs. In the case of DQN, multiple outputs are provided; one for every action a. So, in conclusion, when you read Q(s, a) you should think "the output corresponding to a when we plug the features/images/pixels of s as inputs into our network".
Further question from comments:
I think I still don't get the idea... Let's say we did one iteration through the network with state S and we got following output [A = 0.8, B = 0.1, C = 0.1] (where A, B and C are possible actions). We also got a reward R = 1 and set the y (a.k.a. gamma) to 0.95 . Now, how can we put these variables into the loss function formula https://imgur.com/a/2wTj7Yn? I don't understand what's the prediction if the DQN outputs which action to take? Also, what's the target Q? Could you post the formula with placed variables, please?
First a small correction: DQN does not output which action to take. Given inputs (a state s), it provides one output value per action a, which can be interpreted as an estimate of the Q(s, a) value for the input state s and the action a corresponding to that particular output. These values are typically used afterwards to determine which action to take (for example by selecting the action corresponding to the maximum Q value), so in some sense the action can be derived from the outputs of DQN, but DQN does not directly provide actions to take as outputs.
Anyway, let's consider the example situation. The loss function from the image is:
loss = (r + gamma max_a' Q-hat(s', a') - Q(s, a))^2
Note that there's a small mistake in the image, it has the old state s in the Q-hat instead of the new state s'. s' in there is correct.
In this formula:
r is the observed reward
gamma is (typically) a constant value
Q(s, a) is one of the output values from our Neural Network that we get when we provide it with s as input. Specifically, it is the output value corresponding to the action a that we have executed. So, in your example, if we chose to execute action A in state s, we have Q(s, A) = 0.8.
s' is the state we happen to end up in after having executed action a in state s.
Q-hat(s', a') (which we compute once for every possible subsequent action a') is, again, one of the output values from our Neural Network. This time, it's a value we get when we provide s' as input (instead of s), and again it will be the output value corresponding to action a'.
The Q-hat instead of Q there is because, in DQN, we typically actually use two different Neural Networks. Q-values are computed using the same Neural Network that we also modify by training. Q-hat-values are computed using a different "Target Network". This Target Network is typically a "slower-moving" version of the first network. It is constructed by occasionally (e.g. once every 10K steps) copying the other Network, and leaving its weights frozen in between those copy operations.
Firstly, the Q function is used both in the loss function and for the policy. Actual output of your Q function and the 'ideal' one is used to calculate a loss. Taking the highest value of the output of the Q function for all possible actions in a state is your policy.
Secondly, no, it's not recurrent. The equation is actually slightly different to what you have posted (perhaps a mathematician can correct me on this). It is actually Q(s, a) := r + y * max(Q(s', a')). Note the colon before the equals sign. This is called the assignment operator and means that we update the left side of the equation so that it is equal to the right side once (not recurrently). You can think of it as being the same as the assignment operator in most programming languages (x = x + 1 doesn't cause any problems).
The Q values will propagate through the network as you keep performing updates anyway, but it can take a while.
I want to know if a model can be inversed in modelica. (here inverse means: if in causal statement y= x +a; x and a are input and y is output; but if I want to find 'x' as output and 'y' and 'a' as input, the model is called reversed/inversed model) For example, if I have compressor with input air port and output air port, and port has variables associated with it are pressure(P), temperature(T) and mass flow rate(mdot). I have simple steady state model containing three equations as follow:
OutPort.mdot = InPort.mdot
OutPort.P = rc * InPort.P
OutPort.T = InPort.T * (1 + rc[ (gamma-1)/gamma) - 1][/sup] / eta);
Here, rc, gamma and eta are compression ratio, ratio of specific heat capacitites and efficiency of compressor respectively.
I want to know, if I know values of : gamma, eta, OutPort.mdot, OutPort.P and OutPort.T and InPort.P and InPort.T, can I find the value of rc.
Can I find values of rc and how should be the model of compressor with above equation in Modelica. As far as I know, there are some variables designated as parameters which can not be changed during simulation. How the modelica model should be with above equations
Thanks
Yes, this should not be a problem as long as you make sure that rc is not a parameter, but a normal variable, and you supply the appropriate number of known quantities to achieve a balanced system (roughly, number of unknowns matches number of equations).
E.g. in your case if you know/supply OutPort.P and InPort.P, rc is already determined from eq 2. Then, in the third equation, there are no unknowns left, so either the temperature values are consistent with the equation or you (preferably) leave one temperature value undetermined.
In addition if you only want to compute the parameter rc during steady-state initialization i.e. that nothing changes with time that is also possible:
...
parameter Real rc(fixed=false);
initial equation
Inport.mdot=12; // Or something else indirectly determining rc.
The fixed=false means that rc is indirectly determined from the initialization. However, if the model is not completely stationary it will only find the correct rc during the initialization and then use that afterwards.
I created an part with the AC library, and when I was trying to simulate the model, there is an error says "Current version of the modelica translator can only handle array of components with fixed size".
Not sure what is the meaning of it, and is there anyone has the same issue like this one?
Thank you
enter image description here
Consider the following simple model:
model M
parameter Integer n(start=3, fixed=false);
initial algorithm
n := n;
end M;
It has a parameter n which can be changed before simulation starts. And array dimensions need to be parameter expressions. So you would think that the following model would be legal:
model M2
Real arr[n] = fill(1, n);
parameter Integer n(start=3, fixed=false);
initial algorithm
n := n;
end M2;
But it isn't since Modelica tools will expand the number of equations and variables to get a fixed number. (According to the language specification, n is a structural parameter; it is not well defined what restrictions these have - most Modelica tools seem to require them to behave like constants which means only fixed=true parameters with a binding equation that depends only on other structural parameters or constants).
Just few questions, i hope someone will find time to answer :).
What if we have COUPLED model example: system of n indepedent variables X and n nonlinear partial differential equations PDEf(X,PDEf(X)) with respect to TIME that depends of X,PDEf(X)(partial differential equation depending of variables X ). Can you give some advice? Here is one example:
Let’s say that c is output, or desired variable. Let’s say that r is independent variable.Partial differential equation looks like:
∂c/∂t=D*1/r+∂c/∂r+2(D* (∂^2 c)/(∂r^2 ))
D=constant
r=0:0.1:Rp- Matlab syntaxis, how to represent same in Modelica (I use integrator,but didn't work)?
Here is a code (does not work):
model PDEtest
/* Boundary conditions
1. delta(c)/delta(r)=0 for r=0
2. delta(c)/delta(r)=-j*d for r=Rp*/
parameter Real Rp=88*1e-3; // length
parameter Real initialConc=1000;
parameter Real Dp=1e-14;
parameter Integer np=10; // num. of points
Real cp[np](start=fill(initialConc,np));
Modelica.Blocks.Continuous.Integrator r(k=1); // independent x1
Real j;
protected
parameter Real dr=Rp/np;
parameter Real ts= 0.01; // for using when loop (sample(0,ts) )
algorithm
j:=sin(time); // this should be indepedent variable like x2
r.u:=dr;
while r.y<=Rp loop
for i in 2:np-1 loop
der(cp[i]):=2*Dp/r.y+(cp[i]-cp[i-1])/dr+2*(Dp*(cp[i+1]-2*cp[i]+cp[i-1])/dr^2);
end for;
if r.y==Rp then
cp[np]:=-j*Dp;
end if;
cp[1]:=if time >=0 then initialConc else initialConc;
end while;
annotation (uses(Modelica(version="3.2")));
end PDEtest;
Here are more questions:
This code don’t work in OpenModelica 1.8.1, also don’t work in Dymola 2013demo. How can we have continuos function of variable c, not array of functions ?
Can we place values of array cp in combiTable? And how?
If instead “algorithm” stay “equation” code can’t be succesfull checked.Why? In OpenModelica, error is :could not flattening model :S.
Is there any simplified way to use a set of equation (PDE’s) that are coupled? I know for PDEs library in Modelica, but I think they are complicated. I want to write a function for solving PDE and call these function in “main model”, so that output of function be continuos function of “c”.I don’t know what for doing with array of functions.
Can you give me advice how to understand Modelica language, if we “speak” like in Matlab? For example: Values of independent variable r,we can specife in Matlab, like r=0:TimeStep:Rp…How to do same in Modelica? And please explain me how section “equation” works, is there similarity with Matlab, and is there necessary sequancial approach?
Cheers :)
It's hard to answer your question, since you assuming that Modelica ~ Matlab, but that's not the case. So I won't comment your code, since it's really wrong. Let me give you an example model to the burger equation. Maybe you could use it as starting point.
model burgereqn
Real u[N+2](start=u0);
parameter Real h = 1/(N+1);
parameter Integer N = 10;
parameter Real v = 234;
parameter Real Pi = 3.14159265358979;
parameter Real u0[N+2]={((sin(2*Pi*x[i]))+0.5*sin(Pi*x[i])) for i in 1:N+2};
parameter Real x[N+2] = { h*i for i in 1:N+2};
equation
der(u[1]) = 0;
for i in 2:N+1 loop
der(u[i]) = - ((u[i+1]^2-u[i-1]^2)/(4*(x[i+1]-x[i-1])))
+ (v/(x[i+1]-x[i-1])^2)*(u[i+1]-2*u[i]+u[i+1]);
end for;
der(u[N+2]) = 0;
end burgereqn;
Your further questions:
cp is an continuous variable and the array is representing
every discretization point.
Why you should want to do that, as far as I understand cp is
your desired solution variable.
You should try to use almost always equation section
algorithm sections are usually used in functions. I'm pretty
sure you can represent your desire behaviour with equations.
I don't know that library, but the hard thing on a pde is the
discretization and the solving it self. You may run into issues
while solving the pde with a modelica tool, since usually
a Modelica tool has no specialized solving algorithm for pdes.
Please consider for that question further references. You could
start with Modelica.org.