Learning Julia metaprogramming to automatically construct ANN by expression. While everything works well for inference, a backward stage raises an error:
LoadError: Can't differentiate foreigncall expression
The next code shows where the problem appears. While eval(net(x)) works well, for some reason it throws an error at the gradient computation stage.
# define ANN by expression
net(x) = :($w2 * relu.($w1 * $x .+ $b1) .+ $b2)
# define loss and network evaluation
loss(x, y) = Flux.logitcrossentropy(eval(net(x)), y)
θ = Flux.Params([w1, b1, w2, b2])
# eval network and calculate gradients
gs = gradient(() -> loss(features, labels), θ) # where the problem appears
eval is a primitive which Flux can't differentiate. You'd rather use something like
net = :($w2 * relu.($w1 * x .+ $b1) .+ $b2)
#eval loss(x, y) = Flux.logitcrossentropy($net, y)
That builds the expression of loss and evaluates it once, instead of evaluating (and hence, compiling) the same piece of code every time the loss function is called.
But think twice before metaprogramming. Before resorting to eval, try to write a macro that generates your network and the loss function.
Of course this does not work if the goal is to plug together expression only available at runtime. Genetic programming, where you create a lot of random expressions at runtime, sounds like a justifiable exception. But even then, there maybe are alternatives, like using your own restricted representations (which, for example, you could first symbolically differnetiate, and then "compile" to an anonymous function).
Related
I'm new to Maple and I'm looking for a simple way to automate some tasks. In particular, I'm looking for a way to define custom "action" that perform some steps automatically.
As as an example I would like to define a quick way to compute the determinant of the Hessian of a polynomial. Currently the way I do this is opening Maple, create a new worksheet than performing the following commands:
p := (x, y) -> x^2*y + 3*x^3 + y^3
with(VectorCalculus):
h := Hessian(p(x, y), [x, y])
Determinant(h)
What I would like to do is to compute the hessian determinant directly with something like
HessDet(p)
where HessDet would be a custom command that performs the operations above. How does one achieve something like this in Maple?
First things first: The value assigned to your p is a procedure which can return a polynomial expression, but not itself a polynomial. It's important not to muddle expressions and procedures. Doing so is a common cause of problems for new users.
Being able to throw around p(x,y) may be visually pleasing to your eye, but it serves little programmatic purpose here. The fact that the formal parameters of procedure p happen to be called x and y, along with the fact that you called procedure p with arguments x and y, is actually just another common source of confusion. Don't create procedures merely to call them in this way.
Also, your call p(x,y) makes it look magic that your code snippet "knows" how many arguments would be required by procedure p. So it's already a muddle to have your candidate HessDet accept p as a procedure.
So instead let's keep it straightforward, by writing HessDet to accept a polynomial rather than a procedure. We can programmatically ascertain the names in which this expression of of type polynom.
restart;
HessDet:=proc(p::algebraic)
local H,vars;
vars:=indets(p,
And(name,Non(constant),
satisfies(u->type(p,polynom(anything,u)))));
H:=VectorCalculus:-Hessian(p,[vars[]]);
LinearAlgebra:-Determinant(H);
end proc:
Now some examples of using it,
P := x^2*y + 3*x^3 + y^3;
HessDet(P);
p := (x, y) -> x^2*y + 3*x^3 + y^3;
HessDet(p(x,y));
HessDet(x^3-x^2+4*x);
HessDet(s^2*t + 3*s^3 + t^3);
HessDet(s[r]^2*t[r] + 3*s[r]^3 + t[r]^3);
You might also wonder how you could re-use this custom procedure across sessions, without having to type it in each time. Two reasonable ways are:
Put the (above) defining plaintext definition of HessDet inside a personal initialization file.
Create a (.mla) Maple Library Archive file, then Save your HessDet to that, and then augment the Library search path in your initialization file.
It might look like 2) is more effort, but only the Save step is needed for repeats, and you can store many custom procedures to the same archive. Your choice...
[edit] The OP has asked for clarification of the first part of the above procedure HessDet, which I suspect means the call to indets.
If P is assigned an expression then then the call indets(P,name) will return a set of all the names present in that expression. Basically, it returns the set of all indeterminate subexpressions of the expression which are of type name in Maple's technical sense.
For example,
P := x*y + sin(a*Pi)*x;
x y + sin(a Pi) x
indets( P,
name );
{Pi, a, x, y}
Perhaps the name of the constant Pi is not wanted here. Ie,
indets( P,
And( name,
Non(constant) ) );
{a, x, y}
Perhaps we want only the non-constant names in which the expression is a polynomial? Ie,
indets( P,
And( name,
Non(constant),
satisfies(u->type(p,polynom(anything,u))) ) );
{x, y}
That last result is an advanced way of using the following tests:
type(P, polynom(anything, x));
true
type(P, polynom(anything, y));
true
type(P, polynom(anything, a));
false
A central issue here is that the OP made no mention of what kind of polynomials are to be handled by the custom procedure. So I guessed with some defensive coding, in hope of less surprises later on. The original Question states that the input could be a "polynomial", but we weren't told what kind of coefficients there might be.
Perhaps the coefficients will always be real and exact or numeric. Perhaps the custon procedure should throw an error when not supplied such. These details weren't mentioned in the Question.
I'm trying to build a neural network that will take in the solutions to a system of ODE's and predict the parameters of the system. I'm using Julia and in particular, the DiffEqFlux package. The structure of a network is a few simple Dense layers chained together that predict some intermediate parameters (in this case, some chemical reaction free energies), which then feed into some deterministic (non-trained) layers that convert those parameters into the ones that go into the system of equations (in this case, reaction rate constants). I've tried two different approaches from here:
Chain the ODE solve directly on as the last layer of the network. In this case, the loss function is just comparing the inputs to the outputs.
Have the ODE solve in the loss function, so the network output is just the parameters.
However, in neither case can I get Flux.train! to actually run.
A silly little example for the first option that gives the same error I'm getting (I've tried to keep as many things parallel to my actual case as possible, i.e. the solver, etc., although I did omit the intermediate deterministic layers since they don't seem to make a difference) is shown below.
using Flux, DiffEqFlux, DifferentialEquations
# let's use Chris' favorite example, Lotka-Volterra
function lotka_volterra(du,u,p,t)
x, y = u
α, β, δ, γ = p
du[1] = dx = α*x - β*x*y
du[2] = dy = -δ*y + γ*x*y
end
u0 = [1.0,1.0]
tspan = (0.0,10.0)
# generate a couple sets of solutions to train on
training_params = [[1.5,1.0,3.0,1.0], [1.4,1.1,3.1,0.9]]
training_sols = [solve(ODEProblem(lotka_volterra, u0, tspan, tp)).u[end] for tp in training_params]
model = Chain(Dense(2,3), Dense(3,4), p -> diffeq_adjoint(p, ODEProblem(lotka_volterra, u0, tspan, p), Rodas4())[:,end])
# in this case we just want outputs to match inputs
# (actual parameters we're after are outputs of next-to-last layer)
training_data = zip(training_sols, training_sols)
# mean squared error loss
loss(x,y) = Flux.mse(model(x), y)
p = Flux.params(model[1:2])
Flux.train!(loss, p, training_data, ADAM(0.001))
# gives TypeError: in typeassert, expected Float64, got ForwardDiff.Dual{Nothing, Float64, 8}
I've tried all three solver layers, diffeq_adjoint, diffeq_rd, and diffeq_fd, none of which work, but all of which give different errors that I'm having trouble parsing.
For the other option (which I'd actually prefer, but either way would work), just replace the model and loss function definitions as:
model = Chain(Dense(2,3), Dense(3,4))
function loss(x,y)
p = model(x)
sol = diffeq_adjoint(p, ODEProblem(lotka_volterra, u0, tspan, p), Rodas4())[:,end]
Flux.mse(sol, y)
end
The same error is thrown as above.
I've been hacking at this for over a week now and am completely stumped; any ideas?
You're running into https://github.com/JuliaDiffEq/DiffEqFlux.jl/issues/31, i.e. forward-mode AD for the Jacobian doesn't play nice with Flux.jl right now. To get around this, use Rodas4(autodiff=false) instead.
What benefit does it get from treating functions specially? For example,
function n = f(x)
2*x
endfunction
f(2) //outputs 4
f = #f
f(2) //outputs 4
If handles can be called the same way as functions, then what benefit do we get from functions being treated specially. By specially I mean that variables referring to functions can't be passed as arguments:
function n = m(f,x)
f(x)
end
m(f,2) // generates error since f is called without arguments
Why aren't functions procedures (which are always pointed to by variables) like in other functional languages?
EDIT:
It seems like my question has been completely misunderstood, so I will rephrase it. Compare the following python code
def f(x):
return 2*x
def m(f,x):
return f(x)
m(f,3)
to the octave code
function n = f(x)
2*x
end
function n = m(f,x)
f(x)
end
m(#f,2) % note that we need the #
So my question then is, what exactly is a function "object" in octave? In python, it is simply a value (functions are primitive objects which can be assigned to variables). What benefit does octave/matlab get from treating functions differently from primitive objects like all other functional languages do?
What would the following variables point to (what does the internal structure look like?)
x = 2
function n = f(x)
2*x
end
g = #f
In python, you could simply assign g=f (without needing an indirection with #). Why does octave not also work this way? What do they get from treating functions specially (and not like a primitive value)?
Variables referring to functions can be passed as arguments in matlab. Create a file called func.m with the following code
function [ sqr ] = func( x )
sqr = x.^2;
end
Create a file called 'test.m' like this
function [ output ] = test( f, x )
output = f(x);
end
Now, try the following
f=#func;
output = test(f, 3);
There's no "why is it different". It's a design decision. That's just how matlab/octave works. Which is very similar to how, say, c works.
I do not have intricate knowledge of the inner workings of either, but presumably a function simply becomes a symbol which can be accessed at runtime and used to call the instructions specified in its definition (which could be either interpreted or precompiled instructions). A "function handle" on the other hand, is more comparable to a function pointer, which, just like c, can either be used to redirect to the function it's pointing to, or passed as an argument.
This allows matlab/octave to do stuff like define a function completely in its own file, and not require that file to be run / imported for the function to be loaded into memory. It just needs to be accessible (i.e. in the path), and when matlab/octave starts a new session, it will create the appropriate table of available functions / symbols that can be used in the session. Whereas with python, when you define a function in a file, you need to 'run' / import that file for the function definition to be loaded into memory, as a series of interpreted instructions in the REPL session itself. It's just a different way of doing things, and one isn't necessarily better than the other. They're just different design / implementation decisions. Both work very well.
As for whether matlab/octave is good for functional programming / designed with functional programming in mind, I would say that it would not be my language of choice for functional programming; you can do some interesting functional programming with it, but it was not the kind of programming that it was designed for; it was primarily designed with scientific programming in mind, and it excels at that.
Good evening everyone,
I want to create a function
f(x) = [f1(x), f2(x), ... , fn(x)]
in MatLab, with an arbitrary form and number for the fi. In my current case they are meant to be basis elements for a finite-dimensional function space, so for example a number of multi variable polynomials. I want to able to be able to set form (e.g. hermite/lagrange polynomials, ...) and number via arguments in some sort of "function creating" function, so I would like to solve this for arbitrary functions fi.
Assume for now that the fi are fi:R^d -> R, so vector input to scalar output. This means the result from f should be a n-dim vector containing the output of all n functions. The number of functions n could be fairly large, as there is permutation involved. I also need to evaluate the resulting function very often, so I hope to do it as efficiently as possible.
Currently I see two ways to do this:
Create a cell with each fi using a loop, using something like
funcell{i}=matlabFunction(createpoly(degree, x),'vars',{x})
and one of the functions from the symbolic toolbox and a symbolic x (vector). It is then possible to create the desired function with cellfun, e.g.
f=#(x) cellfun(#(v) v(x), funcell)
This is relatively short, easy and what can be found when doing searches. It even allows extension to vector output using 'UniformOutput',false and cell2mat. On the downside it is very inefficient, first during creation because of matlabFunction and then during evaluation because of cellfun.
The other idea I had is to create a string and use eval. One way to do this would be
stringcell{i}=[char(createpoly(degree, x)),';']
and then use strjoin. In theory this should yield an efficient function. There are two problems however. The first is the use of eval (mostly on principle), the second is inserting the correct arguments. The symbolic toolbox does not allow symbols of the form x(i), so the resulting string will not contain them either. The only remedy I have so far is some sort of string replacement on the xi that are allowed, but this is also far from elegant.
So I do have ways to do what I need right now, but I would appreciate any ideas for a better solution.
From my understanding of the problem, you could do the straightforward:
Initialization step:
my_fns = cell(n, 1); %where n is number of functions
my_fns{1} = #f1; % Assuming f1 is defined in f1.m etc...
my_fns{2} = #f2;
Evaluation at x:
z = zeros(n, 1);
for i=1:n,
z(i) = my_fns{i}(x)
end
For example if you put it in my_evaluate.m:
function z = my_evaluate(my_fns, x)
z = zeros(n, 1);
for i=1:n,
z(i) = my_fns{i}(x)
end
How might this possibly be sped up?
Depends on if you have special structure than can be exploited.
Are there calculations common to some subset of f1 through fn that need not be repeated with each function call? Eg. if the common calculation step is costly, you could do y = f_helper(x) and z(i) = fi(x, y).
Can the functions f1...fn be vector / matrix friendly, allowing evaluation of multiple points with each function call?
The big issue is how fast your function calls f1 through fn are, not how you collect the results from those calls in a vector.
I'm trying to model the effect of different filter "building blocks" on a system which is a construct based on these filters.
I would like the basic filters to be "modular", i.e. they should be "replaceable", without rewriting the construct which is based upon the basic filters.
For example, I have a system of filters G_0, G_1, which is defined in terms of some basic filters called H_0 and H_1.
I'm trying to do the following:
syms z
syms H_0(z) H_1(z)
G_0(z)=H_0(z^(4))*H_0(z^(2))*H_0(z)
G_1(z)=H_1(z^(4))*H_0(z^(2))*H_0(z)
This declares the z-domain I'd like to work in, and a construct of two filters G_0,G_1, based on the basic filters H_0,H_1.
Now, I'm trying to evaluate the construct in terms of some basic filters:
H_1(z) = 1+z^-1
H_0(z) = 1+0*z^-1
What I would like to get at this point is an expanded polynomial of z.
E.g. for the declarations above, I'd like to see that G_0(z)=1, and that G_1(z)=1+z^(-4).
I've tried stuff like "subs(G_0(z))", "formula(G_0(z))", "formula(subs(subs(G_0(z))))", but I keep getting result in terms of H_0 and H_1.
Any advice? Many thanks in advance.
Edit - some clarifications:
In reality, I have 10-20 transfer functions like G_0 and G_1, so I'm trying to avoid re-declaring all of them every time I change the basic blocks H_0 and H_1. The basic blocks H_0 and H_1 would actually be of a much higher degree than they are in the example here.
G_0 and G_1 will not change after being declared, only H_0 and H_1 will.
H_0(z^2) means using z^2 as an argument for H_0(z). So wherever z appears in the declaration of H_0, z^2 should be plugged in
The desired output is a function in terms of z, not H_0 and H_1.
A workable hack is having an m-File containing the declarations of the construct (G_0 and G_1 in this example), which is run every time H_0 and H_1 are redefined. I was wondering if there's a more elegant way of doing it, along the lines of the (non-working) code shown above.
This seems to work quite nicely, and is very easily extendable. I redefined H_0 to H_1 as an example only.
syms z
H_1(z) = 1+z^-1;
H_0(z) = 1+0*z^-1;
G_0=#(Ha,z) Ha(z^(4))*Ha(z^(2))*Ha(z);
G_1=#(Ha,Hb,z) Hb(z^(4))*Ha(z^(2))*Ha(z);
G_0(H_0,z)
G_1(H_0,H_1,z)
H_0=#(z) H_1(z);
G_0(H_0,z)
G_1(H_0,H_1,z)
This seems to be a namespace issue. You can't define a symbolic expression or function in terms of arbitrary/abstract symfuns and then later on define these symfuns explicitly and be able to use them to obtain an exploit form of the original symbolic expression or function (at least not easily). Here's an example of how a symbolic function can be replaced by name:
syms z y(z)
x(z) = y(z);
y(z) = z^2; % Redefines y(z)
subs(x,'y(z)',y)
Unfortunately, this method depends on specifying the function(s) to be substituted exactly – because strings are used, Matlab sees arbitrary/abstract symfuns with different arguments as different functions. So the following example does not work as it returns y(z^2):
syms z y(z)
x(z) = y(z^2); % Function of z^2 instead
y(z) = z^2;
subs(x,'y(z)',y)
But if the last line was changed to subs(x,'y(z^2)',y) it would work.
So one option might be to form strings for case, but that seems overly complex and inelegant. I think that it would make more sense to simply not explicitly (re)define your arbitrary/abstract H_0, H_1, etc. functions and instead use other variables. In terms of the simple example:
syms z y(z)
x(z) = y(z^2);
y_(z) = z^2; % Create new explicit symfun
subs(x,y,y_)
which returns z^4. For your code:
syms z H_0(z) H_1(z)
G_0(z) = H_0(z^4)*H_0(z^2)*H_0(z);
G_1(z) = H_1(z^4)*H_0(z^2)*H_0(z);
H_0_(z) = 1+0*z^-1;
H_1_(z) = 1+z^-1;
subs(G_0, {H_0, H_1}, {H_0_, H_1_})
subs(G_1, {H_0, H_1}, {H_0_, H_1_})
which returns
ans(z) =
1
ans(z) =
1/z^4 + 1
You can then change H_0_ and H_1_, etc. at will and use subs to evaluateG_1andG_2` again.