Could you possibly suggest a differentiable approximation for the round Matlab function?
The round function looks like this:
I would like a differentiable function which [closely] resembles the one in the figure.
You could write the derivative using the dirac delta - it's called 'dirac comb'.
Also you can write the round function as sum of the Heaviside step functions.
for both there are many approximations - just select a differentiable one.
Well, y=x is differentiable and, if you stand far enough back, looks exactly like your target function. It does differ from round in that it maps reals to reals rather than reals to integers.
I don't think it makes sense to talk about the differentiability of functions from the reals to the integers for anything other than trivial functions such as f(r) = 1 for all r in Reals.
You can probably get something closer to the step function by a judicious choice of trigonometric function. For example, this
is a plot of sin(x)+x. If I had more time I could probably tweak it to pass through (k,k) for all integers k, but I haven't so I'll leave that to you or other passers-by.
The derivative of your function is 1 almost everywhere. If possible merely use 1.
If you really need to make the function smooth, consider using a low-pass filter.
Related
I'm using scipy.optimize.minimize with the Newton-CG (Newton Conjugate Gradient) method since I have an objective function for which I know the analytical Jacobian and Hessian. However, I need to add a regularization term R=exp(max(s)) based on the maximum value inside the array parameter "s" that being fit. It isn't entirely obvious to me how to implement derivatives for R. Letting the minimization algorithm do numeric derivatives for the whole objective function isn't an option, by the way, because it is far too complex. Any thoughts, oh wise people of the web?
I want to implement densities of probability measures in Matlab. For that I define density as a function handle such that the integral of some function f (given as a function handle) on the interval [a,b] can be computed by
syms x
int(f(x)*density(x),x,a,b)
When it comes to the Dirac measure the problem is that
int(dirac(x),x,0,b)
delivers the value 1/2 instead of 1 for all b>0. However if I type
int(dirac(x),x,a,b)
where a<0 and b>0 the returned value is 1 as it should be. For this reason multiplying by 2 will not suffice as I want my density to be valid for all intervals [a,b]. I also dont want to distinct cases before integrating so that the code remains valid for a large class of densities.
Does someone know how I can implement the Dirac probability measure (as defined here) in Matlab?
There's no unique, accepted definition for int(delta, 0, b). The problem here isn't that you're getting the "wrong" answer as that you want to impose a different convention somehow on what the delta function is than what was provided by Matlab. (Their choice is defensible but not unique.) If you evaluate this in Wolfram Alpha, for example, it will give you theta(0) - which is not defined as anything in particular. Here theta is the Heaviside function. If you want to impose your own convention, implement your own delta function.
EDIT
I see you wrote a comment on the question, while I was writing this answer, so.... Keep in mind that the Dirac measure or Dirac delta function is not a function at all. The problem you're having, along with what's described below, all relate to trying to give a functional form to something that is inherently not a function. What you are doing is not well-defined in the framework that you have in Matlab.
END OF EDIT
To put the point about conventions in context, the delta function can be defined by different properties. One is that int(delta(x) f(x), a, b) = f(0) when a < 0 < b. This doesn't tell you anything about the integral that you want. Another, which probably leads to an answer as you're getting from Matlab, is to define it as a limit. One (but not the only choice) is the limit of the zero-mean Gaussian as the variance goes to 0.
If you want to use a convention int(delta(x) f(x), a, b) = f(0) when a <= 0 < b, that probably won't get you in much trouble, but just keep in mind that it's a convention you've chosen more than a "right" or
"wrong" answer relative to what you got from Matlab.
As related note, there is a similar choice to be made on the step function (Heaviside function) at x=0. There are conventions in which is is (a) undefined, (b) -1, (c) +1, and (d) 1/2. None is "wrong." This probably corresponds roughly to the choice on the Dirac function since the Heaviside is (roughly) the integral of the Dirac.
I need to take derivatives in Matlab of a lot of equations w.r.t. generic functions, which will provide me with generic derivatives, of the type:
diff(f(x,y),x)
or
D([1],f(x,y)).
What I need is to transform these derivatives into actual symbolic variables, in order to be able to use solve, etc. What I am doing now, but which is highly inefficient, is brute force string replacement. Here is a minimal working example of what I am doing:
syms x y
f(x,y) = sym('f(x,y)')
jacobian(f)
first_d = jacobian(f)
strrep(char(first_d),'D([1], f)(x, y)','fx')
In my real application, I have lots of derivatives to take from lots of equations, so looping such replacements is not the smartest thing to do. Can anybody shed some light into a more efficient solution?
Note: I'm using R2014b. Symbolic Math functionality has changed greatly in recent versions and continues to do so. Users on different versions may need to do slightly different things to achieve the results below, which relies on accessing undocumented functionality.
First, since this is about performance, it is sufficient to simply declare
syms f(x,y)
which also defines x and y as symbolic variables.
As I mention in my comments above, Matlab/MuPAD's symbolic math is all about manipulating strings. Doing this more directly and and adding in you own knowledge of the problem can help speed things up. You want to avoid unnecessary conversions between strings and the sym/symfun types.
1. The first thing to do is investigate how a particular symbolic math function is handling input and output and what lower level private functions it is calling. In the case of your jacobian function example, type edit jacobian in your command window to view the code in the Editor. Much of what you see may be confusing, but you should see this line:
res = mupadmex('symobj::jacobian',Fsym.s,v.s);
This calls the low level 'symobj::jacobian' function and passes in string versions of the function and variables. To call this yourself, you can do (this also assumes you know your variables are x and y):
syms f(x,y)
first_d = mupadmex('symobj::jacobian',char(f),char([x,y]))
This returns [ diff(f(x, y), x), diff(f(x, y), y)]. The undocumented mupadmex function is a direct way of calling MuPAD function from within Matlab – there are others, which are documented.
2. You'll notice that that the first_d output above is symfun class. We actually don't want want the output to be converted back to a symbolic function. To avoid this, we can pass an addition argument to mupadmex:
syms f(x,y)
first_d = mupadmex('symobj::jacobian',char(f),char([x,y]),0)
with now returns the string matrix([[diff(f(x, y), x), diff(f(x, y), y)]]). (I only know this trick of adding the additional 0 argument from having browsed through a lot of Symbolic Math toolbox code.)
3. From this string, we can now find and replace various patterns for partial derivatives with simple variables. The strrep function that you're using is generally a good choice for this. It is much faster than regexprep. However, if you have a large number of different, but similar, patterns to replace, you might do a performance comparison between the two. That would probably be the subject of a separate question.
I'm not sure what your overall goal is or the full extent of your problem, but here is my final code for your example:
syms f(x,y)
first_d = mupadmex('symobj::jacobian',char(f),char([x,y]),0)
first_d = strrep(first_d(9:end-2),'diff(f(x, y), x)','fx');
first_d = sym(strrep(first_d,'diff(f(x, y), y)','fy'));
This returns the symbolic vector [ fx, fy]. If you want a symfun, you'll need to modify the last line slightly. In some simple testing, this basic example is about 10% faster than calling jacobian and converting the result back to a string. If you directly specify the inputs as strings instead of allocating a symbolic function, the result is about 30% faster then your original:
first_d = mupadmex('symobj::jacobian','f(x,y)','[x,y]',0)
first_d = strrep(first_d(9:end-2),'diff(f(x, y), x)','fx');
first_d = sym(strrep(first_d,'diff(f(x, y), y)','fy'));
Using subs, as in this answer, while convenient, is the slowest approach. Converting back and forth to and from strings is costly.
First off: I understand derivatives and the chain rule. I'm not great with math, but I have an understanding.
Numerous tutorials on Backpropogation (let's use this and this) using gradient descent state that we use the derivative of the transfer function (sigmoid or tanh) to calculate the gradient and hence which way to travel next. In these tutorials I see (t-o)(1-o)(o) as the formula for calculating the error for the output neurons, which seems to be the derivative of the error calculation (1/2)(t-o)^2 * (1-o). (t = target, o = output_actual)
Why do I not see the derivative of the transfer function (assuming sigmoid): e^x/((e^x + 1)^2) anywhere? Or for when tanh is used as the transfer function: sech^2(x) ... where
x = weighted input?
Also, some tutorials use (target - actual) , (target - actual)^2 [ Sum of Squares - useful for negative outputs] or the squared error function: (1/2)(target - actual)^2.
Where is the derivative of the transfer function and which is the correct error formula to use?
Why do I not see the derivative of the transfer function (assuming sigmoid): e^x/((e^x + 1)^2) anywhere?
You do, its expressed as with the derivative in the wiki page you link. If we expand the latter we get
(1/(1+e^-x))*(1-1/(1+e^-x)) = e^x/(e^x+1)^2
which is the original form you noted.
Or for when tanh is used as the transfer function: sech^2(x) ... where x = weighted input?
Well, in this case its because the page doesn't mention the tanh as a potential activation function. But in real life it is expressed in a similar way so that we can avoid any unnecessary computations.
(target - actual)^2 [ Sum of Squares - useful for negative outputs] or the squared error function: (1/2)(target - actual)^2.
The difference is only a constant factor. The math comes out a little nicer if you keep the division by 2. In practice the only thing that would change is your learning rate gets implicitly multiplied/divided by 2 depending on which perspective you look at.
Also, some tutorials use (target - actual)
You probably misread. (t-a) would be the derivative of (t-a)^2/2 . Just (t-a) would have a derivative of -1, which I'm fairly sure would hinder learning for a nn.
It is a common calculus topic to find derivatives for functions.
You may also use online mathematica to do so, here: http://www.wolframalpha.com/
The code to enter is
D[ 1/(1+e^(-x)), x ]
You can enter any function using Mathematica notation: http://integrals.wolfram.com/about/input/.
With the derivative, you can plug it in the general formula for error functions. When the derivative is too complex, you can try to use the function Simplify[...] to find better analytic forms.
As for choosing which transfer function to use, you can consider their domains and ranges. The logistic function (1 / (1 + exp(-x)) has range (0,1) but the atan(x) function has range (-1, 1). If you perform mathematical analysis on the learning algorithms, the choice of the transfer function may matter a lot. However, if you are running simulations, the choice of transfer functions should not be critical, as long as they have the S-shape (sigmoidal).
Another thing to pint out, the logistic function (1 / (1 + exp(-x) ) is only one instance of sigmoidal functions. atan(x) is also sigmoidal.
I'm trying to write a generic function for finding the cosine of a value inputted into the function. The formula for cosine that I'm using is:
n
cosx = sum((-1)^n*x^(2n)/(2n)!)
n=1
I've looked at the matlab documentation and this page implies that the "sum" function should be able to do it so I tried to test it by entering:
sum(x^n, n=1..3)
but it just gives me "Error: The expression to the left of the equals sign is not a valid target for an assignment".
Is summing an infinite series something that matlab is able to do by default or do I have to simulate it using a function and loops?
Well if you want to approximate it to a finite number of terms you can do it in Matlab without toolboxes or loops:
sumCos = #(x, n)(sum(((-1).^(0:n)).*(x.^(2*(0:n)))./(factorial(2*(0:n)))));
and then use it like this
sumCos(pi, 30)
The first parameter is the angle, the second is the number of terms you want to take the series to (i.e. effects the precision). This is a numerical solution which I think is really what you're after.
btw I took the liberty of correcting your initial sum, surely n must start from 0 if you are trying to approximate cos
If you want to understand my formula (which surely you do) then you need to read up on some essential Matlab basics namely the colon operator and then the concept of using . to perform element-wise operations.
In MATLAB itself, no, you cannot solve an infinite sum. You would have to estimate it as you suggested. The page you were looking at is part of the Symbolic Math toolbox, which is an add-on to MATLAB. In particular, you were looking at MuPAD, which is rather similar to Mathematica. It is a symbolic math workspace, whereas MATLAB is more of a numeric math workspace. If you own the Symbolic Math toolbox, you can either use MuPAD as you tried to above, or you can use the symsum function from within MATLAB itself to carry out sums of series.