If I define multiple loss layers in a network, will there be multiple back propagation happening from those ends to the beginning of the network? I mean, do they even work that way?
Suppose I have something like this:
Layer1{
}
Layer2{
}
...
Layer_n{
}
Layer_cls1{
bottom:layer_n
top:cls1
}
Layer_cls_loss1{
type:some_loss
bottom:cls1
top:loss1
}
Layer_n1{
bottom:layer_n
..
}
Layer_n2{
}
...
layer_n3{
}
Layer_cls2{
bottom:layer_n3
top:cls2
}
Layer_cls_loss2{
type:some_loss
bottom:cls2
top:loss2
}
layer_n4{
bottom:layer_n3
..
}
...
layer_cls3End{
top:cls_end
bottom:...
}
loss{
bottom:cls_end
top:loss:
type: someloss
}
So basically suppose instead of having one classification/loss per each network, we have several in between as well.
And in case they work, how could I add all the losses together and make my final loss the summation of all previous losses?
Caffe does this for you.
Furthermore, for each loss layer you have loss_weight parameter that allows you to decide how influential this specific loss is with respect to all other losses in the net.
Overall, the loss minimized by caffe is the weighted sum of all loss layers in the model.
Related
I'm facing an issue while migrating my library from the deprecated scriptProcessor to AudioWorklet.
Current implementation with ScriptProcessor
It currently use the AudioProcessingEvent, inputBuffer property, which is an AudioBuffer.
I apply to this inputBuffer a lowpass filter thanks to OfflineAudioContext then analyze the peaks (of bass frequencies) to count and compute BPM candidates.
The issue is that the lowpass filter work can't be done within the AudioWorkletProcessor. (OfflineAudioContext is not defined)
How to apply a lowpass filter to the sample provided by the process method of an AudioWorkletProcessor (the same way as it's doable with the onaudioprocess event data) ? Thanks
[SOLUTION] AudioWorklet implementation
For the end user it will looks like this:
import { createRealTimeBpmProcessor } from 'realtime-bpm-analyzer';
const realtimeAnalyzerNode = await createRealTimeBpmProcessor(audioContext);
// Set the source with the HTML Audio Node
const track = document.getElementById('track');
const source = audioContext.createMediaElementSource(track);
// Lowpass filter
const filter = audioContext.createBiquadFilter();
filter.type = 'lowpass';
// Connect stuff together
source.connect(filter).connect(realtimeAnalyzerNode);
source.connect(audioContext.destination);
realtimeAnalyzerNode.port.onmessage = (event) => {
if (event.data.message === 'BPM') {
console.log('BPM', event);
}
if (event.data.message === 'BPM_STABLE') {
console.log('BPM_STABLE', event);
}
};
You can find the full code under the version 3 (pre-released right now).
You could make sure to apply the lowpass filter to the signal before it reaches the AudioWorkletNode. Something like this should work.
const biquadFilterNode = new BiquadFilterNode(audioContext);
const audioWorkletNode = new AudioWorkletNode(
audioContext,
'the-name-of-your-processor'
);
yourInput
.connect(biquadFilterNode)
.connect(audioWorkletNode);
As a result your process() function inside the AudioWorkletProcessor gets called with the filtered signal.
However I think your current implementation doesn't really use a lowpass filter. I might be wrong but it looks like startRendering() is never called which means the OfflineAudioContext is not processing any data.
If that is true you may not need the lowpass filter at all for your algorithm to work.
For my task, I need to implement a neural net with some arbitrary activation functions. To be more specific these activations are ReLUs, but all have a different first derivative.
I have my implementation of neural nets for that purpose, but it trains very poorly (to some threshold that is quite big) with ReLU for some reason.
So I need a framework that can help me quickly build NN with my ReLUs and apply it to some practical task like MNIST dataset. Language does not matter a lot, but I'd like to stick with C++\C\C#\Golang.
Thank you!
If Java is alright. DL4J easily supports custom activation functions by extending BaseActivationFunction.
public class Sample {
static class MyActivationFunction extends BaseActivationFunction {
#Override
public INDArray getActivation( INDArray in, boolean training ) {
Nd4j.getExecutioner().execAndReturn( new RectifedLinear( in ) );
return in;
}
#Override
public Pair<INDArray, INDArray> backprop( INDArray in, INDArray epsilon ) {
INDArray dLdz = Nd4j.getExecutioner().execAndReturn( new RectifedLinear( in ).derivative() );
dLdz.muli( epsilon );
return new Pair<>( dLdz, null );
}
#Override
public String toString() {
return "myrelu";
}
}
public static void main( String[] args ) throws Exception {
MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
.optimizationAlgo( OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT ).iterations( 1 )
.learningRate( 0.0001 )
.seed( 123 )
.regularization( true )
.l2( 0.005 )
.weightInit( WeightInit.XAVIER )
.updater( Updater.NESTEROVS )
.activation( new MyActivationFunction() ) // << USE CUSTOM ACTIVATION FUNCTION
.list()
.layer( 0, new DenseLayer.Builder().nIn( 768 ).nOut( 200 ).build() )
.layer( 1, new DenseLayer.Builder().nIn( 200 ).dropOut( 0.2 ).nOut( 200 ).build() )
.layer( 2, new RnnOutputLayer.Builder( LossFunction.MCXENT ).activation( Activation.SOFTMAX ).nIn( 200 ).nOut( 10 ).build() )
.pretrain( false ).backprop( true )
.build();
}
}
If you can use Python, then Keras would be the best way. It can use tensorflow backend which is in c++ and supports GPU
https://keras.io/
In General for building a NeuralNet for detecting a number, you can follow these steps:
Datasets:
If you have your own dataset, check the sizes of labels and date to be suitable for the Neural Net.
If you don't have your own dataset, you can use prepared datasets like MNIST and others.
NeuralNetwork:
You can use TensorflowIn this Link or Keras. It has a ready Example with good guide about code to run it and see the results.
It's ready and easy to run and see the result and Tensorflow guide explained the functions that used. so you can with a little search change your function that you need.
I want to know if there is any difference between the two:
1) simple_port
data_valid : simple_port of bit is instance;
keep data_valid.hdl_path() == "data_valid_o";
event data_valid_f is fall(data_valid$) #sim;
on data_valid_f {
-- do some stuff
};
2) event_port
data_valid : event_port is instance;
keep data_valid.hdl_path() == "data_valid_o";
keep data_valid.edge() == fall;
on data_valid$ {
-- do some stuff
};
Thanks in advance!
Put simply, the event_port will detect glitches, i.e. zero time signal changes within a simulation cycle.
The value of the simple_port will be determined by the signal value at the end of the simulation cycle, no matter how many times it toggled before that.
I have a recursive function that needs to compare the results of the current call to the previous call to figure out whether it has reached a convergence. My function does not contain any action - it only contains map, flatMap, and reduceByKey. Since Spark does not evaluate transformations (until an action is called), my next iteration does not get the proper values to compare for convergence.
Here is a skeleton of the function -
def func1(sc: SparkContext, nodes:RDD[List[Long]], didConverge: Boolean, changeCount: Int) RDD[(Long] = {
if (didConverge)
nodes
else {
val currChangeCount = sc.accumulator(0, "xyz")
val newNodes = performSomeOps(nodes, currChangeCount) // does a few map/flatMap/reduceByKey operations
if (currChangeCount.value == changeCount) {
func1(sc, newNodes, true, currChangeCount.value)
} else {
func1(sc, newNode, false, currChangeCount.value)
}
}
}
performSomeOps only contains map, flatMap, and reduceByKey transformations. Since it does not have any action, the code in performSomeOps does not execute. So my currChangeCount does not get the actual count. What that implies, the condition to check for the convergence (currChangeCount.value == changeCount) is going to be invalid. One way to overcome is to force an action within each iteration by calling a count but that is an unnecessary overhead.
I am wondering what I can do to force an action w/o much overhead or is there another way to address this problem?
I believe there is a very important thing you're missing here:
For accumulator updates performed inside actions only, Spark guarantees that each task’s update to the accumulator will only be applied once, i.e. restarted tasks will not update the value. In transformations, users should be aware of that each task’s update may be applied more than once if tasks or job stages are re-executed.
Because of that accumulators cannot be reliably used for managing control flow and are better suited for job monitoring.
Moreover executing an action is not an unnecessary overhead. If you want to know what is the result of the computation you have to perform it. Unless of course the result is trivial. The cheapest action possible is:
rdd.foreach { case _ => }
but it won't address the problem you have here.
In general iterative computations in Spark can be structured as follows:
def func1(chcekpoinInterval: Int)(sc: SparkContext, nodes:RDD[List[Long]],
didConverge: Boolean, changeCount: Int, iteration: Int) RDD[(Long] = {
if (didConverge) nodes
else {
// Compute and cache new nodes
val newNodes = performSomeOps(nodes, currChangeCount).cache
// Periodically checkpoint to avoid stack overflow
if (iteration % checkpointInterval == 0) newNodes.checkpoint
/* Call a function which computes values
that determines control flow. This execute an action on newNodes.
*/
val changeCount = computeChangeCount(newNodes)
// Unpersist old nodes
nodes.unpersist
func1(checkpointInterval)(
sc, newNodes, currChangeCount.value == changeCount,
currChangeCount.value, iteration + 1
)
}
}
I see that these map/flatMap/reduceByKey transformations are updating an accumulator. Therefore the only way to perform all updates is to execute all these functions and count is the easiest way to achieve that and gives the lowest overhead compared to other ways (cache + count, first or collect).
Previous answers put me on the right track to solve a similar convergence detection problem.
foreach is presented in the docs as:
foreach(func) : Run a function func on each element of the dataset. This is usually done for side effects such as updating an Accumulator or interacting with external storage systems.
It seems like instead of using rdd.foreach() as a cheap action to trigger accumulator increments placed in various transformations, it should be used to do the incrementing itself.
I'm unable to produce a scala example, but here's a basic java version, if it can still help:
// Convergence is reached when two iterations
// return the same number of results
long previousCount = -1;
long currentCount = 0;
while (previousCount != currentCount){
rdd = doSomethingThatUpdatesRdd(rdd);
// Count entries in new rdd with foreach + accumulator
rdd.foreach(tuple -> accumulator.add(1));
// Update helper values
previousCount = currentCount;
currentCount = accumulator.sum();
accumulator.reset();
}
// Convergence is reached
I have two pieces v-piece and i-piece which are joined together with join_pieces().
Afterwards, the combination of those two pieces are meant to be differenced as a whole by two cubes in the piece() function.
The problem is the only piece showing a difference is the i_piece and not the v_piece which even though it is connected, it is left whole with no subtraction. I have removed the difference() line and checked to make sure the cubes are intersecting both pieces and they are. I have tried a union in case the difference was only accepting one object but it appears to not have changed anything.
Any suggestions, or answers to try? Thanks.
module join_pieces() {
union() {
v_piece();
translate([0,0,-1*stem_height+INSERT]) {
i_piece();
}
}
}
module piece() {
difference() {
join_pieces();
rotate([0,0,45]) {
cube([AIR,V_PIECE_WIDTH*4, RADIUS], center=true);
}
rotate([0,0,135]) {
cube([AIR,V_PIECE_WIDTH*4, RADIUS], center=true);
}
}
}
piece();
Could you show the variable definitions and the other functions used by these modules? I tried running your code with cubes replacing v_piece and i_piece and putting random numbers as the variables. It seems like your code is correct for what you want to do, running the modified version:
module join_pieces() {
union() {
translate([-2, 0, 0]){
#cube([5, 2, 2]);
}
translate([0,-2,-1*3+2]) {
cube([3, 5, 3]);
}
}
}
module piece() {
difference() {
join_pieces();
rotate([0,0,45]) {
cube([1,3*4, 4], center=true);
}
rotate([0,0,135]) {
cube([1,3*4, 4], center=true);
}
}
}
piece();
You can see that what you have here is fine.
Have you made sure none of your other functions are missing a semi-colon and used the # to display your difference pieces?