I am trying to build a complex neural network using Computation Graph implementation in Deeplearning4J. I need to have multiple outputs so that's why I can't go with the generic MultiLayerConfiguration.
However, my problem is that in this case I do not know how to do the evaluation of my model and I would like at least to know the accuracy.
Has anybody worked with Comp Graphs in dl4j?
First of all yes: tons of people use computation graph. They usually start from our existing examples though and tend to mainly use it for things like seq2seq.
As for your question on evaluation, it's conceptually the same as multi layer network. How you evaluate is likely going to be task specific though. If you think about where evaluation happens, it's always tied to a task (classification,regression,binary classification,..) with an output layer . In the most common case usually you only have 1 output which outputs a classification. In that case you can just use the first array it outputs.
Otherwise for multiple outputs..you'd have to define what you're evaluating. Usually tasks merge to 1 path.
If they don't, you'd have multiple output layers where you want to do an evaluation object per output.
Computation graphs and multi layer network both use a .output method to give you raw arrays. That is typically what you pass to eval.eval.
Related
This is my first time working with DL4J (Deep Learning for Java) and also my first Convolutional Neural Network. My Goal is to use the Convolutional Neural Netowrk to give me some predicted values about an image. I gathered and labelled my images myself. The labels or expected outputs consist of two numbers between 0 and 1 (I just wrote them in the file name like 0.01x0.87.jpg).
Now I can't find any way to use the DataSetIterator Class which DL4J uses so that I can also set my label values.
Is there a simple way to tell DL4J that I want to train my Network to recognize that image 0.01x0.01.jpg should spit out the values 0.01 and 0.01?
What you want to do is usually known as regression. In contrast to classification where you want to either have a 0 or 1 output, in regression any value can be the target.
In your case, you will likely want to use a network architecture that uses either a sigmoid (which forces your values to be between 0 and 1) or an identity (which keeps the values as is, i.e. allows for them to be outside of the 0 to 1 range) activation function.
As you have two values that you are trying to predict, you will have to also define that you are using two outputs.
So much for your model architecture.
For data loading, you can use the ImageRecordReader, but also pass it a PathMultiLabelGenerator of your own. When you implement the PathMultiLabelGenerator interface, you will get the full path of the image as a string, and you can do whatever you want with it, like for example remove the file ending, split on x and parse your filename into a list of DoubleWritable. DoubleWritable is just a simple wrapper class for double so creating that is as easy as just instantiating it by passing the actual value to the constructor.
To create a dataset iterator you can now follow the documentation on RecordReaderDataSetIterator.
Essentially, I have a model with given a set of parameters is able to calculate different thermodynamic properties for different compounds, let's say liquid densities and vapor pressures for example.
When I want to regress the model parameters (e.g. a,b,c,d,e) by fitting data for different compounds, I usually do a lot of sequential operations that I am sure I can easily improve their efficiency. I am thinking about multiobjective optimization or even better-using GPU or multicores of the CPU. But I am a bit lost on where to start from reading just the documentation.
So within my objetive function I usually have something like:
[fval]= objective_function(a,b,c,d,e)
input_comp1=f(constants,a,b,c,d,e)
input_comp2=f(constants,a,b,c,d,e)
exp_density1=load some text file or so.
exp_density2=load some text file or so.
exp_vaporpressure1=load some text file or so.
exp_vaporpressure2=load some text file or so.
densities_1=density(a,b,c,d,e,input_comp1)
vapor_pressures1=vapor_pressures(a,b,c,d,e,input_comp1)
densities_2=density(a,b,c,d,e,input_comp2)
vapor_pressures=vapor_pressures (a,b,c,d,e,input_comp2)
ARD_d1=expression for deviations between experimental and calculated values for density of comp.1
ARD_d2=...
ARD_p1=...
ARD_p2=...
fval= ARD_d1+ARD_d2+ARD_p1+ARD_p2
Which is then evaluated by something like fminsearch but I have also used others in the past, fminsearch worked the best for me. When I do this for just one component It works fast enough for my purpose (but I am a patient man). But now I've extended the model in a way that I need to regress parameters simultaneous from more than one component and it becomes impossible.
I am quite sure this way of doing the calculations can be improved because I can run the calculations for the different compounds simultaneous instead of doing them sequentially, and then evaluate fval when the calculations for all components are done. But how?
I know Matlab has the function TrainAutoencoder(input, settings) to create and train an autoencoder. The result is capable of running the two functions of "Encode" and "Decode".
But this is only applicable to the case of normal autoencoders. What if you want to have a denoising autoencoder? I searched and found some sample codes, where they used the "Network" function to convert the autoencoder to a normal network and then Train(network, noisyInput, smoothOutput)like a denoising autoencoder.
But there are multiple missing parts:
How to use this new network object to "encode" new data points? it doesn't support the encode().
How to get the "latent" variables to the features, out of this "network'?
I appreciate if anyone could help me resolve this issue.
Thanks,
-Moein
At present (2019a), MATALAB does not permit users to add layers manually in autoencoder. If you want to build up your own, you will have start from the scratch by using layers provided by MATLAB;
In order to to use TrainNetwork(...) to train your model, you will have you find out a way to insert your data into an object called imDatastore. The difficulty for autoencoder's data is that there is NO label, which is required by imDatastore, hence you will have to find out a smart way to avoid it--essentially you are to deal with a so-called OCC (One Class Classification) problem.
https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.imagedatastore.html
Use activations(...) to dump outputs from intermediate (hidden) layers
https://www.mathworks.com/help/deeplearning/ref/activations.html?searchHighlight=activations&s_tid=doc_srchtitle
I swang between using MATLAB and Python (Keras) for deep learning for a couple of weeks, eventually I chose the latter, albeit I am a long-term and loyal user to MATLAB and a rookie to Python. My two cents are that there are too many restrictions in the former regarding deep learning.
Good luck.:-)
If you 'simulation' means prediction/inference, simply use activations(...) to dump outputs from any intermediate (hidden) layers as I mentioned earlier so that you can check them.
Another way is that you construct an identical network but with the encoding part only, copy your trained parameters into it, and feed your simulated signals.
I need to write a program where part of the TensorFlow nodes need to keep being there storing some global information(mainly variables and summaries) while the other part need to be changed/reorganized as program runs.
The way I do now is to reconstruct the whole graph in every iteration. But then, I have to store and load those information manually from/to checkpoint files or numpy arrays in every iteration, which makes my code really messy and error prone.
I wonder if there is a way to remove/modify part of my computation graph instead of reset the whole graph?
Changing the structure of TensorFlow graphs isn't really possible. Specifically, there isn't a clean way to remove nodes from a graph, so removing a subgraph and adding another isn't practical. (I've tried this, and it involves surgery on the internals. Ultimately, it's way more effort than it's worth, and you're asking for maintenance headaches.)
There are some workarounds.
Your reconstruction is one of them. You seem to have a pretty good handle on this method, so I won't harp on it, but for the benefit of anyone else who stumbles upon this, a very similar method is a filtered deep copy of the graph. That is, you iterate over the elements and add them in, predicated on some condition. This is most viable if the graph was given to you (i.e., you don't have the functions that built it in the first place) or if the changes are fairly minor. You still pay the price of rebuilding the graph, but sometimes loading and storing can be transparent. Given your scenario, though, this probably isn't a good match.
Another option is to recast the problem as a superset of all possible graphs you're trying to evaluate and rely on dataflow behavior. In other words, build a graph which includes every type of input you're feeding it and only ask for the outputs you need. Good signs this might work are: your network is parametric (perhaps you're just increasing/decreasing widths or layers), the changes are minor (maybe including/excluding inputs), and your operations can handle variable inputs (reductions across a dimension, for instance). In your case, if you have only a small, finite number of tree structures, this could work well. You'll probably just need to add some aggregation or renormalization for your global information.
A third option is to treat the networks as physically split. So instead of thinking of one network with mutable components, treat the boundaries between fixed and changing pieces are inputs and outputs of two separate networks. This does make some things harder: for instance, backprop across both is now ugly (which it sounds like might be a problem for you). But if you can avoid that, then two networks can work pretty well. It ends up feeling a lot like dealing with a separate pretraining phase, which you many already be comfortable with.
Most of these workarounds have a fairly narrow range of problems that they work for, so they might not help in your case. That said, you don't have to go all-or-nothing. If partially splitting the network or creating a supergraph for just some changes works, then it might be that you only have to worry about save/restore for a few cases, which may ease your troubles.
Hope this helps!
Disclaimer: I'm not a mathematical genius, nor do I have any experience with writing neural networks. So, please, forgive whatever idiotic things I happen to say here. ;)
I've always read about neural networks being used for machine learning, but while experimenting with writing simple virtual machines, I began to wonder if they could be applied in another way.
Specifically, can a virtual machine be created as a neural network? If so, how would it work (feel free to use an abstract description here, if you have to)?
I've heard of the Joycean Machine, but I can't find any information other than very, very vague explanations.
EDIT: What I'm looking for here is an explanation of exactly how a neural network-based VM would interpret assembly. How would inputs be handled, etc? Would each individual input be a memory address? Let's brainstorm!
You really made my day buddy...
Since an already trained neural network won't be much different than a regular state machine, there is no point writing a neural network VM for a deterministic instruction set.
It might be interesting to train such a VM with multiple instruction sets or an unknown set. However, I doubt it will be practical to execute such a training and even a %99 correct interpreter will be of any use for conventional bytecode.
The only use of a neural network VM I can think of is executing a program that contains fuzzy logic constructs or AI algorithm heuristics.
Some silly stack machine example to demonstrate the idea:
push [x1]
push [y1] ;start coord
push [x2]
push [y2] ;end coord
pushmap [map] ;some struct
stepastar ;push the next step of A* heuristics to accumulator and update the map
pop ;do sth with is and pop
stepastar ;next step again
... ;stack top is a map
reward ;we liked the coordinate. reinforce the heuristic
stepastar
... ;stack top is a map
punish ;we didn't like the next coordinate. try something different
There is no explict heuristic here. Just assume we keep all state in *map including the heuristic algorithm.
You see it looks silly and not completely context sensitive but a neural network is of no value if it doesn't learn online.
Of course. With a rather complex network no doubt.
Much of the parsing of bytecodes/opcodes is pattern matching which neural networks excel at.
You could certainly do this with a neural network - I could easily see learning the correct state transitions for a given piece of bytecode.
Input could be something like:
Value at top of stack
Value in current accumulator
Byte code at current instruction pointer
Byte value at current data pointer
Previous flags
Output could be something like:
Change to instruction pointer
Change to data pointer
Change to accumulator
Stack operation (push, pop, or nothing)
Memory operation (read to accumulator, write accumulator or nothing)
New flags
However - I'm not sure why you would want to do this in the first place. A neural network would be much less efficient (and potentially make mistakes unless you trained it well enough) compared to just executing the bytecode directly. You'd probably need to write an accurate bytecode evaluator anyway just to create enough training data....
Also, in my experience neural networks tend to be good at pattern recognition but very bad at learning logical operations (like binary addition or XORs) once you get beyond a certain scale (i.e. more than a few bits). So depending on the complexity of your instruction set, the network could take a very large amount of time to train.