I'm trying to implement semantic hashing for the MNISTAutoencoder example, using DL4J. How would I binarize the middle layer activations? In the ideal case, I'm looking for some change to my network setup which gives (almost) binary activations for the middle layer out-of-the-box. Alternatively, I'm happy with some "receipt" to binarize the current RELU activations. Which of both approaches is favorable in terms of generalization capabilities?
My current network setup is:
MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
.seed(12345)
.weightInit(WeightInit.XAVIER)
.updater(new AdaGrad(0.05))
.activation(Activation.RELU)
.l2(0.0001)
.list()
.layer(new DenseLayer.Builder().nIn(784).nOut(250)
.build())
.layer(new DenseLayer.Builder().nIn(250).nOut(10)
.build())
.layer(new DenseLayer.Builder().nIn(10).nOut(250)
.build())
.layer(new OutputLayer.Builder().nIn(250).nOut(784)
.activation(Activation.LEAKYRELU)
.lossFunction(LossFunctions.LossFunction.MSE)
.build())
.build();
After 30 epochs, typical middle layer activations look like:
[[ 11.3044, 12.3678, 7.3547, 1.6518, 1.0068, 0, 5.4340, 2.1388, 2.0708, 2.5764]]
[[ 9.9051, 12.5345, 11.1941, 4.7900, 1.2935, 0, 7.9786, 4.1915, 3.1802, 7.5659]]
[[ 6.4629, 11.1013, 10.8903, 5.4528, 0.8009, 0, 9.4881, 3.6684, 6.4524, 7.2334]]
[[ 2.3953, 0.2429, 3.7125, 4.1561, 0.8607, 0, 11.2486, 7.0178, 2.8771, 2.1996]]
[[ 0, 1.6378, 0.8993, 0.3347, 0.7708, 0, 3.7053, 0, 1.6704, 2.1380]]
[[ 0, 1.5158, 0.7937, 0, 0.8190, 0, 4.7548, 0.0655, 1.4635, 1.8173]]
[[ 6.8344, 5.9989, 10.1286, 2.8528, 1.1178, 0, 9.1865, 10.3677, 5.3564, 4.3420]]
[[ 7.0942, 7.0364, 4.8538, 0.5096, 0.0442, 0, 8.4336, 8.2783, 5.6474, 3.8944]]
[[ 3.6895, 14.9696, 6.5351, 8.0446, 0, 0, 12.7816, 12.7445, 7.8495, 3.8600]]
This can be established by assigning a custom IActivation function to the middle layer. For example:
public static class ActivationBinary extends BaseActivationFunction {
public INDArray getActivation(INDArray in, boolean training) {
in.replaceWhere(Nd4j.ones(in.length()).muli(-1), new LessThan(0));
in.replaceWhere(Nd4j.ones(in.length()), new GreaterThanOrEqual(0));
return in;
}
public org.nd4j.common.primitives.Pair<INDArray, INDArray> backprop(INDArray in, INDArray epsilon) {
this.assertShape(in, epsilon);
Nd4j.getExecutioner().execAndReturn(new TanhDerivative(in, epsilon, in)); // tanh's gradient is a reasonable approximation
return new org.nd4j.common.primitives.Pair(in, (Object)null);
}
public int hashCode() {
return 1;
}
public boolean equals(Object obj) {
return obj instanceof ActivationBinary;
}
public String toString() {
return "Binary";
}
}
Related
I'm trying to clone a mesh using GraphicsBuffer because its read/write option is false, and I can't change that. This is my current code:
public static bool Run(Mesh sourceMesh, out Mesh generatedMesh)
{
GraphicsBuffer sourceDataBuffer = sourceMesh.GetVertexBuffer(0);
GraphicsBuffer sourceIndexBuffer = sourceMesh.GetIndexBuffer();
var vertexCount = sourceDataBuffer.count;
var indexCount = sourceIndexBuffer.count;
byte[] sourceData = new byte[vertexCount * (sourceDataBuffer.stride / sizeof(byte))];
byte[] sourceIndex = new byte[(int)(indexCount * ((float)sourceIndexBuffer.stride / sizeof(byte)))];
sourceDataBuffer.GetData(sourceData);
sourceIndexBuffer.GetData(sourceIndex);
var attributes = sourceMesh.GetVertexAttributes();
generatedMesh = new Mesh();
generatedMesh.vertexBufferTarget |= GraphicsBuffer.Target.Raw;
generatedMesh.SetVertexBufferParams(vertexCount, attributes);
generatedMesh.SetVertexBufferData(sourceData, 0, 0, sourceData.Length);
generatedMesh.SetIndexBufferParams(indexCount, sourceMesh.indexFormat);
generatedMesh.SetIndexBufferData(sourceIndex, 0, 0, sourceIndex.Length);
generatedMesh.subMeshCount = sourceMesh.subMeshCount;
for (int i = 0; i < sourceMesh.subMeshCount; i++)
{
var subMeshDescriptor = sourceMesh.GetSubMesh(i);
generatedMesh.SetSubMesh(i, subMeshDescriptor);
}
sourceDataBuffer.Release();
sourceIndexBuffer.Release();
generatedMesh.RecalculateBounds();
return true; // No error
}
It works like a charm in my test project. But when I try to clone things in my main project, GetData(sourceData) and GetData(sourceIndex) both return arrays of 0s. What could be causing that? Could it be because of the read/write being disabled?
Edit: The problem only happens in bundles with non-readable meshes. I thought GraphicsBuffer should work with them, but it doesn't.
I'm working on a linear data layout where components are alongside each other in memory. Things were going ok until I realized I don't have a way for making offsetof and changetype calls when dealing with nested classes.
For instance, this works as intended:
class Vec2{
x:u8
y:u8
}
const size = offsetof<Vec2>() // 2 -- ok
const ptr = heap.alloc(size)
changeType<Vec2>(ptr).x = 7 // memory = [7,0] -- ok
Naturally this approach fails when nesting classes
class Player{
position:Vec2
health:u8
}
const size = offsetof<Player>() //5 -- want 3, position is a pointer
const ptr = heap.alloc(size)
changeType<Player>(ptr).position.x = 7 //[0,0,0,0,0] -- want [7,0,0], instead accidentally changed pointer 0
The goal is for the memory layout to look like this:
| Player 1 | Player 2 | ...
| x y z h | x y z h |
Ideally I'd love to be able to create 'value-type' fields, or if this isnt a thing, are there alternative approaches?
I'm hoping to avoid extensive boilerplate whenever writing a new component, ie manual size calculation and doing a changetype for each field at its offset etc.
In case anybody is interested I'll post my current solution here. The implementation is a little messy but is certainly automatable using custom scripts or compiler transforms.
Goal: Create a linear proxy for the following class so that the main function behaves as expected:
class Foo {
position: Vec2
health: u8
}
export function main(): Info {
const ptr = heap.alloc(FooProxy.size)
const foo = changetype<FooProxy>(ptr)
foo.health = 3
foo.position.x = 9
foo.position.y = 10
}
Solution: calculate offsets and alignments for each field.
class TypeMetadataBase{
get align():u32{return 0}
get offset():u32{return 0}
}
class TypeMetadata<T> extends TypeMetadataBase{
get align():u32{return alignof<T>()}
get offset():u32{return offsetof<T>()}
constructor(){
super()
if(this.offset == 0)
throw new Error('offset shouldnt be zero, for primitive types use PrimitiveMetadata')
}
};
class PrimitiveMetadata<T> extends TypeMetadataBase{
get align():u32{return sizeof<T>()}
get offset():u32{return sizeof<T>()}
};
class LinearSchema{
metadatas:StaticArray<TypeMetadataBase>
size:u32
offsets:StaticArray<u32>
constructor(metadatas:StaticArray<TypeMetadataBase>){
let align:u32 = 0
const offsets = new StaticArray<u32>(metadatas.length)
for (let i = 0; i < metadatas.length; i++){
if(metadatas[i].align !== 0)
while(align % metadatas[i].align !== 0)
align++
offsets[i] = align
align += metadatas[i].offset
}
this.offsets = offsets
this.metadatas = metadatas
this.size = align
}
}
class Vec2 {
x: u8
y: u8
}
class FooSchema extends LinearSchema{
constructor(){
super([
new PrimitiveMetadata<u8>(),
new TypeMetadata<Vec2>(),
])
}
}
const schema = new FooSchema()
class FooProxy{
static get size():u32{return schema.size}
set health(value:u8){store<u8>(changetype<usize>(this) + schema.offsets[0],value)}
get health():u8{return load<u8>(changetype<usize>(this) + schema.offsets[0])}
get position():Vec2{return changetype<Vec2>(changetype<usize>(this) + schema.offsets[1])}
}
I don't find the way to get the string specified in the dialog box of an S-function (C, level 2) block and save it in a variable, inside the .c file corresponding to the block.
Through *mxGetPr(ssGetSFcnParam(S, 0)) I can only get the first position value in the string.
The parameter entered in the dialog block what it is related to the ssGetSFcnParam(S, 0) is '123'.
In the mask editor of the block the type is 'edit'.
#define S_FUNCTION_NAME ver_file_data
#define S_FUNCTION_LEVEL 2
#define NPARAMS 14
#define DVC_TYPE(S) ssGetSFcnParam(S, 0)
static void mdlInitializeSizes(SimStruct *S){
DECL_AND_INIT_DIMSINFO(outputDimsInfo);
ssSetNumSFcnParams(S, NPARAMS); /* Number of expected parameters */
ssSetSFcnParamTunable(S, 0, 0);
ssSetSFcnParamTunable(S, 1, 0);
#if defined(MATLAB_MEX_FILE) if (ssGetNumSFcnParams(S) == ssGetSFcnParamsCount(S)) {
mdlCheckParameters(S);
if (ssGetErrorStatus(S) != NULL) {
return;
} } else {
return; }
#endif
ssSetNumContStates(S, 0);
ssSetNumDiscStates(S, 0);
if (!ssSetNumInputPorts(S, NUM_INPUTS)) return;
if (!ssSetNumOutputPorts(S, NUM_OUTPUTS)) return;
/* Output Port 0 */
ssSetOutputPortWidth(S, 0, 1);
ssSetOutputPortDataType(S, 0, SS_UINT8);
ssSetNumSampleTimes(S, 1);
ssSetNumRWork(S, 0);
ssSetNumIWork(S, 0);
ssSetNumPWork(S, 0);
ssSetNumModes(S, 0);
ssSetNumNonsampledZCs(S, 0);
}
static void mdlInitializeSampleTimes(SimStruct *S){
ssSetSampleTime(S, 0, -1);
ssSetOffsetTime(S, 0, 0.0);
}
static void mdlOutputs(SimStruct *S, int_T tid){
ssPrintf("DVC_IND_NO = %x\n",*mxGetPr(DVC_IND_NO(S)));
}
mdlRTW (SimStruct *S){
}
Before getting anything through the outputport, I would like to be able to print with ssPrintf("DVC_IND_NO = %x\n",*mxGetPr(DVC_IND_NO(S))); the 123.
Thank you, best regards.
mxGetPr returns a pointer to double, so makes no sense if the parameter is a char array. And in using *mxGetPr you are saying that you want the value that is pointed to (i.e. the first element, if it is a multi-element double array), which is why you are only getting the first value.
If the input is expected to be a string (i.e. array of char) then you should be using mxArrayToString to access it.
Either way, you should definitely be using helper functions like mxIsChar and mxIsDouble to check the data type before using the appropriate function to access the data.
Add the following at the top
#define DVC_IND_NO(S) = ssGetSFcnParam(S,0)
and the follwoing in your mdlOutputs
char* str;
int i = 0;
str = mxArrayToString(DVC_IND_NO(S));
ssPrintf("DVC_IND_NO = ");
while (str[i]!='\0')
{
ssPrintf("%c",str[i]);
i++;
}
We are using titan db to store the graph infomation. we have cassandra + es as backend storage and index. We are trying to load the graph data to represent the graph in the webui.
This is the approach i am following.
public JSONObject getGraph(long vertexId, final int depth) throws Exception {
JSONObject json = new JSONObject();
JSONArray vertices = new JSONArray();
JSONArray edges = new JSONArray();
final int currentDepth = 0;
TitanGraph graph = GraphFactory.getInstance().getGraph();
TitanTransaction tx = graph.newTransaction();
try {
GraphTraversalSource g = tx.traversal();
Vertex parent = graphDao.getVertex(vertexId);
loadGraph(g, parent, currentDepth + 1, depth, vertices, edges);
json.put("vertices", vertices);
json.put("edges", edges);
return json;
} catch (Throwable e) {
log.error(e.getMessage(), e);
if (tx != null) {
tx.rollback();
}
throw new Exception(e.getMessage(), e);
} finally {
if (tx != null) {
tx.close();
}
}
}
private void loadGraph(final GraphTraversalSource g, final Vertex vertex, final int currentDepth,
final int maxDepth, final JSONArray vertices, final JSONArray edges) throws Exception {
vertices.add(toJSONvertex));
List<Edge> edgeList = g.V(vertex.id()).outE().toList();
if (edgeList == null || edgeList.size() <= 0) {
return;
}
for (Edge edge : edgeList) {
Vertex child = edge.inVertex();
edges.add(Schema.toJSON(vertex, edge, child));
if (currentDepth < maxDepth) {
loadGraph(g, child, currentDepth + 1, maxDepth, vertices, edges);
}
}
}
But this is taking a bit lot of time for the depth 3 when we have more number of nodes exist in the tree it is taking about 1 min to load the data.
Please help me are there any better mechanisms to load the graph efficiently?
You might see better performance performing your full query across a single traversal execution - e.g., g.V(vertex.id()).outV().outV().outE() for depth 3 - but any vertices with very high edge cardinality are going to make this query slow no matter how you execute it.
To add to the answer by #Benjamin performing one traversal as opposed to many little ones which are constantly expanding will indeed be faster. Titan uses lazy loading so you should take advantage of that.
The next thing I would recommend is to also multithread each of your traversals and writes. Titan actually supports simultaneous writes very nicely. You can achieve this using Transactions.
I would like some help to optimize this code. In the part where I
initialize all the ArrayLists, it feels insufficient that I have to set
them to default values for the size of the path ArrayList, since I am only
interested for the path from the source node to the destination node. Do you
guys have any idea on how to fix this?
Cheers!
//Shortest path algorithm
public void shortestPath(String source, String end){
/*
* Init the vertices, init the distances
* Set the distance from the start node to itself to 0
*/
copy = g.getMap();
initArrays(copy);
holdCost.add(agent.get(source), 0);
pq.insert(new Vertex(source, 0));
while(!pq.isEmpty()){
Vertex v = pq.getRoot();
pq.delete(pq.getRoot());
visited.set(agent.get(v), true);
for(Edge e : copy.get(v)){
if(!visited.get(agent.get(e))){
if(holdCost.get(agent.get(e)) > (holdCost.get(agent.get(v))
+ e.getWeight())){
holdCost.set(agent.get(e), (holdCost.get(agent.get(v))
+ e.getWeight()));
pq.insert(new Vertex(path.get(agent.get(e)),
holdCost.get(agent.get(e))));
cost.add(holdCost.get(agent.get(e)));
}if(visited.get(agent.get(end))){
break;
}
}
}
}
}
//Initiliaze of distance, nodes etc
public void initArrays(HashMap<String, ArrayList<Edge>> h){
for(String s : h.keySet()){
path.add(s);
holdCost.add(path.indexOf(s), Integer.MAX_VALUE);
visited.add(path.indexOf(s), false);
agent.put(s, path.indexOf(s)); //easy to get the same index as vertices, distances etc
}
}