Lihaoyi PPrint Deep Tree - scala
title: Lihaoyi PPrint Deep Tree
link: Lihaoyi PPrint Deep Tree
I have a "deep" tree of case classes that I want to pprint as "like Scala" source code.
... because I want to view, copy and paste them to make a UnitTest
I am using lihaoyi's pprint to perform the stringify of the values, BUT after a few levels, pprint stops printing.
I've checked in the tokenize thing and the instance is being emitted as-is; you can see that the strings aren't being excaped.
I can (manually) "tweak" my tests to test parts of the results and get my pretty source, but ... it feels wrong.
Original "Lack of deep" printout
Module(
Set(),
Set(
Material(
"testTextLayoutContainer",
List(
Attribute("Position", VecSingle(2)),
Attribute("UV", VecSingle(2)),
Attribute("Color", VecFixed(4))
),
List(
UniformInstance("ProjMtx", MatSingle(4, 4)),
UniformInstance("Texture", Texture(2, VecFixed(4)))
),
Set(
//
// this is the line that's not being pprinted
//
Program(ImGui,testTextLayoutContainer,Set(Attribute(Color,VecFixed(4)), Attribute(Position,VecSingle(2)), Attribute(UV,VecSingle(2))),Set(UniformInstance(Texture,Texture(2,VecFixed(4))), UniformInstance(ProjMtx,MatSingle(4,4))),Set(VaryingCopy(Attribute(Color,VecFixed(4))), VaryingCopy(Attribute(UV,VecSingle(2)))),List(Lookup(UniformInstance(ProjMtx,MatSingle(4,4))), Lookup(Attribute(Position,VecSingle(2))), Literal(0.0), Literal(1.0), Construct(VecSingle(4),List(Lookup(Attribute(Position,VecSingle(2))), Literal(0.0), Literal(1.0))), External(mat*vec,List(Lookup(UniformInstance(ProjMtx,MatSingle(4,4))), Construct(VecSingle(4),List(Lookup(Attribute(Position,VecSingle(2))), Literal(0.0), Literal(1.0)))),VecSingle(4)), Output(Vertex(ProductTransform(Lookup(UniformInstance(ProjMtx,MatSingle(4,4))),Construct(VecSingle(4),List(Lookup(Attribute(Position,VecSingle(2))), Literal(0.0), Literal(1.0))))),External(mat*vec,List(Lookup(UniformInstance(ProjMtx,MatSingle(4,4))), Construct(VecSingle(4),List(Lookup(Attribute(Position,VecSingle(2))), Literal(0.0), Literal(1.0)))),VecSingle(4))), Lookup(Attribute(Color,VecFixed(4))), Output(VaryingCopy(Attribute(Color,VecFixed(4))),Lookup(Attribute(Color,VecFixed(4)))), Lookup(Attribute(UV,VecSingle(2))), Output(VaryingCopy(Attribute(UV,VecSingle(2))),Lookup(Attribute(UV,VecSingle(2))))),List(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))), Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),x), Lookup(UniformInstance(Texture,Texture(2,VecFixed(4)))), Lookup(VaryingCopy(Attribute(UV,VecSingle(2)))), Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),x), External(*,List(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),x), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),x)),VecFixed(1)), Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),y), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),y), External(*,List(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),y), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),y)),VecFixed(1)), Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),z), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),z), External(*,List(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),z), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),z)),VecFixed(1)), Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),w), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),w), External(*,List(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),w), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),w)),VecFixed(1)), Construct(VecFixed(4),List(External(*,List(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),x), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),x)),VecFixed(1)), External(*,List(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),y), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),y)),VecFixed(1)), External(*,List(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),z), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),z)),VecFixed(1)), External(*,List(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),w), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),w)),VecFixed(1)))), Output(Fragment(Construct(VecFixed(4),List(ProductAtomic(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),x),Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),x)), ProductAtomic(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),y),Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),y)), ProductAtomic(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),z),Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),z)), ProductAtomic(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),w),Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),w))))),Construct(VecFixed(4),List(External(*,List(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),x), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),x)),VecFixed(1)), External(*,List(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),y), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),y)),VecFixed(1)), External(*,List(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),z), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),z)),VecFixed(1)), External(*,List(Access(Lookup(VaryingCopy(Attribute(Color,VecFixed(4)))),w), Access(Sample(UniformInstance(Texture,Texture(2,VecFixed(4))),Lookup(VaryingCopy(Attribute(UV,VecSingle(2))))),w)),VecFixed(1)))))))
)
)
)
)
You can provide a height and a width, so if you provide a big enough value it will show it all.
scala> pprint.pprintln(Seq(1, 2, 3), width = Int.MaxValue, height = Int.MaxValue)
so
# pprint.pprintln(1 to 10, 3, 10)
Range.Inclusive(
1,
2...
can be
# pprint.pprintln(1 to 10, Int.MaxValue, Int.MaxValue)
Range.Inclusive(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
Edit:
as mentioned in the docs this is a feature to stop out of memory errors when printing really big data structures.
http://www.lihaoyi.com/upickle-pprint/pprint/#GettingStarted
Related
TF Keras code adaptation from python2.7 to python3
I am working to adapt a python2.7 code that uses keras and tensorflow to implement a CNN but looks like the keras API has changed a little bit since when the original code was idealized. I keep getting an error about "Negative dimension after subtraction" and I can not find out what is causing it.
Unfortunately I am not able to provide an executable piece of code because I was not capable of make the original code works, but the repository containing all the source files can be found here.
The piece of code:
from keras.callbacks import EarlyStopping
from keras.layers.containers import Sequential
from keras.layers.convolutional import Convolution2D, MaxPooling2D
from keras.layers.core import Reshape, Flatten, Dropout, Dense
from keras.layers.embeddings import Embedding
from keras.models import Graph
from keras.preprocessing import sequence
filter_lengths = [3, 4, 5]
self.model = Graph()
'''Embedding Layer'''
self.model.add_input(name='input', input_shape=(max_len,), dtype=int)
self.model.add_node(Embedding(
max_features, emb_dim, input_length=max_len), name='sentence_embeddings', input='input')
'''Convolution Layer & Max Pooling Layer'''
for i in filter_lengths:
model_internal = Sequential()
model_internal.add(
Reshape(dims=(1, self.max_len, emb_dim), input_shape=(self.max_len, emb_dim))
)
model_internal.add(Convolution2D(
nb_filters, i, emb_dim, activation="relu"))
model_internal.add(
MaxPooling2D(pool_size=(self.max_len - i + 1, 1))
)
model_internal.add(Flatten())
self.model.add_node(model_internal, name='unit_' + str(i), input='sentence_embeddings')
What I have tried:
m = tf.keras.Sequential()
m.add(tf.keras.Input(shape=(max_len, ), name="input"))
m.add(tf.keras.layers.Embedding(max_features, emb_dim, input_length=max_len))
filter_lengths = [ 3, 4, 5 ]
for i in filter_lengths:
model_internal = tf.keras.Sequential(name=f'unit_{i}')
model_internal.add(
tf.keras.layers.Reshape(( 1, max_len, emb_dim ), input_shape=( max_len, emb_dim ))
)
model_internal.add(
tf.keras.layers.Convolution2D(100, i, emb_dim, activation="relu")
)
model_internal.add(
tf.keras.layers.MaxPooling2D(pool_size=( max_len - i + 1, 1 ))
)
model_internal.add(
tf.keras.layers.Flatten()
)
m.add(model_internal)
I do not expect a complete solution, what I am really trying to understand is what is the cause to the following error:
Negative dimension size caused by subtracting 3 from 1 for '{{node conv2d_5/Conv2D}} = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], explicit_paddings=[], padding="VALID", strides=[1, 200, 200, 1], use_cudnn_on_gpu=true](Placeholder, conv2d_5/Conv2D/ReadVariableOp)' with input shapes: [?,1,300,200], [3,3,200,100].
PySNMP how to convert to dot oid
How to convert in python SNMPv2-SMI.enterprises.9.1.283 to dot oid? SNMPv2-SMI.enterprises.9.1.283 -> 1.3.6.1.4.1.9.1.283 Thanks, Alexey
You can get usufal data by: print(o.getLabel()) print(o.getMibNode()) print(o.getMibSymbol()) print(o.getOid()) print(o.prettyPrint())
pysnmp implementation to resolve mib names to OID
from pysnmp.smi import builder, view, rfc1902, error
mibBuilder = builder.MibBuilder()
mibView = view.MibViewController(mibBuilder)
mibVar = rfc1902.ObjectIdentity('SNMPv2-SMI', 'enterprises', 9, 1, 283)
mibVar.resolveWithMib(mibView)
print(mibVar.prettyPrint()) # prints SNMPv2-SMI::enterprises.9.1.283
print(tuple(mibVar)) # prints (1, 3, 6, 1, 4, 1, 9, 1, 283)
print(str(mibVar)) # prints 1.3.6.1.4.1.9.1.283
Reference
networkx: remove edge with specific attribute from multigraph
I'd like to remove a specific edge (specific color) from a MultiGraph. How can I do that? Following code does not work. #!/usr/bin/env python import matplotlib.pyplot as plt import networkx as nx G = nx.MultiGraph() # the_colored_graph.add_edge(v1, v2, "red") G.add_edge(1, 2, color="red") G.add_edge(2, 3, color="red") G.add_edge(4, 2, color="green") G.add_edge(2, 4, color="blue") print (G.edges(data=True)) # G.remove_edge(2, 4, color="green") # selected_edge = [(u,v) for u,v,e in G.edges(data=True) if e['color'] == 'green'] print (selected_edge) G.remove_edge(selected_edge[0][0], selected_edge[0][1]) print (G.edges(data=True)) nx.draw(G) plt.show()
When constructing the multigraph, assign a "key" attribute to each edge (the key could be anything that disambiguates the parallel edges - say, the color): G.add_edge(1, 2, color="red", key='red') Remove an edges by specifying the end nodes and the key: G.remove_edge(1, 2, key='red')
Keras does not mach model with classes
I am new to Keras and I am trying to make a Neuronal Network to recognize 38 cases. I created such a model, but it just does not work. There is some problem with last layer I think. I checked summary and it looks like output of last layers is 38 as it should. Can someone help me with making it work?
My code is:
model = Sequential()
model.add(Convolution2D(16, 5, 5, border_mode='valid', input_shape=(168, 192, 3)) )
model.add( Activation('relu') )
model.add( MaxPooling2D(2,2) )
model.add( Convolution2D(16, 5, 5) )
model.add( Activation('relu') )
model.add( MaxPooling2D(2,2) )
model.add( Flatten() )
model.add( Dense(512, activation='relu'))
model.add(Dense(38, activation='softmax'))
model.compile(loss='categorical_crossentropy',optimizer=adam(0.001),metrics=['accuracy'])
train_datagen = ImageDataGenerator(
rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1./255)
train_data_dir = 'data/train'
validation_data_dir = 'data/validation'
train_datagen = ImageDataGenerator(
rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(
'data/train',
target_size=(168, 192),
batch_size=38,
class_mode='binary')
validation_generator = test_datagen.flow_from_directory(
'data/validation',
target_size=(168, 192),
batch_size=38,
class_mode='binary')
model.fit_generator(
train_generator,
steps_per_epoch=2000,
epochs=10,
validation_data=validation_generator,
validation_steps=800)
and the error looks like:
ValueError: Error when checking target: expected dense_129 to have shape (None, 38) but got array with shape (38, 1)
According to Keras documentation of from_from_directory, the specified directory ('data/train' in your case) should contain one subdirectory per class.
Since the error is saying the model is getting an array of shape (38, 1), this means you do not have 38 folders with data/train. (Note do not confuse that the first 38 dimension is the batch size, which coincidentally you have set it to same as number of classes, but does not have to be).
So you should either reaarange your subfolders into one class per subfolder, or load data manually, and flow from memory.
Cryptic python error 'classobj' object has no attribute '__getitem__'. Why am I getting this?
I really wish I could be more specific here but I have read through related questions and none of them seem to relate to the issue that I am experiencing here and I have no understanding of the issue i am experiencing. This is for a homework assignment so I am hesitant to put up all my code for the program, here is a stripped down version. Compile this and you will see the issue.
import copy
class Ordering:
def __init__(self, tuples):
self.pairs = copy.deepcopy(tuples)
self.sorted = []
self.unsorted = []
for x in self.pairs:
self.addUnsorted(left(x))
self.addUnsorted(right(x))
def addUnsorted(self, item):
isPresent = False
for x in self.unsorted:
if x == item:
isPresent = True
if isPresent == False:
self.unsorted.append(left(item))
Here I have created a class, Ordering, that takes a list of the form [('A', 'B'), ('C', 'B'), ('D', 'A')] (where a must come before b, c must come before b, etc.) and is supposed to return it in partial ordered form. I am working on debugging my code to see if it works correctly but I have not been able to yet because of the error message I get back.
When I input the follwing in my terminal:
print Ordering[('A', 'B'), ('C', 'B'), ('D', 'A')]
I get back the following error message:
Traceback (most recent call last): File "<stdin>", line 1, in (module) Type Error: 'classobj' object has no attribute '__getitem__'
Why is this?!
To access an element of a list, use square brackets. To instantiate a class, use parens.
In other words, do not use:
print Ordering[('A', 'B'), ('C', 'B'), ('D', 'A')]
Use:
print Ordering((('A', 'B'), ('C', 'B'), ('D', 'A')))
This will generate another error from deeper in the code but, since this is a homework assignment, I will let you think about that one a bit.
How to use __getitem__:
As a minimal example, here is a class that returns squares via __getitem__:
class HasItems(object):
def __getitem__(self, key):
return key**2
In operation, it looks like this:
>>> a = HasItems()
>>> a[4]
16
Note the square brackets.
Answer to "Why is this?"
Your demo-code is not complete ( ref. comment above ), however the issue with .__getitem__ method is clearly related with a statement to print an object ( which due to other reasons did fail to respond to a request to answer to a called .__getitem__ method ) rather than the Class itself.
>>> aList = [ ('A','B'), ('C','D'), ('E','F')] # the stated format of input
>>> aList # validated to be a list
[('A', 'B'), ('C', 'D'), ('E', 'F')]
>>> type( aList ) # cross-validated
<type 'list'>
>>> for x in aList: # iterator over members
... print x, type( x ) # show value and type
... left( x ) # request as in demo-code
...
('A', 'B') <type 'tuple'>
Traceback (most recent call last): <<< demo-code does not have it
File "<stdin>", line 3, in <module>
NameError: name 'left' is not defined
>>> dir( Ordering ) # .__getitem__ method missing
[ '__doc__', '__init__', '__module__', 'addUnsorted']
>>> dir( aList[0] ) # .__getitem__ method present
['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__',
'__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__',
'__getslice__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__len__',
'__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__',
'count', 'index']