I am new to object orientation, and I am having troubles understanding the following:
import torch.nn as nn
class mynet(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(20, 64)
def forward(self, x):
x = self.fc1(x)
The line self.fc1 = nn.Linear(20, 64) is supposed to create a member variable fc1 to my class, right? But what is the return value of nn.Linear(20, 64)?
According to the documentation, nn.Linear is defined as
class torch.nn.Linear(in_features: int, out_features: int, bias: bool = True).
However, in my basic OOP tutorial I have only seen something like class CLASSNAME(BASECLASS) so that the class CLASSNAME inherits from BASECLASS. What does the documentation mean with its way of writing all that stuff in between the brackets?
Also, the line x=fc1(x) somehow makes it look as if fc1 was a function now.
I seem to lack OOP knowledge here... Any help appreciated!
First lets take a look at this
self.fc1 = nn.Linear(20, 64)
This part is probably familiar to someone with a basic understanding of python and OOP. Here we are simply creating a new instance of an nn.Linear class and initializing the class using positional arguments 20 and 64 corresponding to in_features and out_features respectively. The arguments in the documentation are the expected arguments to be passed to nn.Linear's __init__ method.
Now for the part that's probably a little more confusing
x = self.fc1(x)
The nn.Linear class is a callable since it's parent class, nn.Module, implements a special method named __call__. That means you can treat self.fc1 like a function and do things like x = self.fc1(x), which is equivalent to x = self.fc1.__call__(x).
You can create a little examination:
import torch
import torch.nn as nnn
fc1 = nn.Linear(20, 64)
print(fc1, type(fc1))
ret = fc1(torch.randn(20))
print(ret, type(ret), ret.shape)
Out:
Linear(in_features=20, out_features=64, bias=True) <class 'torch.nn.modules.linear.Linear'>
tensor([-0.2795, 0.8476, -0.8207, 0.3943, 0.1464, -0.2174, 0.6605, 0.6072,
-0.6881, -0.1118, 0.8226, 0.1515, 1.3658, 0.0814, -0.8751, -0.9587,
0.1310, 0.2539, -0.3072, -0.0225, 0.4663, -0.0019, 0.0404, 0.9279,
0.4948, -0.3420, 0.9061, 0.1752, 0.1809, 0.5917, -0.1010, -0.3210,
1.1910, 0.5145, 0.2254, 0.2077, -0.0040, -0.6406, -0.1885, 0.5270,
0.0824, -0.0787, 1.5140, -0.7958, 1.1727, 0.1862, -1.0700, 0.0431,
0.6849, 0.1393, 0.7547, 0.0917, -0.3264, -0.2152, -0.0728, -0.6441,
-0.1162, 0.4154, 0.3486, -0.1693, 0.6697, 0.0229, 0.0311, 0.1433],
grad_fn=<AddBackward0>) <class 'torch.Tensor'> torch.Size([64])
fc1 is of type class 'torch.nn.modules.linear.Linear'.
It needs some "juice" to work. In your case it needs the input tensor torch.randn(20) to return the output of torch.Size([64]).
So fc1 is a class instance that you can run with () in which case the forward() method of a class nn.Linear will be called.
In most cases when working with your modules (like mynet in your case) you will list the modules in __init__, and then in the forward of your module you will be defining what will happen (the behavior).
The three kind of modules in PyTorch are:
Functional modules
Default modules
Custom modules
Custom modules like mynet you created typically use default modules:
nn.Identity()
nn.Embedding()
nn.Linear()
nn.Conv2d()
nn.BatchNorm() (BxHxW)
nn.LayerNorm() (CxHxW)
nn.Dropout()
nn.ReLU()
And many many other modules that I haven't set. But of course, you can create custom modules without any default modules, just by using nn.Parameter(), see the last example.
The third kind functional modules are defined here.
Also check nn.Linear implementation. You may note the F.linear() functional module is used.
You may test the naive implementation of Linear from Fastai Book:
import torch
import torch.nn as nn
import math
class Linear(nn.Module):
def __init__(self, n_in, n_out):
super().__init__()
self.weight = nn.Parameter(torch.randn(n_out, n_in) * math.sqrt(2/n_in))
self.bias = nn.Parameter(torch.zeros(n_out))
def forward(self, x): return x # self.weight.T + self.bias
fc = Linear(20,64)
ret = fc(torch.randn(20))
print(ret.shape) # 64
You may try to understand the difference between the naive implementation provided inside PyTorch.
Related
unclear to me if this is in fact the same question as here or here, apologies if this is a duplicate.
i would like to define a type Ordinate which is simply an Int under-the-hood:
package world
opaque type Ordinate = Int
given Ordering[Ordinate] with {
def compare(x: Ordinate, y: Ordinate): Int = x.compare(y)
}
i would like to be able to leverage the Numeric[Int] and Ordering[Int] methods so that it would be easy to define methods such as
package world
import Ordinate.given
class Boundary(dims: List[(Ordinate, Ordinate)]) {
def contains(o: Ordinate, dimension: Int): Boolean = {
val (min, max) = dims(dimension)
min <= o && o <= max
}
}
...forgetting for the meantime that this would blow up if dims was empty, dimension < 0 or dims.length <= dimension.
when i try and set this up, i get compiler errors at the call site:
value <= is not a member of world.Ordinate, but could be made available as an extension method.
One of the following imports might fix the problem:
import world.given_Ordering_Ordinate.mkOrderingOps
import math.Ordering.Implicits.infixOrderingOps
import math.Ordered.orderingToOrdered
more generally, it would be wicked cool if this were the case without any special given imports for files in the same package as Ordinate and even better, across the codebase. but that may be an anti-pattern that i've carried forward from my Scala 2 coding.
explicit given imports may be a better pattern but i'm still learning Scala 3 from Scala 2 here. i know if i created an implicit val o = Ordering.by(...) in the companion object of Ordinate in Scala 2, with Ordinate as a value class, i would get the effect i'm looking for (zero-cost type abstraction + numeric behaviors).
anyhow, i'm guessing i'm just missing a small detail here, thank you for reading and for any help.
Scala 3 has revised the rules for infix operators so that the author must (explicitly) expose infix operations such as x: T <= y: T for some custom type T.
I've found two ways to address this for an opaque type, both with drawbacks:
at the call site, have import math.Ordering.Implicits.infixOrderingOps in scope, which brings in a given instance that converts Ordering[T] into infix comparators. drawback: any file that wants these comparators needs the import line, adding more import boilerplate as the number of files using this opaque type increases.
package world
import Ordinate.given
import math.Ordering.Implicits.infixOrderingOps // <-- add this line
class Boundary(dims: List[(Ordinate, Ordinate)]) {
def contains(o: Ordinate, dimension: Int): Boolean = {
val (min, max) = dims(dimension)
min <= o && o <= max
}
}
add an infix extension method for each comparator you want to expose. drawback here is boilerplate of having to write out the very thing we're trying not to duplicate in each file.
type Ordinate = Int
object Ordinate {
extension (o: Ordinate) {
infix def <=(x: Ordinate): Boolean = o <= x // <-- add 'infix' here
}
}
i'm guessing for those more experienced with large programs, these drawbacks are better than the drawbacks associated with anything more than this least permission approach to givens. but this still doesn't seem to deliver on the promise of opaque types as a zero-cost abstraction for numeric types. what seems to be missing is something like "import a given and treat it's methods as infix for my type".
I am fitting a number of models to several different datasets. I would like to store the methods and attributes of the datasets (e.g. X, y, trainTestSplit(), etc.) in 'Dataset' objects, and store the the methods and attributes of the models (e.g. fit(), hyperparameters, scores, etc.) in 'Model' objects, and store the 'Models' in the 'Datasets' (several Models for each Dataset).
I have tried several different ways to make this work, including inheritance with the use of super(); indenting (nesting) the Model class inside the dataset class; and with functions in the Dataset class which can be called by the Model class.
This is about as close as I've come:
class Dataset :
def __init__(self, X, y, attr) :
self.X = X
self.y = y
self.attr = attr
def trainTestSplit(self, **kwargs):
self.X_train, self.X_test, self.y_train, self.y_test = train_test_split(self.X, self.y, **kwargs)
class Model(Dataset) :
def __init__(self, regressor):
self.reg = regressor
super().__init__(self)
def fit(self):
self.reg.fit(X=self.X_train, y=self.y_train)
def predict(self):
self.yPredict = self.reg.predict(X=self.X_test)
In the above, the variables are not accessible to the Model class, and so it returns and error.
If this worked, I would expect to be executing the following function calls, as examples.
Creating an instance of a Dataset object:
dataset_1 = Dataset(X, y, 'string')
Splitting data into training and test sets:
dataset_1.trainTestSplit(test_size=0.3))
Creating an instance of a Model, and applying it to a dataset:
dataset_1.svr = Model(SVR(hyperParams))
Fitting a model:
dataset_1.svr.fit()
Actually, if I'm thinking about this in the right way, the fit() method of the Model class could be applied as part of the init, so that it is fit to the dataset on instantiation.
Reading a training score:
dataset_1.svr.training_score_
Because I have so many datasets, and will be fitting numerous models to each, having the methods and attributes stored in this way seemed to make sense, but I'm not sure how to implement it.
Is there any way to instantiate a class (call it a sub-class), such that it inherits or has access to the attributes or objects contained in another class (say a root class), and so the sub-class is contained as an object within the root class? Or am I thinking about this in the wrong way?
So I might have been trying to use classes where a dictionary would be better suited.
The following seems to accomplish what I'm trying to do, which is to store a dataset and the results of models applied to it in the one object (other attributes such as training/CV/test scores etc. are still to be added in the answer below).
If anyone has any suggestions/comments on how this can be done better please comment.
Thanks.
from sklearn.model_selection import train_test_split
class Dataset(object):
def __init__(self, X, y):
self.X = X
self.y = y
self.models = {}
def add_model(self, model, regressor):
self.models[model] = {}
self.models[model]['reg'] = regressor
regressor.fit(self.X_train, self.y_train)
yPredict = regressor.predict(self.X_test)
self.models[model]['yPredict'] = yPredict
def trainTestSplit(self, **kwargs):
self.X_train, self.X_test, self.y_train, self.y_test = train_test_split(self.X, self.y, **kwargs)
I'm trying to extend the rv_discrete scipy class, as it is supposed to work in every case while extending a class.
I just want to add a couple of instance attributes.
from scipy.stats import rv_discrete
class Distribution(rv_discrete):
def __init__(self, realization):
self._realization = realization
self.num = len(realization)
#stuff to obtain random alphabet and probabilities from realization
super().__init__(values=(alphabet,probabilities))
This should allow me to do something like this :
realization = #some values
dist = Distribution(realization)
print(dist.mean())
Instead, I receive this error
ValueError: rv_discrete.__init__(..., values != None, ...)
If I simply create a new rv_discrete object as in the following line of code
dist = rv_discrete(values=(alphabet,probabilities))
It works just fine.
Any idea why? Thank you for your help
I have a pytorch custom layer defined as:
class MyCustomLayer(nn.Module):
def __init__(self):
super(MyCustomLayer, self).__init__()
self.my_parameter = torch.rand(1, requires_grad = True)
# the following allows the previously defined parameter to be recognized as a network parameter when instantiating the model
self.my_registered_parameter = nn.ParameterList([nn.Parameter(self.my_parameter)])
def forward(self, x):
return x*self.my_parameter
I then define my network where the custom layer is used:
class MyNet(nn.Module):
def __init__(self):
super(MyNet, self).__init__()
self.layer1 = MyCustomLayer()
def forward(self, x):
x = self.layer1(x)
return x
Now Let's instantiate MyNet and observe the issue:
# instantiate MyNet and run it over one input value
model = MyNet()
x = torch.tensor(torch.rand(1))
output = model(x)
criterion = nn.MSELoss()
loss = criterion(1, output)
loss.backward()
Iterating through model parameters shows None for custom layer parameter:
for p in model.parameters():
print (p.grad)
None
while directly accessing that parameter shows the correct grad value:
print(model.layer1.my_parameter.grad)
tensor([-1.4370])
This, in turn, prevents the optim step from updating the inner parameters automatically and leaves me with the hassle of having to update those manually. Anyone knows how I can address this issue?
What you did i.e. return x*self.my_registered_parameter[0] worked because you use the registered param for calculating the gradient.
When you call nn.Parameter it returns a new object and hence self.my_parameter that you use for the operation and the one registered are not same.
You can fix this by declaring the my_parameter as nn.Parameter
self.my_parameter = nn.Parameter(torch.rand(1, requires_grad = True))
self.my_registered_parameter= nn.ParameterList([self.some_parameter])
or you don't need to create my_registered_parameter variable at all. When you declare self.my_parameter as nn.Parameter it gets registered as a parameter.
Alright!
I had to switch the parameter variable calls within the custom layer to the nn.ParameterList object (i.e. return x*self.my_registered_parameter[0] instead of x*self.my_parameter ). In this example that meant changing the custom layer's parameter call in forward method to:
def forward(self, x):
return x*self.my_registered_parameter[0]
This is where it would've been nice to have pass by reference!
Now optim updates all the parameters as expected!
I have a group of objects which I am creating a class for that I want to store each object as its own text file. I would really like to store it as a Python class definition which subclasses the main class I am creating. So, I did some poking around and found a Python Code Generator on effbot.org. I did some experimenting with it and here's what I came up with:
#
# a Python code generator backend
#
# fredrik lundh, march 1998
#
# fredrik#pythonware.com
# http://www.pythonware.com
#
# Code taken from http://effbot.org/zone/python-code-generator.htm
import sys, string
class CodeGeneratorBackend:
def begin(self, tab="\t"):
self.code = []
self.tab = tab
self.level = 0
def end(self):
return string.join(self.code, "")
def write(self, string):
self.code.append(self.tab * self.level + string)
def indent(self):
self.level = self.level + 1
def dedent(self):
if self.level == 0:
raise SyntaxError, "internal error in code generator"
self.level = self.level - 1
class Point():
"""Defines a Point. Has x and y."""
def __init__(self, x, y):
self.x = x
self.y = y
def dump_self(self, filename):
self.c = CodeGeneratorBackend()
self.c.begin(tab=" ")
self.c.write("class {0}{1}Point()\n".format(self.x,self.y))
self.c.indent()
self.c.write('"""Defines a Point. Has x and y"""\n')
self.c.write('def __init__(self, x={0}, y={1}):\n'.format(self.x, self.y))
self.c.indent()
self.c.write('self.x = {0}\n'.format(self.x))
self.c.write('self.y = {0}\n'.format(self.y))
self.c.dedent()
self.c.dedent()
f = open(filename,'w')
f.write(self.c.end())
f.close()
if __name__ == "__main__":
p = Point(3,4)
p.dump_self('demo.py')
That feels really ugly, is there a cleaner/better/more pythonic way to do this? Please note, this is not the class I actually intend to do this with, this is a small class I can easily mock up in not too many lines. Also, the subclasses don't need to have the generating function in them, if I need that again, I can just call the code generator from the superclass.
We use Jinja2 to fill in a template. It's much simpler.
The template looks a lot like Python code with a few {{something}} replacements in it.
This is pretty much the best way to generate Python source code. However, you can also generate Python executable code at runtime using the ast library. You can build code using the abstract syntax tree, then pass it to compile() to compile it into executable code. Then you can use eval() to run the code.
I'm not sure whether there is a convenient way to save the compiled code for use later though (ie. in a .pyc file).
Just read your comment to wintermute - ie:
What I have is a bunch of planets that
I want to store each as their own text
files. I'm not particularly attached
to storing them as python source code,
but I am attached to making them
human-readable.
If that's the case, then it seems like you shouldn't need subclasses but should be able to use the same class and distinguish the planets via data alone. And in that case, why not just write the data to files and, when you need the planet objects in your program, read in the data to initialize the objects?
If you needed to do stuff like overriding methods, I could see writing out code - but shouldn't you just be able to have the same methods for all planets, just using different variables?
The advantage of just writing out the data (it can include label type info for readability that you'd skip when you read it in) is that non-Python programmers won't get distracted when reading them, you could use the same files with some other language if necessary, etc.
I'm not sure whether this is especially Pythonic, but you could use operator overloading:
class CodeGenerator:
def __init__(self, indentation='\t'):
self.indentation = indentation
self.level = 0
self.code = ''
def indent(self):
self.level += 1
def dedent(self):
if self.level > 0:
self.level -= 1
def __add__(self, value):
temp = CodeGenerator(indentation=self.indentation)
temp.level = self.level
temp.code = str(self) + ''.join([self.indentation for i in range(0, self.level)]) + str(value)
return temp
def __str__(self):
return str(self.code)
a = CodeGenerator()
a += 'for a in range(1, 3):\n'
a.indent()
a += 'for b in range(4, 6):\n'
a.indent()
a += 'print(a * b)\n'
a.dedent()
a += '# pointless comment\n'
print(a)
This is, of course, far more expensive to implement than your example, and I would be wary of too much meta-programming, but it was a fun exercise. You can extend or use this as you see fit; how about:
adding a write method and redirecting stdout to an object of this to print straight to a script file
inheriting from it to customise output
adding attribute getters and setters
Would be great to hear about whatever you go with :)
From what I understand you are trying to do, I would consider using reflection to dynamically examine a class at runtime and generate output based on that. There is a good tutorial on reflection (A.K.A. introspection) at http://diveintopython3.ep.io/.
You can use the dir() function to get a list of names of the attributes of a given object. The doc string of an object is accessible via the __doc__ attribute. That is, if you want to look at the doc string of a function or class you can do the following:
>>> def foo():
... """A doc string comment."""
... pass
...
>>> print foo.__doc__
A doc string comment.