How does one use type hints for ValueProvider value types passed to PTransform and DoFn classes?
class MyPTransform(beam.PTransform):
def __init__(self, my_value_provider: ValueProvider):
# How do I enforce my_value_provider has value_type of str
self.my_value_provider = my_value_provider
I can make this a RuntimeValueProvider or StaticValueProvider and test this explicitly:
type(my_value_provider.type) == str
How do others do this? I didn't see anything here: https://beam.apache.org/documentation/sdks/python-type-safety
I don't think there is a way to enforce this with python's type checking, although you could always add your own runtime type check to potentially improve error messages.
Alternatively, you can avoid having to use ValueProvider at all by using Flex Templates instead.
I ended up creating a wrapper value provider like so, opinions/comments welcome.
from typing import Generic, TypeVar
from apache_beam.options.value_provider import ValueProvider, StaticValueProvider, RuntimeValueProvider
T = TypeVar('T')
class TypedValueProvider(ValueProvider, Generic[T]):
"""
Providers a wrapper around beam's ValueProvider to allow specifying value type hints
"""
def __init__(self, value_provider: ValueProvider, value_type: Type[T]):
self.value_provider = value_provider
self.value_type = value_type
assert value_provider.value_type == value_type
def is_accessible(self):
return self.value_provider.is_accessible()
def get(self):
return self.value_provider.get()
if __name__ == '__main__':
svp = StaticValueProvider(str, 'blah')
dvp: TypedValueProvider[str] = TypedValueProvider(svp, str)
print(isinstance(dvp, ValueProvider))
assert 'blah' == dvp.get()
rvp = RuntimeValueProvider('key', str, 'default')
RuntimeValueProvider.set_runtime_options({'key': 'value'})
dvp: TypedValueProvider[str] = TypedValueProvider(rvp, str)
print(isinstance(dvp, ValueProvider))
assert 'value' == dvp.get()
I am new to object orientation, and I am having troubles understanding the following:
import torch.nn as nn
class mynet(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(20, 64)
def forward(self, x):
x = self.fc1(x)
The line self.fc1 = nn.Linear(20, 64) is supposed to create a member variable fc1 to my class, right? But what is the return value of nn.Linear(20, 64)?
According to the documentation, nn.Linear is defined as
class torch.nn.Linear(in_features: int, out_features: int, bias: bool = True).
However, in my basic OOP tutorial I have only seen something like class CLASSNAME(BASECLASS) so that the class CLASSNAME inherits from BASECLASS. What does the documentation mean with its way of writing all that stuff in between the brackets?
Also, the line x=fc1(x) somehow makes it look as if fc1 was a function now.
I seem to lack OOP knowledge here... Any help appreciated!
First lets take a look at this
self.fc1 = nn.Linear(20, 64)
This part is probably familiar to someone with a basic understanding of python and OOP. Here we are simply creating a new instance of an nn.Linear class and initializing the class using positional arguments 20 and 64 corresponding to in_features and out_features respectively. The arguments in the documentation are the expected arguments to be passed to nn.Linear's __init__ method.
Now for the part that's probably a little more confusing
x = self.fc1(x)
The nn.Linear class is a callable since it's parent class, nn.Module, implements a special method named __call__. That means you can treat self.fc1 like a function and do things like x = self.fc1(x), which is equivalent to x = self.fc1.__call__(x).
You can create a little examination:
import torch
import torch.nn as nnn
fc1 = nn.Linear(20, 64)
print(fc1, type(fc1))
ret = fc1(torch.randn(20))
print(ret, type(ret), ret.shape)
Out:
Linear(in_features=20, out_features=64, bias=True) <class 'torch.nn.modules.linear.Linear'>
tensor([-0.2795, 0.8476, -0.8207, 0.3943, 0.1464, -0.2174, 0.6605, 0.6072,
-0.6881, -0.1118, 0.8226, 0.1515, 1.3658, 0.0814, -0.8751, -0.9587,
0.1310, 0.2539, -0.3072, -0.0225, 0.4663, -0.0019, 0.0404, 0.9279,
0.4948, -0.3420, 0.9061, 0.1752, 0.1809, 0.5917, -0.1010, -0.3210,
1.1910, 0.5145, 0.2254, 0.2077, -0.0040, -0.6406, -0.1885, 0.5270,
0.0824, -0.0787, 1.5140, -0.7958, 1.1727, 0.1862, -1.0700, 0.0431,
0.6849, 0.1393, 0.7547, 0.0917, -0.3264, -0.2152, -0.0728, -0.6441,
-0.1162, 0.4154, 0.3486, -0.1693, 0.6697, 0.0229, 0.0311, 0.1433],
grad_fn=<AddBackward0>) <class 'torch.Tensor'> torch.Size([64])
fc1 is of type class 'torch.nn.modules.linear.Linear'.
It needs some "juice" to work. In your case it needs the input tensor torch.randn(20) to return the output of torch.Size([64]).
So fc1 is a class instance that you can run with () in which case the forward() method of a class nn.Linear will be called.
In most cases when working with your modules (like mynet in your case) you will list the modules in __init__, and then in the forward of your module you will be defining what will happen (the behavior).
The three kind of modules in PyTorch are:
Functional modules
Default modules
Custom modules
Custom modules like mynet you created typically use default modules:
nn.Identity()
nn.Embedding()
nn.Linear()
nn.Conv2d()
nn.BatchNorm() (BxHxW)
nn.LayerNorm() (CxHxW)
nn.Dropout()
nn.ReLU()
And many many other modules that I haven't set. But of course, you can create custom modules without any default modules, just by using nn.Parameter(), see the last example.
The third kind functional modules are defined here.
Also check nn.Linear implementation. You may note the F.linear() functional module is used.
You may test the naive implementation of Linear from Fastai Book:
import torch
import torch.nn as nn
import math
class Linear(nn.Module):
def __init__(self, n_in, n_out):
super().__init__()
self.weight = nn.Parameter(torch.randn(n_out, n_in) * math.sqrt(2/n_in))
self.bias = nn.Parameter(torch.zeros(n_out))
def forward(self, x): return x # self.weight.T + self.bias
fc = Linear(20,64)
ret = fc(torch.randn(20))
print(ret.shape) # 64
You may try to understand the difference between the naive implementation provided inside PyTorch.
As a newbie to object oriented, I'm practicing class using this:
import numpy as np
class complex():
def __init__(self, arg1, arg2 = None):
self.real = arg1
if arg2 == None:
self.imag = 0
else:
self.imag = arg2
self.num = np.complex(self.real, self.imag)
def __mul__(self, other):
if np.iscomplex(other):
return np.multiply(self.numb, other.num)
else:
return np.multiply(self.num, other)
def __str__(self):
return "Foo"
def test():
a = complex(3)
b = complex(1, 5)
print(a * b)
if __name__ == "__main__":
test()
I'm trying to overload the operator * simultaneously oveloading the class method __mul__ since I got different permutations of arguments passing in. The above code can do (object, object) or (object, 'int').
How to pass self as the second argument something like ('int', object)?
Things I tried from my basic understanding: (correct me if I'm wrong)
Static method with self as second argument, but inbuilt operators cannot be overloaded in a static method.
Overloading the method __mul__, but not sure on a good-pratice/right-way to force self as second argument.
Appreciate any help in advance. Please comment below if you need more info.
I'm trying to extend the rv_discrete scipy class, as it is supposed to work in every case while extending a class.
I just want to add a couple of instance attributes.
from scipy.stats import rv_discrete
class Distribution(rv_discrete):
def __init__(self, realization):
self._realization = realization
self.num = len(realization)
#stuff to obtain random alphabet and probabilities from realization
super().__init__(values=(alphabet,probabilities))
This should allow me to do something like this :
realization = #some values
dist = Distribution(realization)
print(dist.mean())
Instead, I receive this error
ValueError: rv_discrete.__init__(..., values != None, ...)
If I simply create a new rv_discrete object as in the following line of code
dist = rv_discrete(values=(alphabet,probabilities))
It works just fine.
Any idea why? Thank you for your help
I am getting an error....and I know what I'm doing wrong, but not sure how to fix it. I understand I can't add a string to a integer...Any ideas, I'd be grateful!
self.variables['gas'] = 'gas'+add
TypeError: Can't convert 'int' object to str implicitly
My code:
class Cars:
def __init__(self, **kwargs):
self.variables = kwargs
def set_Variable(self, k, v):
self.variables[k] = v
def get_Variable(self, k):
return self.variables.get(k, None)
def add_gas(self, add, gas):
self.variables['gas'] = gas+add
def main():
mercedes = Cars(gas = 3)
print (mercedes.get_Variable('gas'))
print(mercedes.add_gas(4))
if __name__ == "__main__": main()
Your method add_gas needs two arguments, you're only passing one to the function. Also you are printing the returning value though there is no return in the function.
This one worked for me:
class Cars:
def __init__(self, **kwargs):
self.variables = kwargs
def set_Variable(self, k, v):
self.variables[k] = v
def get_Variable(self, k):
return self.variables[k]
def add_gas(self, add):
self.variables['gas'] +=add
def main():
mercedes = Cars(gas = 3)
print (mercedes.get_Variable('gas'))
mercedes.add_gas(4)
print (mercedes.get_Variable('gas'))
print ('There are %s gallons of gas in the car.' % mercedes.get_Variable('gas'))
if __name__ == "__main__":
main()
So, I see two independent problems with your code:
In your error message you showed this line:
self.variables['gas'] = 'gas'+add
While in your code, the line is this:
self.variables['gas'] = gas+add
That are two different meanings. The first tries to concat the variable add to the string 'gas' which will fail as add is an integer and Python won’t convert integers to strings implicitely (the error told you that). So you would want to do 'gas' + str(add).
In your main function, you call the add_gas method like this:
mercedes.add_gas(4)
But you actually defined the method with two parameters, add and gas, so this will fail.
What I think is what you want to do is to increase the self.variables['gas'] value by add when the method is called, so you would want to write your method like this:
def add_gas(self, add):
self.variables['gas'] = self.variables['gas'] + add
Note that you can shorten that using the += operator to just this:
def add_gas(self, add):
self.variables['gas'] += add
And finally, I don’t know why you would want to use a dictionary for your locale instance variables, but unless you have a good reason to do that, why not just define a gas property for your car type?