Related
I use Python. According to the PEP 8 Style Guide, both of these are acceptable -
my_list = [
1, 2, 3,
4, 5, 6,
]
my_list = [
1, 2, 3,
4, 5, 6,
]
I prefer the former. However, if I copy and paste a block like this, VS Code changes the indentation like this -
my_list = [
1, 2, 3,
4, 5, 6,
]
my_list = [
1, 2, 3,
4, 5, 6,
]
Is there a setting to change this behaviour?
I've got this segment of code in a discriminator network for MNIST:
nn.Conv2d(1, 64, 4, 2, 1),
From my understanding, there is 1 input channel (the MNIST image), then we apply a 4x4 kernel to the image in strides of 2 to produce 64 feature maps. Does this mean that we actually have 64 kernels at this layer? Because in order to get 64 different feature maps, we would need 64 separate kernels to convolve over the image?
Then after some ReLu, we have another convolution:
nn.Conv2d(64, 128, 4, 2, 1),
How do we get from 64 to 128? From my understanding of the first example, we have 64 seperate kernels that can produce 64 seperate feature maps. But here we go from 64 feature maps to 128 feature maps? Does that mean that we only have two kernels?
I hope someone can shine some light on whether my understanding is correct!
Your understanding in the first example is correct, you have 64 different kernels to produce 64 different feature maps.
In case of the second example, so the number of input channels not beeing one, you still have as "many" kernels as the number of output feature maps (so 128), which each are trained on a linear combination of the input feature maps. So in your case each of these kernels would have 4x4x64 trainable weights.
All the input channels are connected to each output channel (if group = 1, as by default) by convolution with filters (kernels) -- one for each output channel. Each kernel though has sub-kernels for each input channel.
So in the first layer you have in_channels = 1 and out_channels = 64 meaning that there are 64 kernels (and sub-kernels). In the second layer you have in_channels = 64 and out_channels = 128 meaning that there are 128 kernels each having 64 * 128 sub-kernels.
Here's a simple example of one conv layer taken from cs231n for clarification:
And my implementation in Pytorch:
import torch
from torch import nn
cnn = nn.Conv2d(in_channels=3, out_channels=2, kernel_size=3,
stride=2, padding=1, bias=True, groups=1)
w0 = torch.FloatTensor([[[-1, -1, 0],
[ 1, 1, 1],
[ 1, 1, 0]],
[[ 1, 1, -1],
[ 0, 0, 0],
[ 1, 1, -1]],
[[ 0, -1, 0],
[-1, 0, -1],
[ 1, 0, 1]]])
b0 = torch.FloatTensor([1])
w1 = torch.FloatTensor([[[-1, 0, 0],
[ 1, 1, 1],
[-1, -1, 0]],
[[ 1, -1, -1],
[-1, 1, -1],
[ 1, -1, 0]],
[[ 1, -1, 0],
[ 0, 1, 1],
[ 1, 0, 1]]])
b1 = torch.FloatTensor([0])
cnn.weight = torch.nn.Parameter(torch.stack((w0, w1), 0))
cnn.bias = torch.nn.Parameter(torch.cat((b0, b1), 0))
inpt = torch.FloatTensor([[[ 1, 2, 0, 1, 2],
[ 1, 0, 2, 2, 0],
[ 2, 0, 0, 2, 2],
[ 0, 0, 2, 2, 0],
[ 2, 2, 2, 1, 2]],
[[ 2, 0, 0, 1, 1],
[ 1, 0, 2, 1, 2],
[ 2, 0, 2, 2, 1],
[ 0, 2, 0, 0, 1],
[ 1, 2, 1, 2, 0]],
[[ 0, 0, 2, 1, 2],
[ 0, 1, 0, 2, 0],
[ 1, 1, 0, 0, 2],
[ 0, 0, 0, 1, 1],
[ 0, 1, 2, 0, 2]]])
cnn(inpt.unsqueeze(0))
Output:
tensor([[[[ 7., 9., 10.],
[ 0., 6., 10.],
[ 2., 5., 2.]],
[[ 4., 4., 4.],
[ 5., 1., 2.],
[ 2., 6., 0.]]]])
I have a rankingMat which contains the rankings of equity tickers, where every column represents one ticker and every row represents a point in time. The ranking has been performed in a descending manner, i.e. a 1 in rankingMat identifies the highest rank for that time period (read: row). Ticker/column 4 represents a cash security. This will be important further down the road.
Now, I would like to identify in which equity tickers I am supposed to invest. There exists two conditions:
I only invest in tickers that have a rank less or equal to 3.
In addition, I only invest in tickers that are ranked higher or equal compared to cash (ticker/column 4).
I'm able to get a result that fulfills the first condition:
rankingMat = ...
[NaN, NaN, NaN, NaN, NaN, NaN; ...
1, 5, 2, 3, 6, 4; ...
4, 5, 2, 3, 6, 1; ...
4, 1, 2, 5, 6, 3; ...
6, 4, 5, 2, 1, 3; ...
2, 3, 4, 6, 1, 5; ...
3, 6, 4, 1, 2, 5; ...
2, 5, 6, 1, 4, 3];
portfolio = rankingMat <= 3;
The result looks like this:
portfolio = ...
[0, 0, 0, 0, 0, 0; ...
1, 0, 1, 1, 0, 0; ...
0, 0, 1, 1, 0, 1; ...
0, 1, 1, 0, 0, 1; ...
0, 0, 0, 1, 1, 1; ...
1, 1, 0, 0, 1, 0; ...
1, 0, 0, 1, 1, 0; ...
1, 0, 0, 1, 0, 1]
My problem is condition 2. In every row, I need to compare not only if the integer is less or equal to 3, I also need to make sure that it is less than the integer in column 4 in that particular row. I am looking for a solution that avoids a for-loop. I guess it is possible with indexing. So, any hint is highly appreciated.
The final result should look like this:
portfolio = ...
[0, 0, 0, 0, 0, 0; ...
1, 0, 1, 1, 0, 0; ...
0, 0, 1, 1, 0, 1; ...
0, 1, 1, 0, 0, 1; ...
0, 0, 0, 1, 1, 0; ...
1, 1, 0, 0, 1, 0; ...
0, 0, 0, 1, 0, 0; ...
0, 0, 0, 1, 0, 0]
% Prior to R2016b
portfolio = rankingMat <= 3 & ...
bsxfun(#lt, rankingMat, rankingMat(:,4));
% On or after R2016b
portfolio = rankingMat <= 3 & ...
rankingMat < rankingMat(:,4);
I have a very large collection (millions of documents) with data which looks like:
u'timestamp': 1481454871423.0,
u'_id': ObjectId('584d351772c4d8106cc43116'),
u'data': {
...
},
u'geocode': [{u'providerId': 2, u'placeId': 97459515},
{u'providerId': 3, u'placeId': 237},
{u'providerId': 3, u'placeId': 3}]
I want a query which targets a providerId and placeId pair, and returns 10 records only, within a timestamp range.
To that end I perform queries like:
'geocode.providerId': 3,
'geocode.placeId': 3
'timestamp': { '$gte': 1481454868360L,
'$lt': 1481454954839L }
And I provide a hint, to target the index which looks like:
[('geocode.providerId', 1), ('geocode.placeId', 1), ('timestamp', 1)]
where 1 is ascending. Before iterating over the returned cursor, it is limited to 10 records and sorted ascending on timestamp (which should be it's default state due to the index).
A pymongo query looks like:
collection.find(findDic).hint(hint).sort([('timestamp', pymongo.ASCENDING)]).skip(0).limit(10)
The query explains come back looking like:
{
u'executionStats': {
u'executionTimeMillis': 1270,
u'nReturned': 10,
u'totalKeysExamined': 568686,
u'allPlansExecution': [],
u'executionSuccess': True,
u'executionStages': {
u'needYield': 0,
u'saveState': 4442,
u'memUsage': 54359,
u'restoreState': 4442,
u'memLimit': 33554432,
u'isEOF': 1,
u'inputStage': {
u'needYield': 0,
u'saveState': 4442,
u'restoreState': 4442,
u'isEOF': 1,
u'inputStage': {
u'needYield': 0,
u'docsExamined': 284964,
u'saveState': 4442,
u'restoreState': 4442,
u'isEOF': 1,
u'inputStage': {
u'saveState': 4442,
u'isEOF': 1,
u'seenInvalidated': 0,
u'keysExamined': 568686,
u'nReturned': 284964,
u'invalidates': 0,
u'keyPattern': {u'geocode.providerId': 1,
u'timestamp': 1,
u'geocode.placeId': 1},
u'isUnique': False,
u'needTime': 283722,
u'isMultiKey': True,
u'executionTimeMillisEstimate': 360,
u'dupsTested': 568684,
u'restoreState': 4442,
u'direction': u'forward',
u'indexName': u'geocode.providerId_1_geocode.placeId_1_timestamp_1',
u'isSparse': False,
u'advanced': 284964,
u'stage': u'IXSCAN',
u'dupsDropped': 283720,
u'needYield': 0,
u'isPartial': False,
u'indexBounds': {u'geocode.providerId': [u'[3, 3]'
],
u'timestamp': [u'[-inf.0, 1481455513378)'
],
u'geocode.placeId': [u'[MinKey, MaxKey]'
]},
u'works': 568687,
u'indexVersion': 1,
},
u'nReturned': 252823,
u'needTime': 315863,
u'filter': {u'$and': [{u'geocode.placeId': {u'$eq': 3}},
{u'timestamp': {u'$gte': 1481405886510L}}]},
u'executionTimeMillisEstimate': 970,
u'alreadyHasObj': 0,
u'invalidates': 0,
u'works': 568687,
u'advanced': 252823,
u'stage': u'FETCH',
},
u'nReturned': 0,
u'needTime': 315864,
u'executionTimeMillisEstimate': 1150,
u'invalidates': 0,
u'works': 568688,
u'advanced': 0,
u'stage': u'SORT_KEY_GENERATOR',
},
u'nReturned': 10,
u'needTime': 568688,
u'sortPattern': {u'timestamp': 1},
u'executionTimeMillisEstimate': 1200,
u'limitAmount': 10,
u'invalidates': 0,
u'works': 568699,
u'advanced': 10,
u'stage': u'SORT',
},
u'totalDocsExamined': 284964,
},
u'queryPlanner': {
u'parsedQuery': {u'$and': [{u'geocode.placeId': {u'$eq': 3}},
{u'geocode.providerId': {u'$eq': 3}},
{u'timestamp': {u'$lt': 1481455513378L}},
{u'timestamp': {u'$gte': 1481405886510L}}]},
u'rejectedPlans': [],
u'namespace': u'mxp957.tweet_244de17a-aa75-4da9-a6d5-97b9281a3b55',
u'winningPlan': {
u'sortPattern': {u'timestamp': 1},
u'inputStage': {u'inputStage': {u'filter': {u'$and': [{u'geocode.placeId': {u'$eq': 3}},
{u'timestamp': {u'$gte': 1481405886510L}}]},
u'inputStage': {
u'direction': u'forward',
u'indexName': u'geocode.providerId_1_geocode.placeId_1_timestamp_1',
u'isUnique': False,
u'isSparse': False,
u'isPartial': False,
u'indexBounds': {u'geocode.providerId': [u'[3, 3]'],
u'timestamp': [u'[-inf.0, 1481455513378)'
],
u'geocode.placeId': [u'[MinKey, MaxKey]'
]},
u'isMultiKey': True,
u'stage': u'IXSCAN',
u'indexVersion': 1,
u'keyPattern': {u'geocode.providerId': 1,
u'timestamp': 1,
u'geocode.placeId': 1},
}, u'stage': u'FETCH'},
u'stage': u'SORT_KEY_GENERATOR'},
u'limitAmount': 10,
u'stage': u'SORT',
},
u'indexFilterSet': False,
u'plannerVersion': 1,
},
u'ok': 1.0,
u'serverInfo': {
u'host': u'rabbit',
u'version': u'3.2.11',
u'port': 27017,
u'gitVersion': u'009580ad490190ba33d1c6253ebd8d91808923e4',
},
}
I don't understand why all of these documents need to be examined. In the case above, the size of the collection is only 284587 which means that every record was looked at twice! I want totalKeysExamined to only be 10, but am struggling to see how to achieve this.
I am using MongoDB version 3.2.11 and pymongo.
As Astro mentioned, the issue is that MongoDB was not using the index effectively.
MongoDB team say that this is resolved in later versions:
https://jira.mongodb.org/browse/SERVER-27386
Also an option is to remove providerId from the index. In my use case, providerId is one of two values, and most of the time will always be the same value. It represents which API was used to geocode; my system only supports two, and only has one enabled at any one time.
See the commit that would resolve this:
https://github.com/watfordxp/GeoTweetSearch/commit/420536e4a138fb22e0dd0e61ef9c83c23a9263c1
So the formula that needs to be implemented is:
P = ((1 - delta)/n) + ((delta)*(A)ij / (Sigma(k=1 to n)(A)ik))
where delta = 0.85
n = 8
and A = the adjacency matrix for web pages being surfed
The mathematica code for it is:
A = {{1, 1, 1, 0, 0, 1, 0, 1}, {0, 0, 1, 0, 0, 1, 0, 1}, {1, 1, 0, 1,
0, 1, 1, 0}, {0, 1, 1, 0, 1, 0, 1, 0}, {1, 1, 1, 1, 0, 1, 1,
1}, {1, 1, 1, 0, 0, 1, 1, 0}, {1, 0, 1, 0, 1, 0, 1, 0}, {0, 0, 0,
0, 1, 0, 0, 1}};
n = 8;
\[Delta] = 0.85;
P = Transpose[Table[(1 - \[Delta])/n + (\[Delta]*A[[i, j]])/(Sum[A[[i, k]], {k, 1, n}]), {i, 1, n}, {j, 1, n}]];
Everything else is just plugging in numbers.
Now the main problem I seem to have is getting the A[[i,j]]/Sum[A[i,k]] to work on matlab.
On matlab: when I input A[[i,j]] as A, and sum[A[i,k]] as either (sum(A,2))' or sum(A,1), the P that gets output on matlab becomes a column vector rather than an 8 x 8 matrix.
What am I missing?
There are many ways of doing it. I'll show you one way of using native vectorized MATLAB code, so no need for for-loops or arrayfun or anything like that.
A = [1, 1, 1, 0, 0, 1, 0, 1; 0, 0, 1, 0, 0, 1, 0, 1; 1, 1, 0, 1, 0, 1, 1, 0; 0, 1, 1, 0, 1, 0, 1, 0; 1, 1, 1, 1, 0, 1, 1, 1; 1, 1, 1, 0, 0, 1, 1, 0; 1, 0, 1, 0, 1, 0, 1, 0; 0, 0, 0, 0, 1, 0, 0, 1];
n = size(A, 1);
delta = 0.85;
% First, sum up all columns in each row
rowSums = sum(A,2);
% Then replicate them because the entries in each row of the vector are valid for every column in that specific row. So replicate them so that the output-matrix matches the size of A so we can work element-wise division later
rowSums2 = repmat(rowSums, 1, n);
% Use vectorized code, ./ will yield element-wise division as A and rowSums2 are of the same size now
P = (1 - delta)/n + delta*A ./ rowSums2
I hope I got the desired output right.