Did you know that if you set minimum or maximum number of slides in a VoiceThread assignment, if a student does not meet those requirements, they will not be able to submit the assignment. One known bug in at least one assignment is that a minimum number of slides was set (5), but when the student viewed it, it was changed to maximum number of slides. See answer for resolution.
Related
I'm processing Sentinel 2 L2A data in sen2r as downloaded from googlecloud and spotted an issue: if computing NDVI from bands 4 and 8 at 10 m resolution, I get an extreme offset after January 26th as exemplified by the last 7 values in the following image (this is just one pixel, but I find this consistently over many pixels):
NDVI-time series
Searching the web, I learned that the issue very likely is related to the fact that since January 26th Sentinel 2 products are processed using a new baseline (v0400) which will also affect L2A products. However, I did not find a solution for the problem if working in sen2r.
The relevant section of the description of the major product update states:
Provision of negative radiometric values (implementing an offset): The
dynamic range will be shifted by a band-dependent constant:
BOA_ADD_OFFSET. This offset will allow encoding negative surface
reflectances that may occur over very dark surfaces. From the user’s point
of view, the L2A Bottom of Atmosphere (BOA) reflectance (L2A_BOA) shall
be retrieved from the output radiometry as follows:
• Digital Number DN=0 remains the “NO_DATA” value
• For a given DN in [1; 1;2^15-1], the L2A BOA reflectance value will
be:
L2A_BOAi = (L2A_DNi + BOA_ADD_OFFSETi) / QUANTIFICATION_VALUEi
The radiometric offset value will be reported in a new field in the
General_Info/Product_Image_Characteristics section of the Datastrip and User
Product Metadata. It will be initially set to -1000 Digital counts for all bands.
It is also noted that the percentage of negative surface reflectance pixels per
band will be also reported in the L2A_QUALITY report in the QI_DATA folder of
the tile.
Now my question is: how to fix this issue? I want to standardize my data over time, which does not make sense with that offset. So I need some kind of harmonization between the baselines. How would that work in sen2r?
Thanks in advance for any advice.
I suppose you may have solved this by now. But as you mentioned you will have to subtract the BOA_ADD_OFFSET from the pixel values in a band if you wish compare them with pre 22-01-25 images (Baseline v04.00).
In my processing I have preprocessed using GDAL to subtract the offset for both L1 and L2 images and clamped any resulting negative values to 0 (NODATA) before comparing them with pre 22-01-25 images.
I am working with a 2 y-axis graph, one is generally between 40K and 60K, the other between 5K and 10K. What I would like to do is set the the 40K to be a number such as if the MIN = 42K, start at 40K and increment by 5K. If It is 38K, start at 35K. Similarly for the 2nd y-axis, do the same but based on 2K increments. When I set it to automatic I get basically straight lines or I say do not include 0 and i get huge drastic swings. I can set the starting and set the increment, but that means every day I would have to go in and verify that still works, for example 40K is a good start, but one that that may be too high or too low. (I suppose the fact it is 2 axis has nothing to do with it, but in case it does) The key is dynamically changing based on the result set.
If there is a better way to do this, I would love it. However, this got me close to what I wanted. First, I created 2 calculated fields, MIN and MAX using a windowed function on the data. They look something like this below. Note I did 2x the differences to give a window that is roughly 5x the total distance from min to max. Better math could give a better sizing.
Max_Ln=WINDOW_MAX(SUM([Profit]))+(WINDOW_MAX(SUM([Profit]))-WINDOW_MIN(SUM([Profit])))*2
Min_Ln= WINDOW_MIN(SUM([Profit]))-(WINDOW_MAX(SUM([Profit]))-WINDOW_MIN(SUM([Profit])))*2
I then added both to the Details pane and used this to add reference lines. I added the reference line with no title and no line. This will cause the automatic spacing to take them into account, but not show anything. From there I did the same on the 2nd y-axis and now everything looks good and will adjust dynamically.
According to authors in 1, 2, and 3, Recall is the percentage of relevant items selected out of all the relevant items in the repository, while Precision is the percentage of relevant items out of those items selected by the query.
Therefore, assuming user U gets a top-k recommended list of items, they would be something like:
Recall= (Relevant_Items_Recommended in top-k) / (Relevant_Items)
Precision= (Relevant_Items_Recommended in top-k) / (k_Items_Recommended)
Until that part everything is clear but I do not understand the difference between them and Recall rate#k. How would be the formula to compute recall rate#k?
Finally, I received an explanation from Prof. Yuri Malheiros (paper 1). Althougth recall rate#k as cited in papers cited in the questions seemed to be the normal recall metrics but applied into a top-k, they are not the same. This metric is also used in paper 2, paper 3 and paper 3
The recall rate#k is a percentage that depends on the tests made, i.e., the number of recommendations and each recommendation is a list of items, some items will be correct and some not. If we made 50 different recommendations, let us call it R (regardless of the number of items for each recommendation), to calculate the recall rate is necessary to look at each of the 50 recommendations. If, for each recommendation, at least one recommended item is correct, you can increment a value, in this case, let us call it N. In order to calculate the recall rate#R, it is neccesary to make the N/R.
I've got a bar chart with three months worth of data. Each column in the chart is one month's data showing the percentage of Rows that met a certain criterion for that month. In the first month, 100% of 2 rows meet the measure. In the second month, 24.2% of 641 rows meet the measure. In the 3rd month, 28.3% of 1004 rows meet the measure. My reference line which is supposed to show the average across the entire time-frame is showing 50.8%, the simple average (i.e. [100+24.2+28.3]/3) instead of the weighted average (i.e. [100*2+641*24.2+1004*28.3]/[2+641+1004]).
In the rows shelf, I have a measure called "% that meet the criterion", this is defined as SUM([Criterion])/SUM([NUMBER OF RECORDS])
The criterion measure is 1 for any record that qualifies and null for any that do not qualify.
If I go to Analysis >> Totals >> Show Row Grand Totals, a 4th bar is added, and that bar shows the correct weighted average of the other three bars (26.8%), but I really want this to be shown as a reference line instead of having an extra bar on the chart. (Adding the Grand Total bar also drops the reference line down to 44.8%, which is the simple average of the 4 bars now shown on the chart--I can't think of a less useful piece of information than that).
How can I add the weighted average as a reference line?
Instead of using 'Average' as your aggregation, try using 'Total' instead in the Edit Reference Line dialogue window.
I have to say it's a bit counter-intuitive, but this is what the Tableau online help has to say about it:
http://onlinehelp.tableau.com/current/pro/online/mac/en-us/reflines_addlines.html
Total - places a line at the aggregate of all the values in either the cell, pane, or the entire view. This option is particularly useful when computing a weighted average rather than an average of averages. It is also useful when working with a calculation with a custom aggregation. The total is computed using the underlying data and behaves the same as selecting one of the totals option the Analysis menu.
If you are using Tableau 9, you can make second calculated field using an LOD expression
{ SUM([Criterion]) / SUM([NUMBER OF RECORDS]) }
This will calculate the ratio for the entire data set after applying context and data source filters, without partitioning the data by any of the other dimensions in your view (such as month in your case)
If you place that new field on the detail shelf then you can use it to create a reference line.
There are other ways to generate a weighted average, but this is probably the simplest in your case.
The problem I'm trying to solve is finding the right similarity metric, rescorer heuristic and filtration level for my data. (I'm using 'filtration level' to mean the amount of ratings that a user or item must have associated with it to make it into the production database).
Setup
I'm using mahout's taste collaborative filtering framework. My data comes in the form of triplets where an item's rating are contained in the set {1,2,3,4,5}. I'm using an itemBased recommender atop a logLikelihood similarity metric. I filter out users who rate fewer than 20 items from the production dataset. RMSE looks good (1.17ish) and there is no data capping going on, but there is an odd behavior that is undesireable and borders on error-like.
Question
First Call -- Generate a 'top items' list with no info from the user. To do this I use, what I call, a Centered Sum:
for i in items
for r in i's ratings
sum += r - center
where center = (5+1)/2 , if you allow ratings in the scale of 1 to 5 for example
I use a centered sum instead of average ratings to generate a top items list mainly because I want the number of ratings that an item has received to factor into the ranking.
Second Call -- I ask for 9 similar items to each of the top items returned in the first call. For each top item I asked for similar items for, 7 out of 9 of the similar items returned are the same (as the similar items set returned for the other top items)!
Is it about time to try some rescoring? Maybe multiplying the similarity of two games by (number of co-rated items)/x, where x is tuned (around 50 or something to begin with).
Thanks in advance fellas
You are asking for 50 items similar to some item X. Then you look for 9 similar items for each of those 50. And most of them are the same. Why is that surprising? Similar items ought to be similar to the same other items.
What's a "centered" sum? ranking by sum rather than average still gives you a relatively similar output if the number of items in the sum for each calculation is roughly similar.
What problem are you trying to solve? Because none of this seems to have a bearing on the recommender system you describe that you're using and works. Log-likelihood similarity is not even based on ratings.