I'm trying to write a program that cleans data, using Matlab. This program takes in the max and min that the data can be, and throws out data that is less than the min or greater than the max. There looks like a small issue with the cleaning part. This case ONLY happens when the minimum range of the variable being checked is 0. If this is the case, for one reason or another, the program won't throw away data points that are between 0 and -1. I've been trying to fix this for some time now, and noticed that this is the only case where this happens, and if you try to run a SQL query selecting data that is < 0, it will leave out data between 0 and -1, so effectively the same error as what's happening to me. Wondering if anyone might recognize this and know what it could be.
I would write such a function as:
function data = cleanseData(data, limits)
limits = sort(limits);
data = data( limits(1) <= data & data <= limits(2) );
end
an example usage:
a = rand(100,1)*10;
b = cleanseData(a, [-2 5]);
c = cleanseData(a, [0 -1]);
-1 is less than 0, so 0 should be the max value. And if this is the case it will keep points between -1 and 0 by your definition of the cleaning operation:
and throws out data that is less than the min or greater than the max.
If you want to throw away (using the above definition)
data points that are between 0 and -1
then you need to set 0 as the min value and -1 as the max value --- which does not make sense.
Also, I think you mean
and throws out data that is less than the min AND greater than the max.
It may be that the floats are getting casted to ints before the comparison. I don't know matlab, but in python int(-0.5)==0, which could explain the extra data points getting in. You can test this by setting the min to -1, if you then also get values from -1 to -2 then you'll need to make sure casting isn't being done.
If I try to mimic your situation with SQL, and run the following query against a datatable that has 1.00, 0.00, -0.20, -0.80. -1.00, -1.20 and -2.00 in the column SomeVal, it correctly returns -0.20 and -0.80, which is as expected.
SELECT SomeVal
FROM SomeTable
WHERE (SomeVal < 0) AND (SomeVal > - 1)
The same is true for MatLab. Perhaps there's an error in your code. Dheck the above statement with your own SELECT statement to see if something's amiss.
I can imagine such a bug if you do something like
minimum = 0
if minimum and value < minimum
Related
I know how to 'select' a range in LO (7.2.4.1) Calc BASIC ....
ThisComponent.CurrentController.ActiveSheet.getCellRangeByName("D1:H6")
But how to write a value, e.g. "1", into that range using BASIC?
myRange = ThisComponent.CurrentController.ActiveSheet.getCellRangeByName("D1:H6")
myRange.Value = 1
Gives an "property or method not found" error. But I can't find any properties or values to go after Range to allow me to do what I want. Flailing around and trying
myRange.setValue = 1
myRange.writeValue = 1
myRange.setString = "1"
and numerous other variants don't work either.
Would really appreciate the solution. Thanks.
You can edit the value of an individual cell, but not the entire range. You will have to iterate over all the cells in the range one at a time, changing the value of each of them.
Sub Set1ToD1H6
myRange = ThisComponent.CurrentController.ActiveSheet.getCellRangeByName("D1:H6")
For i = 0 To myRange.getRows().getCount()-1
For j = 0 To myRange.getColumns().getCount()-1
myRange.getCellByPosition(j, i).setValue(1)
Next j
Next i
End Sub
But since the read-write operation to a cell is comparable in time to the read-write operation to a whole range, it is preferable to use another method - to prepare data in an array and write from it to a range in one operation:
Sub Set1ToRange
myRange = ThisComponent.CurrentController.ActiveSheet.getCellRangeByName("D1:H6")
dataOfRange = myRange.getData()
For i = LBound(dataOfRange) To UBound(dataOfRange)
For j = LBound(dataOfRange(i)) To UBound(dataOfRange(i))
dataOfRange(i)(j) = 1
Next j
Next i
myRange.setData(dataOfRange)
End Sub
(For your example, this will be approximately 30 times faster, for a larger range the time winnings will be even more significant)
The .getData() and .setData() methods work on numeric range values. To work with text strings (and numbers), use .getDataArray() and .setDataArray(), for working with cell formulas use .getFormulaArray() and .setFormulaArray()
I have an array of 1 x 400, where all element values are above 1500. However, I have some elements that have values<50 which are wrong measures and I would like to have the mean of the elements before and after the wrong measured data points and replace it in the main array.
For instance, element number 17 is below 50 so I want to take the mean of elements 16 and 18 and replace element 17 with the new mean.
Can someone help me, please? many thanks in advance.
No language is specified in the question, but for Python you could work with List Comprehension:
# array with 400 values, some of which are incorrect
arr = [...]
arr = [arr[i] if arr[i] >= 50 else (arr[i-1]+arr[i+1])/2 for i in range(len(arr))]
That is, if arr[i] is less than 50, it'll be replaced by the average value of the element before and after it. There are two issues with this approach.
If i is the first or last element, then one of the two values will be undefined, and no mean can be obtained. This can be fixed by just using the value of the available neighbour, as specified below
If two values in a row are very low, the leftmost one will use the rightmost one to calculate its value, which will result in a very low value. This is a problem that may not occur for you in practice, but it is an inherent result of the way you wish to recalculate values, and you might want to keep it in mind.
Improved version, keeping in mind the edge cases:
# don't alter the first and last item, even if they're low
arr = [arr[i] if arr[i] >= 50 or i == 0 or i+1 == len(arr) else (arr[i-1]+arr[i+1])/2 for i in range(len(arr))]
# replace the first and last element if needed
if arr[0] < 50:
arr[0] = arr[1]
if arr[len(arr)-1] < 50:
arr[len(arr)-1] = arr[len(arr)-2]
I hope this answer was useful for you, even if you intend to use another language or framework than python.
I'm trying to evaluate the performance of these two loop method, I tried number from 0 to 99999 using for in and while loop clause.
for i in 0..<s.count - 9 {
print("\(i)")
}
var j = 0
while j < s.count - 9 {
print("\(j)")
j = j+1
}
In both loop, will print the current number and add number by 1 until it reaches 99999.
Turns out that for in clause use 0.91 to go through every number, at same time while take much much much longer time (around 80.8).
I searched on Internet and documents, but cannot figure out why.
What cause this huge performance difference?
I have the following code working out the efficient frontier for a portfolio of assets:
lb=Bounds(:,1);
ub=Bounds(:,2);
P = Portfolio('AssetList', AssetList,'LowerBound', lb, 'UpperBound', ub, 'Budget', 1);
P = P.estimateAssetMoments(AssetReturns)
[Passetmean, Passetcovar] = P.getAssetMoments
pwgt = P.estimateFrontier(20);
[prsk, pret] = P.estimatePortMoments(pwgt);
It works fine apart from the fact that it ignores the constraints to some extent (results below). How do I set the constraints to be hard constraints- i.e. prevent it from ignoring an upper bound of zero? For example, when I set an upper and lower bound to zero (i.e. I do not want a particular asset to be included in a portfolio) I still get values in the calculated portfolio weights for that asset, albeit very small ones, coming out as part of the optimised portfolio.
Lower bounds (lb), upper bounds (ub), and one of the portfolio weights (pwgt) are set out below:
lb ub pwgt(:,1)
0 0 1.06685493772574e-16
0 0 4.17200995972422e-16
0 0 0
0 0 2.76688394418301e-16
0 0 3.39138439553466e-16
0.192222222222222 0.252222222222222 0.192222222222222
0.0811111111111111 0.141111111111111 0.105624956477606
0.0912121212121212 0.151212121212121 0.0912121212121212
0.0912121212121212 0.151212121212121 0.0912121212121212
0.0306060606060606 0.0906060606060606 0.0306060606060606
0.0306060606060606 0.0906060606060606 0.0306060606060606
0.121515151515152 0.181515151515152 0.181515151515152
0.0508080808080808 0.110808080808081 0.110808080808081
0.00367003367003366 0.0636700336700337 0.0388531580005063
0.00367003367003366 0.0636700336700337 0.0636700336700338
0.00367003367003366 0.0636700336700337 0.0636700336700337
0 0 0
0 0 0
0 0 1.29236898960272e-16
I could use something like: pwgt=floor(pwgt*1000)/1000;, but is there not a more elegant solution than this?
The point is that your bound has not been ignored.
You are calculating with floating point numbers, and hence 0 and 4.17200995972422e-16 are both close enough to 0 to let your program allow them.
My recommendation would indeed be to round your results (or simply display less decimals with format short), however I would do the rounding like this:
pwgt=round(pwgt*100000)/100000;
Note that the other results may also be 'above' the upper bound, however this will not become visible due to the insignificance.
I had issues like this with a laminate/engineering properties code, which was propogating errors all over everything. I fixed it by taking all of the values I had, and systematically converting them from double to sym, and suddenly my 1e-16 values became real zeros, that I could also eval(val) and still see as zeros! This may help, but you may have to go inside of the .m files you're running, and have the numbers convert to sym with val = sym(val).
I can't remember for certain, but I think Matlab functions MIGHT change sym to double once they receive the data for their own internal processing.
We're running into a problem with one of our reports. In one of our tablixes a textbox has the following expression:
=Iif(Fields!SomeField.Value = 0, 0, Fields!SomeOtherField.Value / Fields!SomeField.Value)
Which should be pretty self-explanatory. If "SomeField" is zero, set the text box value to zero, else set it to "SomeOtherValue / SomeValue".
What has us stumped is that the report still throws a runtime exception "attempted to divide by zero" even though the above expression should prevent that from happening.
We fiddled a bit with the expression just to make sure that the zero-check is working, and
=Iif(Fields!SomeField.Value = 0, "Yes", "No")
works beautifully. Cases where the data is in fact zero resulted in the textbox displaying "Yes" and vice versa. So the check works fine.
My gut feel is that the Report rendering engine throws the exception at run-time, because it "looks" as if we are going to divide by zero, but in actual fact, we're not.
Has anyone run into the same issue before? If so, what did you do to get it working?
IIf will always evaluate both results before deciding which one to actually return.
Try
=IIf(Fields!SomeField.Value = 0, 0, Fields!SomeOtherField.Value / IIf(Fields!SomeField.Value = 0, 1, Fields!SomeField.Value))
This will use 1 as the divisor if SomeOtherField.Value = 0, which does not generate an error. The parent IIf will return the correct 0 for the overall expression.
An easy clean way to prevent a divide by zero error is using the report code area.
In the Menu, go to Report > Report Properties > Code and paste the code below
Public Function Quotient(ByVal numerator As Decimal, denominator As Decimal) As Decimal
If denominator = 0 Then
Return 0
Else
Return numerator / denominator
End If
End Function
To call the function go to the the Textbox expression and type:
=Code.Quotient(SUM(fields!FieldName.Value),SUM(Fields!FieldName2.Value))
In this case I am putting the formula at the Group level so I am using sum. Otherwise it would be:
=Code.Quotient(fields!FieldName.Value,Fields!FieldName2.Value)
From: http://williameduardo.com/development/ssrs/ssrs-divide-by-zero-error/
On reflection, I feel best idea is to multiply by value to power -1, which is a divide:
=IIf
(
Fields!SomeField.Value = 0
, 0
, Fields!SomeOtherField.Value * Fields!SomeField.Value ^ -1
)
This doesn't fire pre-render checks as val * 0 ^ -1 results in Infinity, not error
IIF evaluates both expression even thought the value of Fields!SomeField.Value is 0. Use IF instead of IIF will fix the problem.