I have a field which is a Number(decimal) here's one such example value:
1005.44
now if I try to convert that to a string to simply get me what is seen; I'd expect STR = 1005.44
but instead I get:
1005.4400000000001
thats from STR(ROUND(([FIELD]),2))
I see some other posts with similar issues but no found resolution?
This artifact isn't unique to Tableau and stems from how underlying databases store floating point numbers and deal with functions like rounding.
You should try the following:
str(int([FIELD]*100)/100)
This will multiply the number by 100, convert to an int, divide by 100, and then convert to a string.
Related
I was expecting Decimal to treat number like a String but instead I reproduced small rounding during testing.
let locale = Locale(identifier: "en_GB")
let price: String? = "1535132527181273627"
let decimal: Decimal? = price != nil ? Decimal(string: price!, locale: locale) : nil
XCTAssertEqual(decimal, 1535132527181273627.0)
The result bothers me:
XCTAssertEqual failed: ("Optional(1535132527181274000)") is not equal to ("Optional(1535132527181273497.6)") -
I tried your code and the result was:
XCTAssertEqual failed: ("Optional(1535132527181273627)") is not equal to ("Optional(1535132527181273497.6)")
using Xcode 9.4.1 on a MacBook Pro 2015
I think the problem is in the fact that Double is only able to deal with 15 decimal places. So the number 1535132527181273627.0 cannot be represented accurately. Take a look at Double precision - decimal places
This is a design problem of the Swift compiler.
It has been reported as SR-3317 but that was closed as duplicate of SR-920.
The problem is that the literal 1535132527181273627.0 is parsed as a Double literal (that is, with limited precision) even if the type should be Decimal. Using a String to initialize Decimal is the only workaround for now.
The solution is to redesign the built-in protocol ExpressibleByFloatLiteral to allow for longer numbers.
So it looks like the problem is in the test, not the code. Swift uses Double to store number from the test and it's not long enough, Decimal wasn't rounded but the number to compare in the test was.
Solution is to compare Decimal.
XCTAssertEqual(price, Decimal(string:"1535132527181273627"))
I have a fusiontable based feature collection in Google-Earth-Engine that includes a date column. I would like to cast the FC into an empty image and display with graduated colours for increasing dates - so that when the Inspector is used a human readable date is displayed.
var empty64 = ee.Image().toInt64();
var outlines = empty64.paint({
featureCollection: SomeFeatureCollection,
color: 'StartDate',
});
If I add this to the map as a layer I get the date as a 13-digit format that I can't read. How can I change this?
Thank you!
Per the API reference for Image.paint, color must be an object name or a number. To me, that means the EE API will interpret the object as a number, meaning a Date object will be converted from a string to a numerical representation. In this case, that's the "milliseconds since epoch" format.
Without a way to add metadata to a FeatureCollection (i.e. so you could store the associated datestring along with the other parameters of the feature), I don't think you can show a human readable date in the inspector.
I am sure this has a simple answer, however it alludes me at the moment! I am wanting to convert a float field in a JasperReports report by dividing by an integer (in fact converting hours to days):
$F{Average} / 24
This results in a blank result. I have tried to use:
$F(Average).divide(java.math.BigInteger(24))
and similar, but get compilation errors.
Apologies if this is a bit simple, but it would be good to get some assistance!
You are missing the new keyword:
$F(Average).divide(new java.math.BigDecimal(24, 2, RoundingMode.HALF_UP))
For example, I have an NSDecimal myDecimal. Lets say it represents something like "-1234567.89"
How can I get a clean string representation of that decimal without any beautification / formatting? No localization? Floating point symbol = . and rest only numbers from 0 to 9, and eventually an - if it is negative? I need that string in strict technical manner. A number like 123456789876554432123456789.2231 should therefore not look like nice formatted "123,456,789,876,554,432,123,456,789.2231". You get the point right?
I don't want any formatting. I'm trying all day now to get that right but everything I find always has to do with formatting. So how'd you guys do that?
I don't believe there is such a thing as a "clean" string representation independent of specifying the locale. As many Europeans would point out, 123.45 should be written as 123,45 (using , instead of . for the decimal location). NSDecimalString() (which can be found in the Foundation Functions Reference) takes, as a second parameter a locale specification. If some locale uses the format you desire, pass that locale as the second parameter (see the Internationalization Guide for more info on locales).
Alternatively, you can use an NSNumberFormatter, which will give you more controll over the string representation.
For easy (localized) control use an NSNumberFormatter.
float number = 12.345;
NSString* numberString = [NSString stringWithFormat:#"%f", number];
That will give you consistent formatting regardless of the user's current locale.
I'm using GWT 2.0 and when I try to use NumberFormat to format a Double the results are not as expected:
NumberFormat format = NumberFormat.getFormat( "#.########" );
Double d = new Double("0.256281093911");
format.format(d);
formatted string: 0.02147484
As you can see the formatted value is wrong (this can be seen in the gwt showcase). Is this something related to the custom format I'm using (#.########)? or is this a bug in the GWT formatter?
If this is a bug, have someone found a workaround?
Thanks.
link text
you might try NumberFormat.getDecimalFormat. Also, it might have something to do with your browser's locale and/or it's decimal delimiter.
A Double is stored in binary format in java in general. Working with binary float values is in most cases not what you want - conversion between binary and decimal floats causes rounding errors.
Try "BigDecimal", which stores float values in DECIMAL format, not in binary format.
Sample code:
NumberFormat format = NumberFormat.getFormat("#.########");
format.format(new BigDecimal("0.256281093911"));