SMath Studio Forum
»
SMath Studio
»
Bugs & Problems
»
Strange rounding issues for small custom units
Rank: Newbie
Groups: Registered
Joined: 23/10/2017(UTC) Posts: 1
|
I implemented several units to deal with field unit conversions for oil field calculations.
In the oil field we use strange units:
millidarcies (mD), centipoise (cP), and barrels/day,
the other units have largely already been implemented by smath: wheee!.
mD is an absurdly small unit which I defined as:
<property dimension="253"> <add factor="4.0468564224" exp="3" system="Imperial" id="295"> <synonym name="acre" lang:rus="акр" /> </add>
<add factor="9.869233" exp="-13" system="Imperial" id="441"> <synonym name="D Darcy" /> <extension for="D"> <prefix name="m" /> </extension> </add>
where 295 is <add connection="m^2" id="253" />
which I hope is alright.
if I make a statement, like
k:=20'mD
"and then query k,
I get results that seem to round to 1 decimal place, ie
2*10^-14 m^2
ohh my goodness that is ~1.5%, which is poor performance compared to all the other awesome features!!
it seems to do this in decimal mode, and not in fraction mode, which sort of makes sense, that the fractions would not be rounded the same way.
Seems like a wierd floating point error.
also the behavior isn't always reproducible. very strange.
thanks for the awesome program!
|
|
|
|
Rank: Guest
Groups: Registered
Joined: 04/07/2015(UTC) Posts: 6,866 Was thanked: 981 time(s) in 809 post(s)
|
Originally Posted by: Jean Giraud ... many more Engineering don't come as easily as 2*3=6 m² You don't have to freak Smath unit system, just like this: ... rather like this
|
|
|
|
SMath Studio Forum
»
SMath Studio
»
Bugs & Problems
»
Strange rounding issues for small custom units
Forum Jump
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.