calc: Precision
7.2 Precision
=============
The ‘p’ (‘calc-precision’) command controls the precision to which
floating-point calculations are carried. The precision must be at least
3 digits and may be arbitrarily high, within the limits of memory and
time. This affects only floats: Integer and rational calculations are
always carried out with as many digits as necessary.
The ‘p’ key prompts for the current precision. If you wish you can
instead give the precision as a numeric prefix argument.
Many internal calculations are carried to one or two digits higher
precision than normal. Results are rounded down afterward to the
current precision. Unless a special display mode has been selected,
floats are always displayed with their full stored precision, i.e., what
you see is what you get. Reducing the current precision does not round
values already on the stack, but those values will be rounded down
before being used in any calculation. The ‘c 0’ through ‘c 9’ commands
(
Conversions) can be used to round an existing value to a new
precision.
It is important to distinguish the concepts of “precision” and
“accuracy”. In the normal usage of these words, the number 123.4567 has
a precision of 7 digits but an accuracy of 4 digits. The precision is
the total number of digits not counting leading or trailing zeros
(regardless of the position of the decimal point). The accuracy is
simply the number of digits after the decimal point (again not counting
trailing zeros). In Calc you control the precision, not the accuracy of
computations. If you were to set the accuracy instead, then
calculations like ‘exp(100)’ would generate many more digits than you
would typically need, while ‘exp(-100)’ would probably round to zero!
In Calc, both these computations give you exactly 12 (or the requested
number of) significant digits.
The only Calc features that deal with accuracy instead of precision
are fixed-point display mode for floats (‘d f’; Float Formats),
and the rounding functions like ‘floor’ and ‘round’ (Integer
Truncation). Also, ‘c 0’ through ‘c 9’ deal with both precision and
accuracy depending on the magnitudes of the numbers involved.
If you need to work with a particular fixed accuracy (say, dollars
and cents with two digits after the decimal point), one solution is to
work with integers and an “implied” decimal point. For example, $8.99
divided by 6 would be entered ‘899 <RET> 6 /’, yielding 149.833
(actually $1.49833 with our implied decimal point); pressing ‘R’ would
round this to 150 cents, i.e., $1.50.
Floats, for still more on floating-point precision and
related issues.
Info Catalog
calc: General Mode Commands
calc: Mode Settings
calc: Inverse and Hyperbolic