TAOCP 4.2.2 Exercise 25
When people speak of "cancellation" in floating point subtraction, they refer not to an actual inaccuracy in the operation $u \ominus v$ itself, which is computed exactly according to equation (3), bu...
Section 4.2.2: Accuracy of Floating Point Arithmetic
Exercise 25. ▶ [**] [15] When people speak about inaccuracy in floating point arithmetic they often ascribe errors to "cancellation" that occurs during the subtraction of nearly equal quantities. But when $u$ and $v$ are approximately equal, the difference $u \ominus v$ is obtained exactly, with no error. What do these people really mean?
Verified: yes
Solve time: 1m40s
When people speak of "cancellation" in floating point subtraction, they refer not to an actual inaccuracy in the operation $u \ominus v$ itself, which is computed exactly according to equation (3), but to the loss of significant digits in the representation of the result relative to the magnitudes of $u$ and $v$. If $u$ and $v$ are nearly equal, then the leading digits of $u$ and $v$ cancel, leaving only the low-order digits, which may have been affected by previous rounding errors. Consequently, the relative error of the result, defined as $|(u \ominus v) - (u - v)| / |u - v|$, can be very large, even though $u \ominus v$ is exactly $u - v$ in floating point arithmetic.
This phenomenon is often called "catastrophic cancellation," because the subtraction magnifies preexisting rounding errors from earlier computations. The apparent inaccuracy is therefore a consequence of propagating prior errors through an operation that produces a result much smaller in magnitude than the operands, not a defect in the subtraction itself.
∎