AP CSP Big Idea 2 Overflow Roundoff
AP CSP Overflow & Roundoff Errors: Complete Guide (2025‑2026)
Overflow occurs when a calculation produces a result too large for the number of bits allocated to store it — the extra bits are dropped, producing a wrong answer without any error message. Roundoff error occurs because most decimal fractions cannot be represented exactly in binary — the computer stores an approximation, and tiny errors accumulate through repeated calculations. Both are consequences of the fundamental limitation of representing infinite mathematical values with finite bits.
Contents
Integer Overflow
Overflow silently wraps around to a small value — no error message, just a wrong answer. Programs must be designed to detect when results exceed the maximum storable value.
A game stores the player’s score in an 8-bit unsigned integer (maximum value: 255). A player reaches score 250 and earns 10 more points. What does the game display?
Work out what happens when 250 + 10 is stored in 8 bits. What does the player see?
250 + 10 = 260. In 8 bits, the maximum is 255 (11111111). 260 in binary is 100000100 — a 9-bit number. The 9th bit is dropped, leaving 00000100 = 4. The game displays a score of 4. The player loses all their progress. This is overflow: the result exceeds the storage capacity, the overflow bit is silently dropped, and a nonsensical small value is stored. Real games use 32 or 64-bit integers to avoid this.
Roundoff Error
- 1/3 = 0.3333... cannot be stored exactly in decimal
- Similarly, 0.1 in binary = 0.0001100110011... (repeating)
- Computer stores a close approximation
- 0.1 + 0.2 = 0.30000000000000004 in most languages
- Tiny error, but can accumulate in repeated calculations
- Financial calculations: $0.001 error x millions of transactions
- Engineering: small errors accumulate in simulations
- Scientific computing: thousands of iterations amplify tiny errors
- Equality comparisons: if x = 0.3 fails when x = 0.2999...
- AP exam: recognize that decimal operations may not be exact
A program adds 0.1 to a running total 10 times. A student expects the result to be exactly 1.0.
What result will the program actually produce? Why?
The result will be approximately 0.9999999999999999 or 1.0000000000000002 depending on the language and implementation. Each addition of 0.1 introduces a tiny roundoff error (because 0.1 cannot be stored exactly in binary). After 10 additions, these tiny errors accumulate to a visible difference from exactly 1.0. This is why comparing floating-point results with exact equality (if total = 1.0) is unreliable in code — instead, programs check whether the result is within a small tolerance (e.g., within 0.0001 of 1.0).
Real-World Consequences
- Ariane 5 rocket (1996): software reused from Ariane 4; larger velocity values caused 64-bit to 16-bit overflow; rocket self-destructed $370M of hardware
- Pac-Man kill screen: level 256 caused overflow in the level counter, corrupting the right half of the screen
- Y2K: 2-digit year storage meant 2000 stored as ‘00’, potentially less than 1999’s ‘99’
- 32-bit Unix timestamp expires: Jan 19, 2038
- Integer overflow in game scores rolling over to 0 or negative
- Vancouver Stock Exchange (1982): index rounded down each recalculation; 22 months later it was 574 instead of 1098
- Patriot missile (1991): time accumulated in 0.1-second intervals; 100-hour drift caused 1/3 mile targeting error
- Financial software: fractional cent roundoff across millions of transactions
- Scientific simulations: small errors amplify across thousands of iterations
- Nuclear calculations: precision requirements in safety-critical code
A bank’s interest calculation program uses integer arithmetic for speed. When calculating 3% of $100, the program computes 3/100 = 0 (integer division drops the decimal). Millions of customers receive $0 interest.
What went wrong? Is this overflow, roundoff, or something else?
This is integer truncation, closely related to roundoff. When dividing two integers in integer arithmetic, the fractional part is discarded (not rounded — truncated). 3 / 100 = 0 in integer arithmetic. The fix is to use floating-point arithmetic for financial calculations, or to multiply first (3 x 100,000 / 100 = 3,000 cents) and work in integer cents rather than fractional dollars. Financial systems universally avoid floating-point arithmetic for money due to both truncation and roundoff concerns.
Common Exam Pitfalls
Overflow usually happens silently. The program continues running with a wrong value. This is what makes it dangerous — there is no crash to alert you to the problem.
Roundoff affects any program doing decimal arithmetic, including simple financial calculations. 0.1 + 0.2 = 0.3 fails in virtually every major programming language.
A runtime error stops the program. Overflow does not stop the program — it produces a wrong answer silently. This makes overflow harder to detect than a runtime error.
More bits improve precision but do not eliminate roundoff. 0.1 cannot be represented exactly in binary regardless of how many bits you use, because it is a repeating fraction in base-2.
Check for Understanding
1. A variable storing an 8-bit unsigned integer contains the value 255 (the maximum). The program adds 1. What is the most likely result?
- The program displays an error message and stops.
- The variable now stores 256.
- The variable wraps around to 0 due to integer overflow.
- The variable stores 255 (the addition is ignored).
2. A student writes code to check if a calculated floating-point value equals exactly 0.3 using IF result = 0.3. This check will most likely:
- Always return true if the calculation is mathematically correct.
- Fail intermittently because 0.3 cannot be represented exactly in binary, and the result may be 0.29999... or 0.30000...1
- Return true only if the result was calculated using addition.
- Generate a syntax error because floating-point comparisons use different operators.
3. Consider statements about overflow and roundoff:
I. Overflow occurs when a calculation produces a value too large for the allocated number of bits.
II. Roundoff error can be completely eliminated by using more bits.
III. Both overflow and roundoff can produce incorrect results without generating an error message.
Which are correct?
- I only
- I and III only
- II and III only
- I, II, and III
4. The Ariane 5 rocket exploded 37 seconds after launch due to an overflow error. The software had been used successfully on Ariane 4. The most likely cause of the overflow was:
- The Ariane 5 software was programmed in a different language than Ariane 4.
- Ariane 5 flew faster than Ariane 4, producing velocity values too large for the 16-bit storage that was sufficient for Ariane 4.
- The launch occurred at a higher altitude, causing atmospheric interference.
- The rocket was carrying more payload than the software was designed to handle.
5. Which of the following best describes a consequence of roundoff error accumulating over many calculations?
- Programs slow down as roundoff errors consume processing time.
- Calculated results drift progressively further from mathematically exact values with each operation.
- The program terminates after a fixed number of roundoff events.
- Memory usage increases as roundoff errors require additional storage bits.
6. A financial application needs to calculate totals for millions of transactions involving amounts like $0.10 and $0.25. Which approach best avoids roundoff errors?
- Use floating-point arithmetic and round to two decimal places after each calculation.
- Store all amounts as integers representing cents (e.g., $0.10 = 10 cents) and use integer arithmetic.
- Use a programming language that does not have roundoff errors.
- Perform all calculations at least three times and average the results.
Frequently Asked Questions
Get in Touch
Whether you're a student, parent, or teacher — I'd love to hear from you.
Just want free AP CS resources?
Enter your email below and check the subscribe box — no message needed. Students get daily practice questions and study tips. Teachers get curriculum resources and teaching strategies.
Message Sent!
Thanks for reaching out. I'll get back to you within 24 hours.
Prefer email? Reach me directly at [email protected]