Significant Digits and Rounding

Summary:  A common question is “how should we round the results of this calculation?” The answer comes from the concept of significant digits.

What are Significant Digits?

What strikes you about the difference between a measurement of 1.615 in and 1.6 in? The first measurement is more precise than the second. Specifically, the first measurement is accurate to the nearest thousandth (0.001) of an inch, while the second measurement is accurate to only the nearest tenth (0.1) of an inch.

No measurement in the real world is exact; all have some degree of “slop”. When we present a numerical result of 1.615 in, it is understood to be accurate to the nearest thousandth: we know that the true measurement is between 1.6145 and 1.6155 in. By contrast, if we present a value of 1.6 in, we are really saying that it is 1.6 to the nearest tenth: the true measurement could be anywhere between 1.55 and 1.65 in.

You can see that every real-world number carries information both about its magnitude and about its precision. 1.615 in and 1.6 in have about the same magnitude, but the first one is more precise. We talk about that level of precision as how many significant digits the number has.

The significant digits in a number start at the first non-zero digit and end at the last digit. Examples: 1417 has four significant digits, and so do 1.417 and 0.00001417. What about 14.1700? It has six significant digits, not four, because only zeroes at the start of a number are non-significant. Finally, 14.07 has four significant digits.

There can be some ambiguity with trailing zeroes in a large whole number. For instance, we quote the average distance from earth to sun as 93 million miles. In that form, the number has two significant digits. (Remember what significant digits mean: they mark the non-“slop” part of the measurement. All we’re saying is that the average distance is 92½ to 93½ million. But suppose we write the number as 93,000,000? Does it now have eight significant digits? Are we saying the average distance is between 92,999,999.5 and 93,000,000.5 miles? Surely not!

When you see a large round whole number, the zeroes may represent “slop”. To get around this problem, numbers are often expressed in scientific notation. For instance, the figure of 93 million miles is 9.3×10⁷ miles (“nine point three times ten to the seventh”). On your calculator it appears as 9.3E7. You’ll find some more about scientific notation later on this page.

Practice

How many significant digits are in each of these numbers? 4800, 4800.0, 4.8, 0.0000067, .0000067.

Answers:

4800 has two to four significant digits. The 4 and 8 are definitely significant, but just by looking at the number we can’t tell whether it’s accurate to the nearest whole number (4800, four significant digits), to the nearest ten (480x, three significant digits), or to the nearest hundred (48xx, two significant digits).

4800.0 has five significant digits. In the previous example (plain 4800), the two zeroes might indicate precision of measurement or be there simply as place holders. But with 4800.0 the last zero is obviously not needed as a place holder and therefore it must be significant.

4.8 has two significant digits.

0.0000067 and .0000067 are the same number. Leading zeroes are never significant, and therefore this number has two significant digits.

Rounding the Results of Calculations

Now you see what’s wrong with “25 feet divided by 6.0 equals 4.166666667 feet”, which is what your calculator will tell you. A measurement of 25 feet is accurate only to the nearest foot; you can’t get an answer that is accurate to a billionth of a foot!

The rule for multiplying and dividing is this: find the number of significant digits in each factor. The answer will have the smaller number of significant digits. Example: 25 and 6.0 each have two significant digits. The smaller of 2 and 2 is 2; therefore the answer will have two significant digits. 25÷6.0 = 4.2 is correct.

Example: Multiply 81.7×2.405. Your calculator says 196.4805. But 81.7 has three significant digits and 2.405 has four; therefore the answer must have three significant digits (smaller of 3 and 4). 81.7×2.405 = 196.

That last illustration makes an important point: when multiplying or dividing, your guide to rounding is not the decimal places but the number of significant digits. Even though each factor had one or more decimal places, the answer has no decimal places because it must have only three significant digits.

For powers and roots, the answer should have the same number of significant digits as the original. For example, taking the square root of 167.21 (five significant digits), your calculator gives 12.93097057, which you round to 12.931 (five significant digits). Why is that? Because the square root is defined as the number such that, when you multiply itself, you get back to the original. If the original has five significant digits, then the number that you multiply by itself to get back to the original must have five significat digits.

Practice

Compute and round your answer properly: 34.78×11.7÷0.17. Answer: the least significant number has two significant digits, and therefore the answer must have two significant digits. You take the 2393.682353 from your calculator and round it to 2400 (or 2.4×10² in scientific notation).

Compute and round your answer properly: 16.2². Answer: the original has three significant digits, and the answer must have the same. Your calculator gives 262.44, which you round to 262.

Three people will share a lottery prize of $24.8 million equally. How much will each one receive, before taxes? Answer: 24.8÷3 = 8.2666666667 on the calculator. 24.8 has three significant digits. The 3 looks like it has only one significant digit, but it is an exact number: not between 2½ and 3½ people but 3 people exactly. Think of it as being 3.000000000... people. Therefore the answer will have three significant digits: each person gets $8.27 million.

Quick Rules of Thumb

Computing mean and standard deviation involves lots of multiplications and additions plus a division and maybe a square root. For small data sets, on the order of 10 to 20 numbers, when the dust settles your mean should usually have the same number of decimal places as your data, and your standard deviation should usually have one more decimal place than your data.

This rule of thumb isn’t always correct, but it’s simple to apply and in our class we usually care more about the leading digits than about how many are significant.

Example: Compute the mean and standard deviation of 1.4, 2.21, 3.8, 9.2, and 2.77. Answer: the less precise numbers have one decimal place, and therefore the mean should have one decimal place and the standard deviation should have two decimal places. The TI-83 gives a mean of 3.876 → 3.9 and standard deviation of 3.10161732 → 3.10.

How to Round an Answer

Once you’ve computed an unrounded answer, how do you round it correctly? Decide how many significant digits (or decimal places) you’ll need, and then round all at once. Example: if the TI-83 computes a mean of numbers with one decimal place as 3.876, that needs to be rounded to one decimal place. Draw a line at the spot where the rounding must happen: 3.8|76. Because what’s to the right of the line is bigger than 5, you round up to 3.9.

If the first digit after the line is 5 to 9, round up; if the first digit after the line is 0 to 4, round down. Example: to round 2.884 and 2.885 to the nearest hundredth, write 2.88|4 → 2.88 and 2.88|5 → 2.89.

It’s important to round all at once, not digit by digit. Example: Round 30.4746 to the nearest hundredth.

Incorrect solution: 30.4746 → 30.475 → 30.48.

Correct solution: 30.47|46 → 30.47.

What’s to the right of the line is less than 5, and you round down to 30.47.

Practice

Round to the nearest whole number: 17.514, 24.500, 24.501, 27.499.

Answers: 17.514 → 18 (5 to 9, round up), 24.500 → 25 (5 to 9, round up), 24.501 → 25 (5 to 9, round up), 27.499 → 27 (0 to 4, round down).

Round to one decimal place: 17.545, 24.451, 38.989.

Answers: 17.545 → 17.5 (0 to 4, round down), 24.451 → 24.5 (5 to 9, round up), 38.987 → 39.0 (38.9|87 — the 8 of 87 means that the 9 of 38.9 must increase; 38.9 rolls over to 39.0).

The Big No-no

Never round in the middle of a calculation; always round only the final answer.

Example: 1.2×1.2×1.5. Since the factors have two significant digits, the answer will have two significant digits. But you must carry along your intermediate results without rounding. 1.2×1.2 = 1.44, 1.44×1.5 = 2.16 → 2.2. But if you had rounded 1.44 to 1.4 in mid-stream, you would have 1.4×1.5 = 2.1, which is off. Always wait till the end of a calculation to round.

When you’re working with your calculator, never re-enter an intermediate result you see on your screen. Instead, chain your later calculation to the earlier one. Example: (√2.00)×6.000. The factors have three and four significant digits, and therefore the answer will have three significant digits. Find √2 on your calculator; you should get something like 1.414213562. Do not re-enter this number. Instead, press the [×] key and then 6 and [ENTER]. Your calculator will display Ans*6 and then an answer of 8.485281374, which you round to 8.49. Ans tells you that the previous answer is being carried along in its original unrounded form.

Suppose instead you had rounded, entering 1.41×6? You would get 8.46, pretty far off from the correct answer. Always let the calculator carry full precision along for you.

Suppose you have a big decimal, like 11.25055509, and you’re supposed to square it and round the answer to one decimal place? While it is wrong to round to one decimal place before squaring, you certainly don’t need to type in eight decimals. You are usually safe if you compute with two more significant figures than your ultimate answer will have. In this case you’re squaring 11, and the ultimate answer will be 120-odd; with one decimal place that’s four significant figures. You enter 11.2506 and square it: 126.576 → 126.6.

Practice

A goat is tethered to a post in your back yard with a chain 8.4 feet long. Assuming the goat doesn’t eat the chain, how much area of your yard will the goat be able to munch on? (Recall that the area of a circle is πr².)

Answer: 8.4 has two significant digits, and therefore the answer will also have two significant digits. You need 8.4²×π. Enter 8.4, then press the [x²] key and [ENTER] to obtain 70.56. Now press the [×] key. To enter π, find the symbol in gold just over the [^] key (which is above the [÷] key). Press [2nd] [π] and [ENTER] to obtain 221.6707776; round to two significant digits for an answer of 220 square feet.

(By the way, it’s possible to do quite complicated calculations in one step on your calculator. That’s a fine approach, as long as you think about the order of operations and put in parentheses where they are needed. Both methods will give the same answer.)

Scientific Notation

Scientific notation was developed to express very large and very small numbers.

To write a large number in scientific notation, move the decimal point to the left until it is between the first and second significant digits; the number of places moved is the exponent. For example, 167 becomes 1.67, but the decimal point moved two places left. Therefore 167 = 1.67×10² or 1.67E2.

Scientific notation removes the guesswork about how significant a large number is. 9.3E7 has two significant digits; it is accurate to the nearest million miles. 9.30000E7 has five significant digits, and it is accurate to the nearest hundred miles (0.00001×10⁷ = 100 miles).

To write a small number in scientific notation, move the decimal point right until it has just passed the first non-zero digit. Write the number of places moved as a negative number in the exponent. Example: 0.0000894 must move the decimal point five places right to become 8.94E–5 or 8.94×10^–5.

Notice that big numbers end up with a positive exponent after the E and small numbers end up with a negative exponent after the E.

To convert a number from scientific notation to ordinary decimals, reverse the process. A positive exponent indicates a big number: move the decimal point to the right. A negative exponent indicates a small number: move the decimal point to the left. Example: if the probability of an event is 6.014E–4 or 6.014×10^–4, you must move the decimal point four places left to convert it to 0.0006014. If the population of the earth is about 6.1×10⁹ people, you move the decimal point right nine places to convert to 6,100,000,000.

To enter scientific notation in your calculator, find the EE symbol in gold just over the comma key (which is above the 7). Example: to enter 6.1×10⁹, press [6] [.] [1] [2nd] [EE] 9. If you then press the [ENTER] key, you will probably see 6100000000. In normal mode, the calculator tries to present results in ordinary numbers if they fit on the screen.

Practice

Write in scientific notation: 6,370,000 m (radius of the earth, assuming accuracy to the nearest 1000 m); 0.00000110 in (diameter of a polio virus).

Answers: 6,370,000 accurate to the nearest 1000 m must be rounded as 6,370,|000: it has four significant digits. Since the decimal point must move six places left, the exponent is 6: 6.370×10⁶ m. In 0.00000110 in, there are three significant digits including that last zero. The decimal point moves six places to the right: 1.10×10^–6.