AST’s Online Testing Courses, and Power of Two Bugs in Excel 2007

The Association for Software Testing has recently started offering the Black Box Testing Course, designed by Cem Kaner, James Bach, and Becky Fiedler, as a free benefit to AST members. I took the first round (taught by Cem, Scott Barber, and Jon Hagar) and was very impressed. I expected that any course designed by Cem would reflect a deep understanding of testing – and it did – but what surprised me was the deep understanding of teaching. Pedagogically, this course stood head-and-shoulders above the previous online courses I’ve taken from UC Berkeley Extension and Foothill College.

I’ll write more about it soon, but for now I want to highlight one exercise in particular. We were told:

You are testing a program that includes an Integer Square Root function. The function reads a 32-bit word that is stored in memory, interprets the contents as an unsigned integer and then computes the square root of the integer, returning the result as a floating point number.

And then asked a number of thoughtful questions about what test strategy we would take, how many square roots we would(n’t) test, etc. It’s an interesting problem, and many different suggestions were made. One of common thread was the importance of testing around powers of two, especially just over and under boundaries of 8, 16, and 32-bit integers. Now, Doug Hoffman has an excellent article showing that interesting errors can occur far from any of these boundaries…but testing around powers of two was one of our baselines.

That said, I’m fascinated to learn that Excel 2007 has a major calculation error at 2**16-1. Many (or all?) formulas that should evaluate to 65535 instead evaluate as 100000. So for example:

  • 850 x 77.1 – 1 = 65534
  • 850 x 77.1 = 100000

As a test engineer, I fully understand that not every bug will ever be caught…but it fascinates me that Microsoft‘s test team doesn’t seem to have thoroughly tested the boundary between 16-bit and 32-bit integers. This is of course being discussed on Slashdot, Digg, and many other places. I’m curious how Microsoft’ll react, and what the repercussions will be.

Advertisements

8 Responses to “AST’s Online Testing Courses, and Power of Two Bugs in Excel 2007”

  1. Michael Bolton Says:

    When you’re dealing with an output variable, what are the “boundary” input values?

    —Michael B.

  2. Cem Kaner Says:

    Michael asked:

    When you’re dealing with an output variable, what are the “boundary” input values?

    The AST BBST course is actually a series. We taught the first course in the series, the BBST-Foundations course in August/September. We’ll teach another instance of this one starting in mid-October. But this is only the first course in the series.

    The next course (coming in November) is on Bug Advocacy. The third course (coming probably in January or February) is Domain Testing.

    It is in Domain Testing that we consider, carefully, the question:
    When you’re dealing with an output variable, what are the “boundary” input values?

    – Cem

  3. Biff Says:

    Interesting over-generalization of the problem. This bug is a display error that occurs in about 12 out of 18446744073709551616 possible floating point binary numbers.

    How dare those “QA” people miss this.

    You should get a job with the media – they’re also good at oversensationalizing minute issues.

  4. Ben Simo Says:

    @Biff

    Don’t underestimate the potential impact of a bug that occurs in only “12 out of 18446744073709551616” possible values. Users could make very wrong decisions based on calculations if they see incorrectly displayed values. Yes, there is a very small chance of encountering the error. But when it is encountered, the impact could be big.

    I don’t blame the “QA” people at Microsoft. This is a good illustration of the complexity of software and the impossibility of complete testing coverage.

    Minute issues can create big problems.

  5. tod hilton Says:

    I just came across your blog (and this post) via Thinking Tester so my comment is late, but here’s the official update as posted on the Excel Team blog:

    Issue: http://blogs.msdn.com/excel/archive/2007/09/25/calculation-issue-update.aspx

    Fix: http://blogs.msdn.com/excel/archive/2007/10/09/calculation-issue-update-fix-available.aspx

  6. Wolfram Arnold Says:

    Great post, Jeff. Doug Hoffman’s article is amazing, and their finding comes at no surprise to me engineer. What’s scary to me is that the lead architect and QA guy stumbled over the problem and test requirements over dinner. I have a dream that all engineers brainstorm testing needs _before_ they write the first line of code. Microsoft’s issue with Excel case in point.

  7. Research Funding and Advisory Board for the Black Box Software Testing (BBST) Course « Cem Kaner, J.D., Ph.D. Says:

    […] each. The courses are offered, for free, to AST members. AST is starting its second teaching of the Foundations course this week. We’ll teach Bug Advocacy in a month. As we develop these courses, we are […]

  8. Riyaj Shaikh Says:

    I have been working with Performance reporting team who shares Performance reports of Organization to GM’s. Imagine that a team whose revenue/loss is usually in range of 60-62 k from last few quarters , got increased to 1 million this quarter. What is impact ? What decision a GM could take ? Ben Simo is very right 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: