Microsoft and Beta Testing

Running beta versions is not for the faint of heart

Tens of millions of lines of code. That's not the amount of work I have to do before the year 2000; it's a measure of the amount of work Microsoft is putting into Windows 2000 (Win2K, formerly Windows NT 5.0). The question users ask me repeatedly about Win2K code is, "How can Microsoft test all the permutations of hardware and software we run into every day on the job?"

The answer is simple: Microsoft can't. Microsoft will test thousands—if not tens of thousands—of hardware and software combinations for Win2K. Undoubtedly, though, your enterprise will have hardware and software combinations that Microsoft hasn't tested, and some of these combinations won't work.

Consider NT 4.0's Service Pack 4 (SP4). Microsoft delayed the release of this service pack for a long time while the company tested, added features to, and retested the product. Yet, when users installed the final version of SP4, quite a few folks ended up with dead systems. Worse than the bugs that manifested themselves obviously were the more insidious problems that didn't show up until weeks later, when a user accessed a particular feature. For example, one of my NT desktop systems ran perfectly with a couple of SP4 betas and release candidates (RCs). Then, with the release code, the system developed a video driver bug. I received a hotfix for the bug, but the experience made me wary of deploying the service pack and ruined my confidence in running unreleased versions of SP4.

How will Microsoft avoid those annoying bugs users discover after the company releases the product? Simple: Microsoft will get users to pay for the privilege of testing the product to find those bugs before the product's final release. On the surface, paying for a new version of software that comes with bugs but without formal support seems foolish, but Microsoft discovered that the prestige of having beta software was enough to get people to fork over cash for the right to tell their friends that they have the beta code.

Win2K Beta
You can get a copy of the Win2K beta code in several ways. You can receive the beta version if you are a major corporate customer or one of Microsoft's beta testers, or you can pay for it. Microsoft isn't officially selling Win2K yet, but you can get it if you're willing to spend money. If you're a Microsoft Developer Network (MSDN) subscriber, you'll receive the Win2K beta with your subscription. If you attend one of Microsoft's Win2K road shows, you can pay $295 to hear about Win2K and receive a copy of the Win2K beta. Finally, you can buy a workstation from a major system vendor, such as Dell, with the Win2K beta preinstalled. However, good luck getting support if you install legacy hardware. It's not that the vendor won't want to provide support—you just might be the first person to run your problem configuration on the vendor's system.

I don't understand why someone would choose to purchase a system with a beta version. Why buy a new system that runs a buggy OS and has almost no vendor support? What happens when your applications don't run? Application vendors probably won't want to spend time helping users running an unsupported OS. However, companies such as Dell are in business to make money, and if those companies are offering a beta configuration, customers must be requesting it. That doesn't mean the idea makes sense, though.

Office 2000 Beta
As Office 2000's release date gets closer (Microsoft plans to ship the software sometime in April 1999), Microsoft has moved into the final stages of the beta cycle. At Comdex, Microsoft announced that the company expected 70,000 users to take advantage of the Corporate Preview Program. Approximately half as many users will sign up for the Consumer Preview Program that Microsoft announced in its Office email newsletter and on its Office Web site (http://www.microsoft.com/office). So, approximately 100,000 users will each send Microsoft $19.95 to test the latest Office 2000 beta. The preview versions will time out in mid-1999. However, you can install them to test your system's Year 2000 (Y2K) compliance if your system clocks are set between December 15, 1999, and March 30, 2000.

If 10 percent of the users who send $19.95 to Microsoft to test the Office 2000 beta actually use the product regularly, and 10 percent of those users actually send reports back to Microsoft, Microsoft will have 1000 beta testers for the preview version of the code. The $2,000,000 that Microsoft collects will probably cover the administrative costs of producing and distributing the CD-ROMs. Microsoft will provide technical support only through a Microsoft newsgroup server, which will probably include a fair amount of peer support. Even if the people enrolled in the preview programs discover only a small number of bugs, these programs are a huge win for Microsoft. The programs let Microsoft truthfully claim that huge numbers of beta testers and end users provided input in the product design cycle. The company can also claim that it provided an opportunity for users to be involved in beta testing.

Beta Testing
I don't want to sound as though I think beta-testing programs are bad. I wrote this article on a system that runs Win2K beta 2 and Office 2000 beta 1. But part of my job is learning as much as possible about new products while they're still in beta. I maintain extra computers on which I perform regular backups, and I'm prepared to do complete OS reinstalls (a task I perform with annoying frequency) at a moment's notice.

Running beta versions is not for the occasional user or the faint of heart. Beta versions make major—and in many cases permanent—changes to the systems you install them on. In addition, beta versions often aren't feature-complete or they lack functionality in crucial areas.

Widespread beta testing is becoming de rigueur in the computer industry. But if you don't need to get involved in beta testing, don't. Testing the latest beta doesn't provide you any benefit other than early exposure to code that might or might not be the same code that the final version of the product ships with. Being able to say you're running the Win2K beta is cool, but the first major system crash you experience might convince you that purchasing the beta wasn't worth the investment. Helping software companies get good products out the door is a great idea, but getting your job done is a more important one.

End of Article

I agree wholeheartedly with David Chernicoff’s Forefront: “Microsoft and Beta Testing” (February). Although beta testing gives you advanced exposure to prerelease code, betas often don’t resemble the finished product. Why waste your time and effort to unlearn and relearn functionality that might be considerably different between the interim betas and the version of a product that ships? I’d rather have a stable system and the guidance of third-party books than deal with a buggy product and the confusing documentation that sometimes accompanies a beta. I share the author’s view that getting your job done is more important, and I agree with the saying, “We are all beta testers—–it’s just that some of us pay for the privilege.”<br>--Robert Slovick

Robert Slovick August 06, 1999

As David Chernicoff stated in his February Forefront column, “Running beta versions is not for the faint of heart.” I agree.
For weeks, I stared at the Microsoft Office 2000 Corporate Preview Pack (CPP), which sat on a shelf in the same condition as it was in when I received it in the mail. Finally, I got the urge to install at least one part of the new Office product. I’d heard Outlook 2000 was pretty stable, so I opted to install just that piece. When I inserted the CD-ROM, I knew I was in trouble. Instantly, I was taken out of the install loop and the application began to install the entire Office package without my permission.
I winced, then prayed that everything would be OK. After rebooting my system, I discovered that Word, Access, and Excel all seemed fine. What I didn’t take into account were the data access components (also beta) that the software installed.
Within a couple of days, the roof fell in. As I opened an existing Visual InterDev 6.0 project, I discovered that ODBC was definitely in dire straits. Visual InterDev repeatedly faulted, and within a couple of hours, I felt that Dr. Watson was a long-lost cousin of mine who had returned home.
Eventually, I gave up. I had an install party 2 days later. Needless to say, the CPP is back on the shelf, and I’m scared of installing release code. This experience is one that I’ll carry with me for years and tell others about. Users need to go through this experience once in their life. I have. Tell me it gets better.<br>--Christopher Scragg

Christopher Scragg August 06, 1999

You must log on before posting a comment.

If you don't have a username & password, please register now.