Is 4 + 4 > 8?
Your pen tip hovers over the signature line on a purchase order for an 8-way
system. Suddenly you are overcome by indecision, and your pen quivers as your hand starts to shake. Is an 8-way system the right investment for your company and your applications? Will it give you the performance you need? A bead of sweat forms on your temple and slides down your cheek. A cold shiver runs down your spine, and you slowly place the pen on your desk. You realize that purchasing an 8-way system is no simple decision--you need answers before you can put ink on paper.
| Is an 8-way system the right investment for your company and your applications? |
If you believed Microsoft's promise of scalability, you wouldn't hesitate to sign the purchase order. In 1997, Microsoft held its Scalability Day and promised scalability of 8-way systems with the Windows NT Server 4.0, Enterprise Edition family of products, with support for 16-way systems not far behind. I remember this promise clearly because I was in the audience.
Has Microsoft kept its promise? The answer depends on whom you ask. At a
recent Windows NT Magazine Professionals Conference, a Microsoft
repre-sentative proudly announced that Microsoft had achieved good
scalability with 8-way systems, citing Transaction Processing Council (TPC) test
results that showed a 65 percent performance improvement for 8-way vs. 4-way
systems. This number surprised me because the Windows NT Magazine Lab
tests haven't shown such an improvement. I decided to analyze Microsoft's data.
Microsoft Math
Microsoft compared a 4-way system running NT Server and SQL Server with a
4-way system running NT Server, Enterprise Edition (NTS/E) and SQL Server,
Enterprise Edition (SQL/E). As Figure 1, page 72 shows, the standard NT Server and SQL Server configuration generated a score of 9800 transactions per minute (tpm). The NTS/E and SQL/E configuration generated a score of 12,100tpm, a 23 percent improvement. The reason for this dramatic improvement is NTS/E and SQL/E's very large memory (VLM) support. The 4-way system Microsoft tested had 4GB of RAM, which NTS/E and SQL/E fully utilized.
Microsoft then pitted the NT Server and SQL Server results (9800tpm)
against an 8-way system with 4GB of RAM running NTS/E and SQL/E. The 8-way NTS/E and SQL/E configuration scored much higher.
Figure 2, page 72, shows the result: 16,200tpm. This number is a whopping 65 percent improvement over the 4-way system--pretty impressive.
However, Microsoft's comparison was unfair. Microsoft's results included
not only the improvement of an 8-way over a 4-way system but also the
improvement you get with VLM support. If you can upgrade your 4-way system to 4GB of RAM and run NTS/E and SQL/E for a 24 percent performance improvement, why buy an 8-way system? Upgrading a 4-way system is far cheaper than buying an 8-way system.
I still wondered about the true difference between a 4-way and an
8-way configuration (without VLM support). Fortunately, Microsoft provided the pertinent information, so I just needed to do some math. The 4-way system with 4GB of RAM running NTS/E and SQL/E produced 12,100tpm. The 8-way system with 4GB of RAM running NTS/E and SQL/E produced 16,200tpm. The increase between these two configurations is 34 percent.
Yes, that's right. Using Microsoft's own tests, the performance improvement between a 4-way system and an 8-way system is only 34 percent. And you can bet that Microsoft reports its best-case test results, so your performance improvement will vary depending on the applications you run.
The Plot Thickens
Like you, I take Microsoft's benchmark numbers as gospel and don't bother
running independent tests to verify the results. Yeah, right.
The Lab Guys and I ran a series of tests in the Lab to verify Microsoft's benchmarking numbers. Unlike Microsoft, we didn't use separate 4-way and 8-way systems. We tested two 8-way systems: Data General's AViiON 8600, reviewed on page 74, and NCR's WorldMark 4380, reviewed on page 78. Both systems had 4GB of RAM and ran NTS/E. To test the difference between 4-way and 8-way systems, we disabled (via software configuration) some processors in each system. We tested three configurations: 4-way, 6-way, and 8-way. (For other 8-way systems, see "Vendor List for 8-Processor Servers.")
In all fairness to Microsoft, the tests we ran on the AViiON 8600 showed
performance improvement similar to Microsoft's TPC benchmarks for 4-way and
8-way systems. The AIM Domain Server Mix test showed an improvement of 31
percent for sustained performance in the 8-way configuration and a 50 percent improvement for peak performance. (For more information about AIM Technology's tests, see "AIM Technology Server Benchmark Test," page 76.) Sustained
performance is the key measurement because it represents the workload a system
can handle under stress. Our sustained performance result of 31 percent for the
AViiON 8600 8-way configuration isn't far from Microsoft's TPC benchmark of 34
percent.
A more interesting finding was the spread of improvement between 4-way,
6-way, and 8-way configurations. We discovered that the greatest performance
improvement is between a 4-way and a 6-way configuration. The AIM WNT Sustained
Performance test showed a 22 percent improvement from a 4-way to a 6-way system
and a 31 percent improvement from a 4-way to an 8-way system. That's only a 7
percent performance improvement between the 6-way and 8-way configurations. The
AIM WNT Peak Performance test showed a 32 percent improvement from a 4-way to a
6-way system and a 50 percent improvement from a 4-way to an 8-way system.
Again, only a marginal performance improvement of 14 percent between the 6-way
and 8-way configurations. Based on these numbers, a 6-way system is more
impressive than an 8-way system.
Our tests also showed that 8-way configurations are not uniformly scalable.
Our AViiON 8600 test results were similar to Microsoft's TPC tests, but our
WorldMark 4380 test results didn't measure up. The AIM WNT Sustained Performance
test showed a 5 percent improvement from a 4-way to a 6-way system and an
11 percent improvement from a 4-way to an 8-way system, which is a performance
gain of only 6 percent between 6-way and 8-way configurations. The AIM WNT Peak
Performance test showed a 13 percent improvement from a 4-way to a 6-way system
and only a 12 percent improvement from a 4-way to an 8-way system (performance
dropped between the 6-way and 8-way configurations).
The WorldMark 4380 has a different multiple-processor architecture than the
AViiON 8600 has. Based on our test results, the WorldMark 4380 is better suited
for a multiple-application environment than for scaling up one application.
When I contacted Microsoft to discuss 4-way, 6-way, and 8-way performance,
the company reported a slightly different spread than our test results showed.
According to Microsoft's TPC tests, the performance difference between a 4-way
and a 6-way configuration is 17 percent. The difference between a 4-way and an
8-way configuration is 34 percent, which yields a 15 percent spread between a
6-way and an 8-way system. Although 15 percent isn't phenomenal, it's better
than the 7 percent difference we saw on our AViiON 8600 sustained performance
tests.
Microsoft's TPC benchmarks focus on NTS/E and SQL/E. In contrast, the AIM
tests we ran focused on the operating system (OS), which was NTS/E, rather than
on BackOffice, Enterprise Edition components (e.g., SQL/E and Exchange,
Enterprise Edition). We ran preliminary BackOffice benchmarks to verify that
they report the same performance trends we found.
Previous
