Conor O'Mahony's Database Diary

Your source of IBM database software news (DB2, Informix, Hadoop, & more)

Archive for the ‘Benchmark’ Category

Anatomy of an Oracle Marketing Claim

with 10 comments

Yesterday, Oracle announced a new TPC-C benchmark result. They claim:

In this benchmark, the Sun Fire X4800 M2 server equipped with eight Intel® Xeon® E7-8870 processors and 4TB of Samsung’s Green DDR3 memory, is nearly 3x faster than the best published eight-processor result posted by an IBM p570 server equipped with eight Power 6 processors and running DB2. Moreover, Oracle Database 11g running on the Sun Fire X4800 M2 server is nearly 60 percent faster than the best DB2 result running on IBM’s x86 server.

Let’s have a closer look at this claim, starting with the first part: “nearly 3x faster than the best published eight-processor result posted by an IBM p570 server“. Interestingly, Oracle do not lead by comparing their new leading x86 result with IBM’s leading x86 result. Instead they choose to compare their new result to an IBM result from 2007, exploiting the fact that even though this IBM result was on a different platform, it uses the same number of processors. Of course, we all know that the advances in hardware, storage, networking, and software technology over half a decade are simply too great to form any basis for reasonable comparison. Thankfully, most people will see straight through this shallow attempt by Oracle to make themselves look better than they are. I cannot imagine any reasonable person claiming that Oracle’s x86 solutions offer 3x the performance of IBM’s Power Systems solutions, when comparing today’s technology. I’m sure most people will agree that this first comparison is simply meaningless.

Okay, now let’s look at the second claim: “nearly 60 percent faster than the best DB2 result running on IBM’s x86 server“. Oracle now compare their new leading x86 result with IBM’s leading x86 result. However, if you look at the benchmark details, you will see that IBM’s result uses half the number of CPU processors, CPU cores, and CPU threads. If you look at performance per core, the Oracle result achieves 60,046 tpmC per CPU core, while the IBM result achieves 75,367 tpmC per core. While Oracle claims to be 60% faster, if you take into account relevant system size and determine the performance per core, IBM is actually 25% faster than Oracle.

Finally, let’s not forget the price/performance metric from these benchmark results. This new Oracle result achieved US$.98/tpmC, whereas the leading IBM x86 result achieved US$.59/tpmC. That’s correct, when you determine the cost of processing each transaction for these two benchmark results IBM is 39% less expensive than Oracle. (BTW, I haven’t had a chance yet to determine if Oracle Used their Usual TPC Price/Performance Tactics for this benchmark result, as the result details are not yet available to me; but if they have, the IBM system will prove to be even less expensive again than the Oracle system.)

Benchmark results are as of January 17, 2012: Source: Transaction Processing Performance Council (TPC),
Oracle result: Oracle Sun Fire X4800 M2 server (8 chips/80 cores/160 threads) – 4,803,718 tpmC, US$.98/tpmC, available 06/26/12.
IBM results: IBM System p 570 server (8 chips/16 cores/32 threads) -1,616,162 tpmC, US$3.54 /tpmC, available 11/21/2007. IBM System x3850 X5 (4 chips/40 cores/80 threads) – 3,014,684 tpmC, US$.59/tpmC, available 09/22/11.


Written by Conor O'Mahony

January 18, 2012 at 11:01 am

Benchmark Results for Informix TimeSeries in Meter Data Management

leave a comment »

AMT-SYBEX are a leading provider of platforms for traditional and smart metering. They created a Meterflow Benchmark to help customers choose the best underpinning infrastructure for their platform, and they worked with IBM to run that benchmark with Informix TimeSeries. I previous blogged about Why Informix Rules for Time Series Data Management. Well, the results of this benchmark further illustrate the benefits of Informix TimeSeries. The following quote is from the resulting AMT-SYBEX case study:

We believe that this represents ground breaking levels of performance which is ten times faster than other published benchmarks in this area.

As you can see, Informix is 10x faster than the leading database software they previously worked with. If you read the Executive Summary, you will also see that IBM Informix enjoys almost linear scalability when going from 10 million meters up to 100 million meters, which is a great testament to the efficiency of operation for Informix TimeSeries.

Written by Conor O'Mahony

September 26, 2011 at 1:50 pm

Industry Benchmark Result for DB2 pureScale: SAP Transaction Banking (TRBK) Benchmark

with 7 comments

A couple of years ago, IBM introduced the pureScale feature, which provides application cluster transparency (allowing you to create shared-disk database clusters). At the time, IBM had taken their industry-leading clustering architecture from the mainframe, and brought it to Unix environments. IBM subsequently also brought it to Linux environments.

Today, IBM announced its first public industry benchmark result for this cluster technology. IBM achieved a record result for the SAP Transaction Banking (TRBK) Benchmark, processing more than 56 million posting transactions per hour and more than 22 million balanced accounts per hour. The results were achieved using IBM DB2® 9.7 on SUSE Linux® Enterprise Server. The cluster contained five IBM System x 3690 X5 database servers, and used the IBM System Storage® DS8800 disk system. The servers were configured to take over workload in case of a single system failure, thereby supporting high application availability. For more details, see the official certification from SAP.

Written by Conor O'Mahony

September 12, 2011 at 11:16 am

IBM DB2 Improves Performance Lead for x86-64 Systems with a new Record-Breaking Result

with 2 comments

Last year, I blogged about how IBM DB2 has the Leading x86-based TPC-C Result. Well, IBM has further cemented DB2’s position as the leading database for x86-64 systems with a new record-breaking TPC-C benchmark result. The new benchmark result achieved more than 3 million transactions per minute on an IBM System x 3850 X5. The entire system for this result is housed in a single, space-saving 42U rack. The system runs DB2 9.7 on SUSE Linux. It has four Intel Xeon E7-8870 processors running at 2.40GHz (4 processors/40 cores/80 threads). It should also be noted that the system uses Solid State Drive (SSD) storage for faster database access.

For me, the most interesting aspect of this result is not just the performance; it is the price for that performance. And, of course, price/performance is a key consideration for all systems, but especially for cost-conscious x86-64 purchasing decisions. The new system costs US$0.59 per tpmC. As of today, this is the lowest cost of any system in the Top 10 TPC-C performance results (by the way, the next lowest cost also features DB2). See for yourself at TPC-C – Top Ten Performance Results.

If you look at the TPC-C – Top Ten Price/Performance Results, you will see some results from Oracle that offer better price/performance. However, these Oracle results are for very small benchmark systems; approximately one-tenth the size of the IBM DB2 systems. And they include only Oracle Database 11g Standard Edition One; whereas the IBM results include the full enterprise edition of DB2. Not only do the IBM benchmark system give you more product capability for your money, but you can clearly see that the performance of the IBM systems and the cost per transaction for the IBM systems both scale up very nicely.

IBM System x®3850 X5 (Intel Xeon E7-8870 processors 2.40GHz, 4 processors/40 cores/80 threads) TPC-C result of 3,014,684 tpmC, $.59 USD/tpmC, available 9/22/11, DB2 9.7, SUSE Linux® Enterprise Server 11 (SP1)

Written by Conor O'Mahony

August 15, 2011 at 10:25 am

Oracle uses 9x CPUs to Achieve only 3x the Performance for TPC-C Benchmark

with 5 comments

According to IDC’s latest server market share report, released earlier this week, Oracle is languishing in fourth place with 6.6 points of market share (IBM has 30.5 points of market share). When you consider that Oracle/Sun has lost server market share in each of the past seven quarters, you have to imagine that Oracle are desperate to stop the rot. Well, today our friends at Redwood Shores attempted to stem the tide by announcing a new TPC-C benchmark result for a cluster of SPARC systems. However, the benchmark result is far from impressive. Sure, the benchmark system has a huge throughput. However, it is woefully inefficient.

Today’s Oracle benchmark result uses 27 64-core Sun SPARC T3-4 servers to process more than 30M tpmC*. In contrast, IBM’s most recent clustered TPC-C result uses 3 64-core IBM Power 780 servers to process more than 10M tpmC**. There are many ways to look at this. You could claim that Oracle uses nine times the number of CPU cores to achieve only three time the performance. Alternatively, you could claim that each CPU core of the IBM system is able to achieve three times the performance of a CPU core in the Oracle system. Either way, in my opinion, it points to a very inefficient benchmark run. Such inefficiencies are surely a concern for customers who are paying for Oracle Database based upon the number of CPU cores in their systems.

At first glance, the cost efficiency of this new benchmark system from Oracle may appear to be impressive—their system costs 1.01 USD per tpmC. However, if you scratch below the surface, you will find that number is quite deceptive. Oracle do not use the perpetual licenses that you would expect, and Oracle do not use the kinds of support contracts that you would expect. If they did use the licenses and support contracts that are most commonly used, then the system costs would skyrocket, and the relative cost inefficiencies of this system would be plain for all to see. For prior coverage of Oracle’s price/performance tactics, see Sun and Oracle TPC Price/Performance Tactics Revealed.

Also, you should be aware that Oracle have once again resorted to sacrificing data integrity for performance in its benchmark systems. They have turned off page integrity checking—I imagine because, according to the Oracle documentation, it incurs a performance degradation of between 1% and 10%. So, even though it is highly unlikely that you would run a production system without page integrity checking, Oracle has chosen to do just that in the interests of squeezing extra performance out of its system.

Given all this context of misleading cost information and questionable system settings, it was timely to read the following article yesterday… Larry Ellison Hearsay: “We Can’t Be Successful if We Don’t Lie to Customers”

Results on Transaction Processing Performance Council Web site at Results as of 12/02/10.
* Oracle SPARC SuperCluster with T3-4 Servers (27 x 64 core) (108 chips, 1728 cores, 13824 threads); 30,249,688 tpmC; $1.01/tpmC; available 6/1/11.
** IBM Power 780 cluster (3 x 64 core) (24 chips, 192 cores, 768 threads); 10,366,254 tpmC; $1.38/tpmC; available 10/13/10

Written by Conor O'Mahony

December 2, 2010 at 3:57 pm

IBM DB2 has the Leading x86-based TPC-C Result

with one comment

As of yesterday, IBM DB2 has the best x86-based TPC-C result in the industry. IBM’s x86-based system achieved more than 2.3 million transactions per minute on TPC-C benchmark. The benchmark uses the IBM System x 3850 X5, which uses the latest Intel Xeon 7500 Series processor technology. And importantly, the benchmark system is cost efficient, with a price of $0.64 USD per tpmC.

The x3850 X5 achieved this tpmC result using DB2 9.7 and SUSE Linux Enterprise Server 11 (SP1). The configuration included 1.5TB of memory using the IBM MAX5 for System x and eXFlash SSD storage. MAX5 is an industry-first technology from IBM that decouples the memory from the processor, eliminating the need to buy another server to support memory-intensive workloads. Basically, it allows you to increase the system’s memory and storage, which in turn supports larger, faster databases.

For more details, see the benchmark result.

Written by Conor O'Mahony

November 17, 2010 at 11:02 am

IBM Again Shatters World Record on Two-Tier SAP SD Benchmark

leave a comment »

IBM today announced a new two-tier SAP Sales and Distribution (SD) benchmark result that crushes the Oracle/Sun results. The new benchmark result uses a 256-core Power 795 system with DB2 to support 126,063 SAP SD benchmark users. This is more than three times the number of users than Oracle’s largest system (a 256-core Sun SPARC Enterprise M9000 running Oracle Database). The IBM system was also able to handle more than three times the number of SAP SD users than Oracle’s result from September of this year that runs four clustered 32-core Sun Fire X4470 servers.

This new result is also the first to break the 500,000 SAPS level on a single system with more than 688,000 SAPS. The SAP Application Performance Standard (SAPS) is a measure of system throughput for business deliverables like customer sales orders or invoices. This result follows hot on the heels of last month’s result from IBM, which used a 128-core Power 795 system running DB2 to support 70,032 users. If you look at the effect of doubling the number of CPU cores from one benchmark to the next, you can see that the number of users supported is almost doubled. This near linear scaling is very reassuring for organizations whose SAP environments are growing significantly.

This benchmark result indicates that IBM can support your largest SAP systems, and that IBM can support the growth of those systems in a very efficient manner. But don’t forget that IBM Power Systems have tremendous virtualization capabilities. In fact, the latest optional PowerVM virtualization software allows customers to run more than 1,000 virtual servers on a single physical system. This means that you could also use a system like one in this benchmark to consolidate your various existing SAP environments onto one physical system, vastly simplifying your infrastructure and reducing costs in data center floor space, energy, management resources, and database and web application software.

Written by Conor O'Mahony

November 17, 2010 at 10:41 am

%d bloggers like this: