In the Media

By Scott St. John

Think back to 2009. The U.S. was in the throes of The Great Recession and the first signs of hope had just begun to appear. In February of 2009, Congress passed the American Recovery and Reinvestment Act. Four days later, the president signed it into law making $787 billion (later increased to $840 billion) available to spur economic recovery, including $26.5 billion for broadband and other infrastructure projects.

In June of 2009, Pew Research published the Home Broadband Adoption Report which indicated a nearly 10 percent rise in home broadband adoption from the previous year across most socioeconomic classes. Thirty-four percent of the respondents cited they had subscribed to broadband services for “faster speeds,” which was up from 29 percent the year before. Being retrospective, the significance of the data is that even in the darkest hours of the American economy in recent history, Americans were still willing to pay more for bandwidth. This growing demand for connectivity was further fueled by the launch of the iPad in 2010; the advent of tablet computing, mobile apps, and the whole shebang. Within a year, broadband connections boomed to an estimated 600 million globally.

Fast forward just a couple years to 2012 and chuck in some things like LTE, 4G, OTT, M2M and several reiterations of the iPad, iPhone, Android, Windows Phone, and other bandwidth-hungry devices. These factors created an explosion in consumption, pervasive connectivity, an almost an infinite combination of connectivity, services, and devices. Communications Service Providers (CSPs) are now faced with an unprecedented volume, complexity, and demand for broadband data services which may be great for business but not so good for legacy systems. Scalability now moves to the forefront of CSPs’ concerns and the need for a reliable reference point at which they can gauge the rigidity of their infrastructure becomes essential.

Karl Whitelock, Director OSS BSS Strategy, Stratecast Frost & Sullivan commented:

“The exponential growth of traffic that the industry is experiencing has been phenomenal, which increases the urgency for CSPs to interrogate the capabilities of existing transaction processing systems. The role of benchmark tests, which examines the most stringent operating environments and scenarios, are needed in today’s market in order to provide CSPs with a true picture of how their future investments in these and related business support systems will address their business strategies.”

The Rise of Benchmarks

According to the American Heritage Dictionary, a benchmark is defined as, “a standard by which something can be measured or judged.” Historically, benchmarks are literally “marks” cut into stone for the placement of angling rods so that surveyors could have an accurate and definitive point of reference. I like this definition better. CSPs need a definitive and accurate point of reference by which they can evaluate the performance of communication technology products.

Being a bit visionary and true to their namesake, Oracle announced their advocacy for establishing industry benchmarking standards in January of 2009 and quickly got behind the TM Forum’s (TMF) budding benchmark program. In a press release, they announced that their Communications Billing and Revenue Management application had achieved a benchmark of supporting 100 million subscribers. Liam Maxwell, vice president of products for Oracle Communications was quoted as saying:

“Communications service providers worldwide are increasingly concerned with business support system (BSS) performance and scalability. To ensure that service providers are best able to evaluate competing products, Oracle is strongly advocating for an industry benchmark standard – particularly one that builds upon the TMF’s work to define BSS processes by formalizing the objective evaluations of billing and revenue management applications on performance and scalability, among other factors. Oracle is ready to meet this challenge.”

To which Martin Creaner, president of the TMF added: “This industry has long lacked standard benchmarks to help communications service providers validate marketing claims, and measure and compare the performance and scalability of billing solutions. We applaud Oracle’s leadership in championing a billing and revenue management benchmark standard for the industry. This will complement the existing TM Forum Billing Benchmark Program, and we are excited to work with Oracle to deliver this value capability to our members and the industry.”

The 100 million subscriber benchmark announcement all but solidified Oracle’s leadership position as the de facto choice for CSPs at the time. In hindsight, it pokes a little at Oracle’s former relationship with IBM, which has since soured. This is best illustrated by the recent $10 million prize issued by Oracle CEO Larry Ellison for anyone that could prove that IBM could run database software even half as fast as Oracle’s. The benchmark was actually conducted and achieved on IBM hardware in IBM’s Labs in Beaverton, Oregon.

Not to be outdone, at Mobile World Congress this year, industry Goliath and IBM pals Amdocs announced that they had achieved a new benchmark of 220 million subscribers and touted it as the, “the largest benchmark customer base ever reported.” Clearly raising the bar, and more than doubling the previous announcement by rival Oracle.

The Dark Horse

Then an unexpected announcement resonated above much of the benchmark noise. Billing innovator and up-and-comer, Redknee not only jumped into the race, but took the pole position with partner Microsoft, by announcing breakthrough support for 250 million subscribers. This benchmark also raised the bar by showing near linear scalability at peak performance, processing an average of 1,249
invoices per second. The benchmark was conducted with Intel, Microsoft, NEC, Redknee, and X-IO Technologies. This combined solution establishes a leadership position by proving top-tier scalability for the combined solution--a place typically reserved for bellwethers Oracle and Amdocs.

“In today’s marketplace, it is typical for one customer to have multiple subscriptions. This is only going to become more prevalent as we see 4G/LTE networks roll-out, which will support new connected devices and the associated applications and the ecosystems that these create,” commented Lucas Skoczkowski from Redknee. “Ticking the ‘number of subscribers’ box is a first, but a crucial element of scalability that Tier-1 service providers need to consider. Agility, configurability and lower TCO must also be taken into account.”

The test environment consisted of 15 quad-core NEC servers using Intel Xeon 7300 processors running the Redknee Turnkey Converged Billing solution. The single SQL Server 2012 database server consisted of a NEC Express5800/A1080a enterprise server loaded with 8 Intel Xeon E7-8870 CPUs that contain 10 cores each and 1TB of memory. Hyper-threading was enabled which allowed SQL Server 2012 to utilize 160 threads of processing capacity providing a 25-30% performance gain (Fig. 1).

(Fig. 1 Benchmark testing environment)



But what’s unique and interesting about this benchmark is that to achieve it, a number of technical innovations had to be incorporated including self-tuning database storage with Microsoft SQL Server 2012 and data compression by X-IO Technologies in addition to optimal hardware and software performance. This lead to significant, additional efficiencies including:

• Significantly reduced physical Input/Output (IO) Operations and workload, therefore contributing to lower hardware costs
• Significantly reduced data center footprint due to storage products that seamlessly meld HDDs and SSDs
• 80 percent reduction in storage space through compression
• 70 percent reduction in database license costs, compared to the license costs of other leading database providers

Karl Whitelock, director OSS BSS strategy, Stratecast Frost & Sullivan commented:

“For CSPs to capitalize on next-generation broadband services and business models, enabled from investments in LTE and fiber-based transport capacity, they must look to the advantages of a high performance, multi-tenant, and scalable billing solution. These solutions are essential to enable personalized pricing plans and to deliver a high-value customer experience. This benchmark test validates that a real-time converged billing solution can manage today’s demands and tomorrow’s business expectations.”

The Beef with Benchmarks

While their intent is good, benchmarks don’t provide the complete picture. In fact, actual geographic benchmarks don’t provide the complete picture either, but they do provide a reference point which surveyors can use to better analyze the landscape. It’s really no different for communications information technology (CommIT). The benchmark should be taken into consideration with other key factors to provide an objective analysis of CommIT products.

If we compare the benchmarks from Amdocs and Redknee, there are a myriad of factors beneath the surface that can be scrutinized if one were to actually pop the hood and look. But, the net difference between benchmarks is only 30 million subscribers. And what’s 30 million subs between friends? Let’s face it too, if you throw enough hardware or money at any benchmark, you’re bound to eventually reach it. Right? So what’s the value of a benchmark if you have no point of reference to provide it with meaningful context?

CSPs need another metric to serve as a reference point to contextualize benchmarks. CSPs need a way to understand how the benchmark relates to their respective environment (and budget). The number of subscribers supported or achieved, by itself, is not enough. One way CSPs can better grasp the relative scalability of a benchmark solution is by understanding the total cost of ownership of the benchmark environment.

The 250-million-subscriber benchmark recently achieved by Microsoft and Redknee was reported (by Microsoft) to have been conducted at 50 percent of the hardware and software costs when compared
to the industry standard. This savings was achieved by Microsoft’s SQL Server’s lower licensing and maintenance costs, and its ability to run on standard Intel Xeon boxes. Significant savings were also achieved by lowering overall management costs, lowering storage requirements due to database compression, and the linear scalability of the solution.

Microsoft’s Managing Director for the Telecommunications Industry Johan Norvik remarked, “We are thrilled by the results of the Redknee benchmark on SQL Server 2012 that further validate SQL Server as the data platform on which you can bet your business, as it helps deliver predictable, linear, and next-generation performance. However, the key value that the solution delivers is to help our joint customers win more customers and to scale the business while achieving a low TCO.”

“Another important metric is how the system can scale. We demonstrated, in this benchmark test, that we can scale the solution to 250 million subscribers with near linear scalability,” added Lucas Skoczkowski from Redknee. “This is important for communication service providers because it gives them the assurance that they will have predictable performance as the solution scales, as well as providing a true understanding of the total cost of ownership over the lifetime of the solution. These are key metrics that service providers are assessing when looking to make investments in their back-office systems.”

With the explosive demand for broadband services and the seemingly infinite combination of services and billing models, scalability is at the forefront of CSPs’ concerns. However, while scalability may be the goal, savings may be the prize as the solution that supports the most subscribers at the lowest total cost of ownership enables both scalability and profitability. Profitability that can be used to fund innovation initiatives, promote competitive differentiation, improve the customer experience, and even subsidize billing transformation projects.