Blog Post by Stephen Fowler, VP and General Manager
July 6, 2018
Limelight Networks owns and operates one of the largest private backbone CDNs in the world. Connected to hundreds of ISPs and last mile providers, with 80+ POPs across 40+ metro locations and exceptional levels of cache hit efficiency, we enable our customers to:
This network combined with the Limelight software stack and Operating System, enables optimized performance in a secure infrastructure, provides simplified management and delivery of video to any device, guarantees content purge and provides security through integrated DDoS, WAF and the largest SSL footprint.
Sitting above this lies the Limelight product set for Delivery, Video, Storage and Security, enhanced with our Control Self-Service Portal and advanced Edge Analytics capability.
The ongoing development of Limelight’s software products--the addition of new features, functionality, enhancements, look and feel, custom solutions and the ongoing maintenance of products--is obviously a core part of our business. The methodologies we use to get products to market have evolved over time to better suit the needs of our customers, through the exploitation of maturing technologies and through advancement in our integration processes.
As with any business whose value is driven through the features and functionality of their software, Limelight Networks adds value to its customers by providing software solutions and products that enable them to offer exceptional solutions to their customers and end users.
In a competitive landscape, this clearly means that the lifecycle of getting progressive features and functions to the market has time pressures and constraints. It also introduces the challenges of ensuring the features and functions have high levels of quality and stability whilst shortening, wherever possible, the time to get the software into the production environment.
As we will go on to explore, Limelight has developed its QA function across all products to ensure that timescales are optimized, quality is exceptional and stability is at the level expected by our customers who rely on us as a key component of their business process.
One of the attributes that Limelight values is “exceptional teaming.” Regardless of product, the teaming of Development, QA, Operations and the processes we have developed, the learning we embed into those processes and the high standards we set for ourselves is evident across our development and QA teams in locations around the world.
Limelight opened the first Ukraine Technology Centre (UTC) in 2010. This facility, which was established specifically for the formal QA of our software products, was the first step in building the structure that exists today. The challenges faced at that time were many but centered on regression testing that used ill-defined test cases and test plans against a huge library of objects and configurations. Documentation wasn’t where it needed to be and much time was wasted in locating the objects and configurations required. The feedback loop from Production, Operations and Development has always been strong but in those early days, the volume of bugs and errors flowing back was beyond the resources we had in place to handle them. Clearly something had to change.
As a data-driven company, we analyzed the feedback and the challenges steered us toward the early implementation of the approach and techniques that we use today. Key elements of this approach were to avoid the use of huge libraries and third-party web services, and drive towards a highly (99% +) automation of all test cases. The principle adopted was simple; each test case is “self-organized” and able to configure the environment with all required conditions for the case being tested, all of this is achieved in a matter of milliseconds. To support this approach, origin services were built that could be controlled directly by test cases and configured for any HTTP response or non-HTTP responses for negative testing, in addition, the ability to emulate hung, slow or interrupted responses. With this core approach in place, we could now cover all known cases and fully replace the huge library with automated test cases inside a 4-month timeframe.
This approach was then subsequently rolled out to cover Disk I/O, DNS, TCP and other key areas supporting our delivery product.
In the years following, our team grew and our QA philosophy and approach matured rapidly. The structured method now established (including estimates, test preparation, test execution, automation and regression) forms the underlying framework for QA across all Limelight products. In collaboration with the Product Management and Development teams, we provide realistic and achievable commitments to ensure positive cadence and -low-risk releases with minimal rollbacks and consequent customer impact.
Deep, complex, component-level testing for all features is now an embedded characteristic of our process. The challenge this brings in the core components of our network is the variety of inputs required (such as client request objects, configuration files, DNS responses, origin responses etc.) to drive an output response. Every input therefore needs to be controlled to prevent unexpected or “false positive” outputs. This led us to develop the Limelight Test Framework (LTF), which gives us control over all high-level inputs and a stable environment for test execution.
In 2015, Limelight merged all QA groups from core products into a single Core QA organization to ensure the approaches and positive impact felt in our Content Delivery software was mirrored across all Limelight products.
Today, we continue to drive the maturity of our QA approach and have custom emulators for all aspects of our CDN. Examples include HTTP/S flow emulation, TCP behavior emulation, DNS response control, file system emulators, and both complex and standard kernel calls--all of which enable us to underwrite the quality and stability of our software and ensure it provides the best level of service for our customers.
Establishing a single, accountable organization for QA across all software products has delivered great benefit to Limelight’s Development process. The now-standard approach has the benefit of reducing test time for similar software test planning, testing time, and engineer training. It allows us to reuse resources from other pools and to simplify and standardize in our automation activities.
Automation and the process of Continuous Integration is key to our continued evolution. A team of automation engineers is focused solely on test case automation, bringing consistency in both quality and stability. Continuous Integration also allows us to embed approaches in our automation for multiple reuse across a number of projects. Similarly, the number of toolsets used and supported has been reduced and the resultant efficiency gains reinvested in proactive activities. Finally, Continuous Integration as an approach continues to enhance our quality, stability and time to market across the entire product set.
Ensuring that the ongoing investment in improvement gives us a tangible return is key. These improvements can only be achieved with the management and tracking of metrics and KPIs (Key Performance Indicators). Not only do they inform where we need to focus but also gives us a baseline and improvement measurement that ensures that changes we make are effective.
Of all the improvements we measure, the biggest is the noticeable improvement to our customers in the quality, features, stability and reliability of our software products. One aspect of our metrics & KPIs therefore are very focused on outcome-based measurements. Examples include; quality of code as measured during the test / QA cycle, any errors whether they be minor or major, the ratio of releases to any rollbacks and critically, the volume of incidents reported by our customers across our software product set. These metrics, without exception have shown solid improvement in every aspect and are reported at a senior level within the company on a regular basis.
When you pair this with the improvements in test automation, regression, and continuous integration, Limelight sees continued high value in the investments made in QA, automation and of outcome-based measurements and the direct impact on customer satisfaction.
Features, functionality, time to market and stability are all critical to Limelight’s continued leadership and success in the CDN marketplace. These attributes are all embedded in the fabric of our development function--and this function, represented by our QA capability, is at the heart of a progressive drive to continue providing industry leading capabilities to our customers.