7 tips for battle-testing new technologies | CIO
Friday, August 31, 2018
11:11 PM
CIOs offer recommendations for conducting proof-of-concept technology tests, as well as vendor "bake-offs."
Courting new technologies is a risk for IT leaders, who must wisely allocate finite budgets to hardware, software and services they believe will support business operations. Bad bets can set the company back years and ultimately prove catastrophic for CIOs.
AdChoices
ADVERTISING
This is particularly true in the digital era, as a proliferation of newer technologies such as cloud, analytics and machine learning render vendor selection more vital than ever. To select the best tools for the tasks, CIOs conduct "proof-of-concepts" (PoCs), essentially pilot tests in which they model how technologies will run in their organizations.
Frequently these experiments include "bake-offs," or competitions between vendors with similar solutions. Technologies that don't run well are quickly jettisoned in an era where failing fast is fashionable and getting a viable solution into production is the ultimate goal.
CIOs recently shared with CIO.com details about PoCs and bakeoffs they have conducted, along with tips on how other IT leaders should approach battle-testing new tech.
Incumbents vs. upstarts
Shortly after joining AutoNation to run IT infrastructure in 2016, Adam Rasner realized he had a serious data management problem. The car and automotive company suffered thousands of failed backup jobs per month as AutoNation's data growth taxed legacy storage systems.
Rasner initiated a bake-off between one of AutoNations' incumbent storage vendors, as well as two emerging, yet unproven startups. It was a dicey situation with no clear resolution. Rasner knew the incumbent might not scale to meet AutoNations' data demands, while analysts warned him that endorsing either of the two startups, neither of which had many Fortune 100 references at the time, was risky. Even so, Rasner says that he tested the three solutions side by side on the same equipment in an AutoNation data center.
He wrote the incumbent off when it hit a limit that would incur additional hardware investment. One of the startups shot itself in the foot when it falsely reported a backup job as complete but lost the data in the process. Although the vendor patched the problem with firmware, the damage was done. "You have an issue like that it's really hard to get that confidence back," Rasner says. "They couldn't recover from that."
Cohesity emerged as the winner, ultimately reducing the per-terabyte storage cost for AutoNation by 40 percent. Rasner says he's since extended use of Cohesity technology to secondary storage for document management, and plans to boost his investment in the technology as he scales the company's infrastructure. Cohesity, Rasner says, has met AutoNation's demands every step of the way.
Tip: Vet your vendors
Before inviting new companies for a PoC or bake-off, inquire about their current customers and their financial security, Rasner says. For example, AutoNation sent its procurement team to check the financials of all three vendors financials and report back to IT.
"We've taken a few risks on a few different products, and it's turned out pretty well — you just have to careful," Rasner says. "You want to be innovative and try new things, but you also don't want to put the company's money on the line when it's not rock-solid."
Betting on unproven tech
Zoom CIO Harry Moseley has initiated more PoCs than he cares to count. Not just at Zoom, but also in his previous capacity as CIO of organizations such as Blackstone Group and KPMG. But one proof-of-concept stands out for Moseley: an early test of Amazon Web Services several years before many organizations were even thinking of thinking of endorsing the cloud. His team performed penetration testing and code reviews prior to uploading thousands of documents into AWS. While that bet paid off, Moseley says he's also tested commercial software only to reverse course and build proprietary applications for a fraction of the cost.
In short, the PoC is a test bed with no guarantee of success, intended to let a company discover the art of the possible. Moseley advises CIOs to:
Tip: Assess your risk appetite
CIOs working for companies in regulated industries can't afford to bet the company on unproven technologies for critical business functions. For example, an enterprise working in a heavily regulated field won't able to stomach too much risk. "Some organizations don't want to be on the bleeding edge," Moseley says.
Tip: Influence product roadmaps
But it sometimes pays to give startups a chance, as it affords CIOs the opportunity to get better pricing and potentially influence a product roadmap for years to come. "You have a lot of leverage to steer and guide the platform," Moseley says. "It's akin to having a seat at the table."
Sniffing out snake oil
National Life CIO Tom Anfuso says the insurance company does a lot of its own homework for its PoCs, which recently included a data service platform, as well as new policy administration and access control solutions. Anfuso recommends:
Tip: Beware lightweight MVPs
Anfuso says CIOs should direct the structure of vendor mock-ups in PoCs. For example, some vendors may show little functionality in a minimum viable product, which should immediately raise suspicions; the vendor could be steering CIOs toward a low hurdle simply to get them to buy the product.
In general, CIOs should take care not to use the MVP concept to guide large-scale systems integrations projects. "The graveyard of enterprise IT is littered with the corpses of MVPs that were supposed to become much bigger," Anfuso says.
Tip: Practice good faith in bake-offs
Sometimes enterprises are the ones pulling the rope-a-dope, pitting multiple companies against each other to force a better deal. But Anfuso says bake-offs must be done in good faith, and CIOs shouldn't put any company through its paces unless they are truly interested in them. "You can get a bad name, when you have zero interest selecting [a vendor]," Anfuso says.
The importance of metrics
TiVo CIO Steve Palmucci finds himself regularly running PoCs and bake-offs as he re-platforms IT around best-of-breed tools in the wake of Rovi's acquisition of TiVo two years ago. Palmucci has conducted PoCs for anything from collaboration and communication tools to security, server and storage systems.
Palmucci says CIOs should:
Tip: Loop in the business
Many PoCs are completed within the context of IT, but Palmucci likes to extend PoCs to more functional areas of the business. This gives him a better idea of how a tool will impact a business user. For example, TiVo executives recently gave new video conferencing technology a spin for a staff meeting.
Tip: Measuring is a must
While it's not necessary to have a detailed playbook or checklist to conduct a PoC, IT should introduce structured ways to gauge test results, including quantitative and qualitative measurements. Quantitative testing will help CIOs determine functionality, performance and fit within the business. Qualitative testing takes into account skillsets within IT to administer and manage a technology, as well as how willing a vendor is to provide support if needed.
Vytvořeno v aplikaci Microsoft OneNote 2016
Komentáře
Okomentovat