Customer Profiles
Getting to the Core of Backup Problems
By eliminating network congestion, Force10 curbed the unpredictable backup times hurting this insurance firm’s productivity.
On those occasional nights when the backup process for Tower Hill Insurance Group’s policy management application took a full 16 hours, employees arriving for work would find themselves locked out of the application. "No one could do anything," says Colin Hines, Tower Hill’s Senior Administrator of network and security infrastructure. "It was difficult to explain to our executive group that we couldn’t reliably predict how long the backup would take."
Based in Gainesville, FL, and with offices in Kentucky, Tower Hill is a leading property and casualty insurer providing coverage for homes, rental properties, personal automobiles, and commercial property to 500,000 policy holders in Florida. The company has a Web-enabled IT and billing infrastructure that supports its 400 employees and enables it to attract new agents and cost effectively scale its business.
Each night Tower Hill backs up data relating to policies and claims — about 4 Terabytes. With the total volume of company data doubling every two years and hitting 15 Terabytes this year, the strain on the network began to show. In particular, the time needed to backup its business-critical policy management application became unpredictable.
To ensure data integrity, IT first backs up the application to capture daily changes, then runs an update process that inserts information such as policy renewal dates, and then backs up the application again. While the update process completed in a predictable timeframe, each backup took anywhere from four to seven hours. On average, the three-stage backup and update process required 13 hours. When backup times ran longer, employees arriving for work were locked out of the system.
Ripple Effects
The network became the first thing the server group blamed for the varying backup times, Hines notes, since they could easily check out their equipment. Hines, on the other hand, was spending 25 to 30 percent of his time monitoring and troubleshooting the network, and finding work arounds to congestion problems. When he determined that the volume of data coming into blades on his two core routers was greater than the blades’ connection to the routers’ backplanes, he helped the server team work around the oversubscription problem by balancing where servers plugged into the network.
It bought them some time, but it meant Hines needed to direct the server team each time a new server was brought on line. The unpredictable backup times impacted IT in other ways, as well. Routing maintenance and software releases, for example, had to be done on weekends as it was impossible to do them at night.
With traffic volumes continuing to escalate, Hines knew something had to give. "Our core couldn’t handle the amount of data we were pushing through it," he says. "And I want the network to be something you can just plug into and it goes." It was time to upgrade the network core.
Wanted: Performance and Reliability
When Hines began evaluating new core devices, he knew what he wanted. "We buy best in breed," he notes. "We needed line-rate throughput, non-blocking Gigabit interfaces, and access lists. And we needed reliability."
Tower Hill’s search for a new core router led them to Force10. Hines installed a Force10 TeraScale E1200 switch/router in early 2005 and has watched the combined time for both pre-and post-update backups of the policy management application drop to a consistent 2.5 hours. What used to be a 13 hour or more backup and update process is now routinely done in five.
"One of the really attractive things about the Force10 E1200 is that it’s full rate line speed, non-blocking. It’s not oversubscribed, as were other core switches I evaluated," says Hines.
Each server now has true Gigabit access to every other server and device on the network. In addition, the E1200 supports active redundant links with immediate failover if one link fails. As a result, each Tower Hill server with dual-Gigabit interfaces has a 2-Gbps connection to the core whereas the previous core routers only supported on active connection in a dual-homed configuration.
In addition, the E1200 is a fully redundant system, ensuring continuous uptime for the network. And Hines has been impressed with the level of support Force10 provides. "We’re a moderate-size business," he notes. "I’ve never had support this excellent."
Core Benefits
With a line-rate, congestion-free core switch/router in place, Tower Hill is assured of predictable performance from the network. As a result, backup times have been dramatically reduced, ensuring critical applications are always available during business hours and freeing IT staff to perform routing maintenance, software releases and other tasks in the evening rather than wait until weekends. IT’s management overhead has also dropped, as the burden of ongoing network monitoring and troubleshooting has been eliminated.
"Force10 gave us the ability to exonerate the network," Hines says. Tower Hill now has a predictable network with plenty of headroom to accommodate ever increasing traffic loads.
Go