On January 12th, 2007 at 4:31am, Bob Turley, CIO of the iPremier Company, received a panicked phone call from his IT operations staff. Their external facing website was “locked up” and could not be accessed by anyone, including their customers. iPremier is a web-based business that generates revenue through solely processing online orders. While the web server was down, the company could not accept any new orders or allow their customers to view their products. An inadequately third-party managed and configured router/firewall allowed hackers to execute a DOS (Denial of Service) attack on iPremier. I recommend purchasing a new firewall solution that will be managed and configured by the Company’s IT staff internally. This level of control will allow the company to tailor the level of security they desire and give them the ability to mitigate threats accordingly.
Summary of Facts
At 4:31am on January 12th, 2007, Bob Turley (CIO of iPremier) received a panicked call from Leon Ledbetter in operations. Leon stated that the Company’s website was down and that customers could not access the site. He also stated emails containing the phrase “ha ha” were being received by the mail server. The Company’s technical operations team leader, Joanne Ripley called Mr. Turley at 4:39am stating she could not access their equipment from the line to their office and indicated she was driving toward the Qdata co-located facility.
She confirmed that Qdata (their hosting company) claims there was not a connectivity issue in or out of the building. Ripley stated the she would try to restart the web server once she reached the facility and confirmed she had an outdated copy of emergency procedures in her vehicle. Shortly after Mr. Turley hung up with Ripley he received a call from Warren Spangler, VP of business development. Mr. Spangler was concerned with the affect on the Company’s stock price, calling the police or FBI, and basic PR issues resulting from the incident. After Mr. Turley hung up with Spangler, he received a call from Ripley stating that Leon from operations notified Mr. Spangler of the incident. Turley then proceeded to call Tim Mandel, the Company’s CTO. Mandel advised Turley not to pull the Internet connection so further logging could be obtained. He also disclosed advanced granular logging would not be available due to space constraints from finance issues.
He also received a call from Peter Stewart, the Company’s legal counsel, advising him to pull the plug on the Internet connection. Stewart advised that Jack Samuelson, the Company’s CEO, asked him to provide legal advice on the matter. Ripley also called in and reported she could not access the NOC due to knowledge and staffing issues at the Qdata facility. At this point Samuelson called Turley directly and advised that his main concern was getting the Company back online and reiterated that Turley should not worry about any PR issues at this moment in time. Shortly after, Ripley obtained access to the hosted firewall and determined the shutdown was due to a SYN flood type of DOS attack. A SYN flood occurs when “external hosts attempt to overwhelm the server machine by sending a constant stream of TCP connection requests, forcing the server to allocate resources for each new connection until all resources are exhausted (Lemon, 2002).
Ripley attempted to block access from the originating IP address but quickly learned that zombie machines were being used in the attack, rendering this approach useless. He hung up with Ripley then received another call from her at 5:46am stating that the attack suddenly stopped. Ripley confirmed the web site was back online and the business was running as usual.
This DOS attack prevented iPremier from selling products, and letting customers view products, on their website. This is the sole presence of the business, and when the website is down, the company cannot generate revenue. All responsible parties and managers were quickly involved and attempted to mitigate negative consequences to the company. There appeared to be a slight disconnect between the legal and functional leadership advice across the board. The website was only down for a little over an hour and iPremier states there was no substantial impact to the business at this time. Problem and Alternatives
This denial of service attack occurred due to inadequate firewall configuration and management. This problem was further amplified by the fact that the firewall service was hosted by a third-party vendor, Qdata. iPremier recognized staffing and general IT knowledge and management issues with Qdata in the past, but chose not to act upon their discoveries. iPremier did not have any active monitoring of the firewall and only knew there was a series business operating issue only when the web server was fully unreachable and unresponsive.
One solution is for iPremier to purchase their own firewall where they can apply the proper configuration to prevent further attacks. This level of management will also provide them with advantages where they could fully monitor the device and to setup alarms indicating when there is a potential issue. iPremier will have full control over the device allowing them to customize the level of security they desire. The drawback is the knowledge necessary to adequately configure and maintain the device.
Another solution is to continue their firewall service with Qdata after a complete audit was performed. Any weaknesses in the security design would have to be addressed and some type of guarantee of service would have to be put in writing. I would also recommend that iPremier obtains some type of service level agreement from Qdata to ensure a prompt response time during an incident. Some type of contingency plan must also be put into place allowing them to immediately access the device locally and remotely during a serious issue.
The advantage here is keeping the current provider which would not require them to make any network or equipment changes. The obvious disadvantage would be continuing service with a company who has already failed them in the past showing poor credibility.
A third solution would be for iPremier to switch to a firewall and security provider who would provide a high level of service for a fee they would feel comfortable with. The same type of access requirements would be required during a serious issue to allow iPremier access when deemed necessary. “Shopping” around for a vendor would allow them to heavily research best in industry providers with a proven track record. The advantage here is obtaining a top-level service provider with the right “know how” but ultimately iPremier would have to reconfigure some equipment and make network changes. According to Allen, Gabbard, May, Hayes, & Sledge (2003), using a managed service provider is a viable solution for distributing security operations and responsibilities where the organization still owns the associated risks, but allows sharing and mitigation of the risks. Decision and Conclusion
Regardless of which solution iPremier chooses to implement, they must perform a full audit to uncover exactly what allow this DOS attack to occur. We know the weakness was in the firewall, but knowing the root cause will allow iPremier to use this tragedy as a learning experience to build upon in the future. Joanne Ripley seems competent and willing to implement a “better” solution and I recommend engaging her in all discussions involving this incident moving forward. Internal control of a firewall solution will allow iPremier management to get as involved as they want to be. The CTO and CIO could also potentially pull reports from the device allowing them to make future security planning decisions. According to Applegate, Austin, & Soule, (2009), switching IT systems can become difficult and costly once it is ingrained into day-to-day activities.
This aspect of the security solution should definitely be examined and taken into consideration. Allowing management and IT staff to be involved in the Company’s security solution will help keep up security awareness in the workplace as well. One person may miss a potential threat that another staff member could recognize. Someone as plugged in and competent as Ripley should only be allowed to make changes to the device. Staff should only have enough access to perform the necessary task at hand. This would typically only involve read-only access to the security devices. iPremier conducts all of its business through their website and they cannot afford for it to be down for any amount of time. References
Allen, J., Gabbard, D., May, C., Hayes, E., & Sledge, C. (2003). Outsourcing managed security services (No. CMU/SEI-SIM-012). CARNEGIE-MELLON UNIV PITTSBURGH PA SOFTWARE ENGINEERING INST.
Applegate, L. M., Austin, R. D., & Soule, D. L. (2009). Corporate information strategy and management: Text and cases. Boston: McGraw-Hill Irwin.
Lemon, J. (2002, February). Resisting SYN Flood DoS Attacks with a SYN Cache. In BSDCon (Vol. 2002, pp. 89-97).