The Information Technology Services Division should regularly review and update its Disaster Recovery planning documentation to enhance the division’s ability to ensure continuous service of critical IT components.
Read Full Report
The Office of Internal Audit (OIA) conducted a management audit of the Disaster Recovery (DR) Plan prepared by the Information Technology Services Division (ITSD), a division of the Department of Finance and Administrative Services (DFAS). The audit was included in the fiscal year (FY) 2013 approved audit plan.
Having a current and reliable DR Plan is a necessity for all municipal governments. A well-designed DR Plan is a tool for ensuring that required technology is available on an ongoing basis. Although a plan cannot anticipate all potential threats, DR planning provides a framework for managing the disruption through damage assessment and activation of recovery processes when the unexpected occurs.
ITSD’s current DR Plan was completed in February, 2012. This version of the plan was a complete re-write of previous DR plans and was based on City-wide critical IT needs. ITSD utilizes a layered approach to prevent data loss. For critical systems, the first line of defense is redundancy. Redundant systems located at hot sites provide failover capability. Secondary protection is provided through daily performance of backups to external media.
The DR Plan is currently undergoing testing through a series of User Acceptance Tests (UATs). Each test is designed to test a specific recovery procedure. If any part of a test fails, the methodology calls for re-testing to ensure that recovery steps will be successful in an emergency.
Does the Disaster Recovery Plan permit ITSD to sufficiently recover critical IT systems to allow City Departments to resume normal functions after a disruption?
- ITSD has not obtained step-by-step recovery procedures from three City departments describing services to be provided by ITSD in an emergency. The lack of detail may negatively impact recovery efforts or interrupt availability of critical services required by external departments.
- Two servers that support Tier 1 internal service priorities are located at the secondary hot site, which is limited to approximately 45 minutes of backup electricity if utility power is unavailable. Should an electrical outage exceeding 45 minutes occur, these servers would not be available for failover.
- A division-specific emergency purchasing procedure does not currently exist. Should a large dollar server or component require replacement, ITSD may not be able to acquire the required equipment in a timely fashion.
- Due to employee retirements, transfers and resignations, the DR Plan contact information was outdated at the time the audit began. In a disaster scenario, the lack of current contact information may delay the assembly of an appropriate recovery team.
- The maintenance interval for updating the DR Plan has not been formally established. Without ongoing DR plan updates, recovery of essential systems may be delayed or outdated recovery steps may be attempted.
Have backup and recovery procedures been established and tested to ensure availability of data?
ITSD does not currently perform formal tests of recovery from backup media. Should a disaster necessitate recovery from backup, restoration of critical services may be delayed beyond the 24- hour recovery time objective.
Is the Disaster Recovery Plan tested regularly to ensure that key IT systems can be effectively recovered?
Current DR Plan testing does not consider the unexpected. Without periodic drills, recovery personnel may lack preparation to quickly execute recovery procedures under unusual or unforeseen circumstances.