The job
Senior Site Reliability Engineer (Disaster Recovery
About the Role
SunCore Digital is seeking a hands-on Senior Site Reliability Engineer (Disaster Recoveryr to assess, implement, test, and document service and data recovery capabilities.
This is an engineering position, not solely a policy, compliance, or coordination role. The successful candidate will identify recovery risks, build and harden shared recovery capabilities, automate recovery-specific processes where practical, and prove that critical systems and data can be restored. Once system-specific procedures are tested and documented, the owning teams will be trained to execute and maintain them.
The engineer may draw on application developers when recovery improvements require changes to application code, databases, data flows, or system-specific behavior. Recovery procedures must ultimately be understood and maintainable by the teams that own the affected systems.
Responsibilities
Recovery assessment
- Inventory critical applications, services, databases, storage systems, queues, infrastructure, and external recovery dependencies.
- Determine what is currently backed up and how those backups are created, retained, protected, and monitored.
- Identify systems with missing, incomplete, unverified, or person-dependent recovery processes.
- Assess risks related to data loss, service loss, infrastructure failure, configuration loss, credential availability, and third-party dependencies.
- Distinguish configured backups from proven recoverability.
- Identify manual recovery procedures and undocumented knowledge.
- Document confirmed capabilities, untested capabilities, and unknowns.
Backup and restoration
- Design and implement backup improvements for critical data and configuration, then transfer ongoing operation to the designated long-term owner.
- Translate approved recovery, retention, security, and access requirements into backup controls and monitoring.
- Create, test, and harden restoration procedures with system owners, then transfer routine execution and system-specific maintenance to those teams.
- Execute restoration tests using representative environments and data.
- Validate backup completeness and integrity.
- Measure recovery duration and potential data loss during exercises.
- Establish monitoring and escalation so failed or incomplete backups are detected by the accountable long-term owner.
- Automate backup verification, restoration validation, recovery evidence collection, and other repeatable recovery processes where practical; transfer routine operation of system-specific automation to the owning teams after it is tested and documented.
A backup will not be treated as reliable solely because a scheduled job reports success. Restoration must be tested.
Service recovery and failover
- Design and implement shared service-recovery and failover capabilities, transfer supporting shared components to the designated long-term owner, and ensure system-owning teams retain responsibility for application-specific recovery behavior and procedures after handoff.
- Identify the infrastructure, application, database, network, access, and vendor dependencies required during recovery.
- Establish safe recovery sequencing.
- Develop procedures for partial outages, regional failures, infrastructure loss, data corruption, and service dependency failures.
- Implement shared recovery mitigations directly and coordinate application-specific changes with system owners, who retain responsibility for those changes.
- Test restored services for technical functionality.
- Work with QA to validate that critical business workflows and data remain correct after recovery.
- Document conditions under which recovery, rollback, or failover may be unsafe.
Recovery objectives and planning
- Work with engineering and business leadership to document recovery requirements for critical systems.
- Translate approved business requirements into technical recovery capabilities.
- Measure actual recovery performance against defined expectations.
- Identify where current architecture cannot meet required recovery expectations.
- Provide technical options and evidence to support leadership decisions.
- Maintain a recovery dependency map for critical systems.
Final recovery priorities, risk acceptance, and business requirements remain with leadership.
Disaster-recovery exercises
- Plan and execute recovery exercises.
- Develop test scenarios for service, infrastructure, data, credential, and dependency failures.
- Ensure exercises produce objective evidence.
- Record results, defects, recovery times, data-loss observations, and unresolved risks.
- Implement shared recovery corrective actions and coordinate system-specific corrective work with the owning teams.
- Repeat exercises after material changes.
- Ensure procedures can be followed by qualified staff who did not author them.
Documentation and knowledge transfer
- Establish recovery runbook standards and create initial system-specific runbooks with the owning teams.
- Document required access, tools, credentials, dependencies, procedures, validation steps, and escalation paths.
- Clearly label untested procedures.
- Train system owners and relevant operational staff to execute and maintain the system-specific procedures they own.
- Reduce reliance on undocumented individual knowledge.
- Ensure application teams understand their ongoing recovery responsibilities.
- Maintain shared recovery evidence and exercise results; system owners maintain the accuracy of their service-specific runbooks.
Security and collaboration
- Work with Security Operations to review recovery implementations that affect production access, sensitive data, credentials, networks, infrastructure, or security controls.
- Work with SRE on service dependencies, failure behavior, observability, and operational response.