Major Incident Manager

What you will do

System Health Monitoring, Alert Triaging, and Error Budget Management: Dedicate time to monitoring critical security infrastructure (e.g., identity platforms, firewalls, compliance systems) and core infrastructure components. Focus on using and maintaining dashboards tied to Service Level Objectives (SLOs) , triaging high-severity alerts, and analyzing the current Error Budget burn rate to guide prioritization for the rest of the day.

Infrastructure as Code (IaC) and Policy as Code Development: Spend the largest portion of time writing, reviewing, and testing code (e.g., Python, Go, Terraform, or proprietary tools) to automate the deployment, configuration, and security hardening of infrastructure. This involves treating infrastructure and security policies as software to ensure consistency and prevent configuration drift.

Toil Elimination and Automation of Operational Tasks: Identify, scope, and implement automated solutions for manual, repetitive, and time-consuming tasks (toil) related to security patching, compliance checks, certificate rotations, or infrastructure maintenance. The goal is to continuously reduce the operational workload for the team.

Security Pipeline and Observability Maintenance: Maintain and enhance the DevSecOps security tools integrated into the CI/CD pipelines (e.g., static analysis, vulnerability scanning, security configuration checks). Ensure the end-to-end logging, metrics, and tracing (observability) systems for both infrastructure and security tools are robust, accurate, and provide immediate diagnostic capability during incidents.

Incident Response Engineering and Post-Mortem Action: Participate in the on-call rotation and actively engage in engineering solutions derived from post-mortems. This means turning incident root causes into preventative measures implemented via code, improving runbooks into automated actions, and reducing Mean Time To Resolution (MTTR) for future incidents.

Experience and qualifications

Deep IaC Expertise: Professional experience provisioning and managing complex infrastructure using tools like Terraform or CloudFormation (AWS), or similar tools like Ansible or Puppet for configuration management.

Cloud/Platform Experience: Hands-on experience with a major cloud provider (AWS, GCP, Azure) or managing large-scale internal/private cloud infrastructure.

SLO/SLI Implementation: Practical experience defining, measuring, and reporting on Service Level Indicators (SLIs) and Service Level Objectives (SLOs) for critical services.

Logging/Metrics/Tracing Stacks: Proven experience with modern observability platforms (e.g., Prometheus/Grafana, ELK/EFK stack, proprietary systems, or vendor solutions like Datadog/Splunk) for proactive issue identification.

Networking: Strong understanding of core networking concepts (TCP/IP, DNS, Load Balancing, Firewalls, Proxies) sufficient to debug complex service connectivity and latency issues.

Automation of Security Controls: Experience implementing security best practices via code, such as automated vulnerability scanning, configuration hardening, secret management (e.g., HashiCorp Vault), and key rotation.

Identity and Access Management (IAM): Practical experience managing large-scale IAM systems (e.g., implementing least-privilege policies, single sign-on).

Incident Management: Experience running or significantly contributing to post-incident reviews (post-mortems) and prioritizing resulting engineering work (error budget management).