Site Reliability Engineer – HPC, Observability & Incident Response

About the role

The Institute Technology Innovation Institute (TII) is seeking a Site Reliability Engineer with expertise in high‑performance computing (HPC) and observability. You will help maintain the reliability of large‑scale CPU and GPU clusters, develop monitoring solutions, and lead incident response across compute, storage, networking, and orchestration platforms.

Key responsibilities

• Design, implement and maintain observability systems (metrics, logs, traces) for infrastructure and workloads.
• Create actionable alerts, dashboards and health checks for HPC clusters, Kubernetes, GPU workloads, storage and networking.
• Define service‑level indicators and objectives with platform owners and research teams.
• Detect, triage and respond to incidents, coordinating investigation, escalation, mitigation and service recovery.
• Participate in on‑call rotation, improve operational readiness with runbooks and escalation procedures.
• Diagnose complex failures including GPU errors, scheduling issues, storage bottlenecks and network faults.
• Conduct root‑cause analysis, blameless incident reviews and implement corrective actions.
• Develop automation for incident enrichment, diagnostics, remediation and recovery.

Required profile

• Proven experience in site reliability engineering, HPC operations or infrastructure operations with incident detection and response responsibilities.
• Strong Linux administration and troubleshooting skills (processes, memory, filesystems, networking, cgroups).
• Bachelor's or Master's degree in Computer Science, Electrical Engineering or related field.

Required skills

• Linux administration
• HPC cluster management (CPU/GPU)
• Kubernetes or Slurm orchestration
• Observability tools: Prometheus, VictoriaMetrics, Grafana, centralized logging
• Programming/scripting: Python, Bash, Go, Rust
• Infrastructure‑as‑code, configuration management, GitOps
• Incident management, escalation, root‑cause analysis
• Optional: GPU telemetry, high‑performance storage, RDMA networking, distributed training, AWS, GCP, Azure