Principal AI Infrastructure Engineer - Highrise AI

Hut

United States4d ago

Free account needed to apply or save. It takes a minute, and we'll bring you right back to this job.

Key details

Location
United States
Work style
Remote
Level
Lead / Staff+
Experience
8–11 yrs
Function
DevOps / Infrastructure
Focus area
AI / ML
Posted
4d ago

Skills

ansiblebashchange controlconfiguration managementdefect trackingethernethigh-performance computingkuberneteslinuxpcieperformance optimizationperformance testingpythonroot cause analysissretroubleshooting

About the role

 

 

About Highrise AI 

Highrise AI is a GPU-native infrastructure platform built for modern AI workloads. Designed for developers and enterprises operating in performance-critical and security-sensitive environments, Highrise delivers scalable GPU compute across dedicated clusters, managed cloud, and confidential environments. 

Highrise combines bare-metal performance with full-stack orchestration, API-driven access, and hardware-enforced isolation to support training, fine-tuning, and production-scale deployment of AI models. From reserved GPU capacity to enterprise-grade confidential compute, Highrise provides flexible infrastructure models tailored to the needs of builders operating at scale. 

Beyond infrastructure, Highrise integrates disciplined operations, advanced automation, and on-site engineering expertise to sustain high uptime and predictable performance across deployments. These capabilities extend beyond Highrise-owned fleets: through managed services, Highrise also operates third-party GPU hardware on behalf of customers and partners, applying the same operational standards to infrastructure it does not own. 

Highrise is a subsidiary of Hut 8 Corp., an energy infrastructure platform. Hut 8 integrates power, digital infrastructure, and compute at scale. Highrise serves as the compute and operations layer—designing, deploying, and operating next-generation GPU clusters from commissioning through production. 

ROLE OVERVIEW 

We are seeking a senior, hands-on AI Cluster Integration Engineer to bring large-scale GPU compute clusters from physical installation to validated, commercially ready production. Working alongside a team of senior infrastructure engineers, this person will own critical elements of cluster bring-up, bare-metal provisioning, advanced systems configuration, performance optimization, and acceptance testing. 

After deployment, the engineer will provide Tier 3 technical escalation and sustained performance support for operating clusters. The focus is on compute-platform readiness and workload performance rather than primary ownership of network switch and fabric configuration. 

KEY RESPONSIBILITIES 

Cluster Bring-Up and Systems Integration 

  • Lead technical bring-up of materially complete, racked, powered, and cabled AI clusters through production readiness and customer handoff. 
  • Establish and validate system baselines across BIOS/BMC, firmware, GPU drivers, operating systems, and cluster software stacks. 
  • Deploy and optimize automated bare-metal provisioning, imaging, node configuration, and lifecycle workflows at scale. 
  • Configure and troubleshoot advanced Linux and GPU systems, including kernel, driver, accelerator, and host-level performance settings. 
  • Integrate and validate Kubernetes and Slurm environments in partnership with platform, networking, and operations engineers. 

Performance Engineering and Acceptance 

  • Execute NVIDIA Field Diagnostics, L11/L12 validation, burn-in, stability, and hardware health qualification procedures. 
  • Run and analyze NCCL collective communication tests, GPU benchmarks, and distributed workload performance at node, rack, and cluster scale. 
  • Troubleshoot GPU, NVLink/NVSwitch, PCIe, RDMA, and InfiniBand/Ethernet performance issues, partnering with network engineers on fabric-related remediation. 
  • Identify configuration bottlenecks and optimize throughput, latency, scaling efficiency, and cluster reliability. 
  • Own technical acceptance criteria, test evidence, defect tracking, remediation verification, and readiness recommendations for commercial service. 

Tier 3 Operations and Reliability 

  • Serve as a senior escalation point for complex production incidents involving compute nodes, GPU software, provisioning, scheduling, and degraded performance. 
  • Perform deep root cause analysis, coordinate vendor engineering escalations, and lead corrective actions for recurring or systemic issues. 
  • Support fleet-level upgrades, configuration changes, cluster expansions, and post-maintenance validation with appropriate change controls. 
  • Develop diagnostic tooling, automation, runbooks, and operational handoff material for SRE, NOC, and site support teams. 

Cross-Functional Execution 

  • Work closely with infrastructure engineering, site operations, OEMs, and software/platform teams to resolve cross-stack issues and meet delivery milestones. 
  • Provide concise technical status, benchmark findings, risks, and acceptance recommendations to engineering leadership and stakeholders. 

ABOUT YOU 

Education and Experience 

  • Bachelor's degree in computer science, engineering, or a related field, or equivalent practical experience. 
  • 8+ years of hands-on Linux systems engineering, HPC, cloud, or large-scale infrastructure experience, including senior-level troubleshooting. 
  • Proven experience bringing GPU or HPC clusters into production and qualifying systems against performance and reliability targets. 
  • Strong expertise in Linux, automation (Python/Bash/Ansible), bare-metal provisioning, configuration management, and systems diagnostics. 
  • Hands-on experience with NVIDIA GPU platforms, NCCL, Field Diagnostics, and multi-node performance testing. 

Get jobs like this as soon as they're posted

JobSuited checks company career sites all day and ranks every opening against your profile. Free to join.

Create a free account