Software Engineer - Hosted Model Infrastructure

Technologies We Use

Different backend languages, including Java, Rust, Python and Go

Model serving engines for GPU-accelerated inference

Docker and Kubernetes for containerization and orchestration

Industry-standard build tooling, including Gradle and GitHub

Core Responsibilities

Building high-performance model serving infrastructure that integrates with security models, hardware constraints, and different inference engines

Designing intelligent request handling including authentication, rate limiting, concurrency control, and audit logging for multi-tenant model access

Building and maintaining packaging and deployment pipelines enabling fast, secure, and reliable model rollouts across on-premises and air-gapped environments

Developing observability for production AI systems to enable easy service monitoring and fast incident triage and resolution

Debugging complex issues and performance problems throughout the stack, including open source inference engines, container runtimes, and GPU drivers, in environments you cannot always access directly

Designing and running testing and benchmarking infrastructure that validates model deployments across varying GPU hardware before they reach production

Working with product teams and customers to understand requirements, debug production issues, and deliver the models and capabilities they need

Integrating hosted model infrastructure with Palantir's deployment, configuration, and identity systems

What We Value

Ownership mindset and bias toward quality. Our software runs in environments where direct access for debugging is limited or unavailable.

High empathy for customer needs and drive to deliver reliable, easy-to-use models

Ability to work effectively across multiple languages and layers of the stack, from backend services and ML tooling to container orchestration and deployment configuration

Strong debugging skills and motivation to trace problems from application code through containers, orchestration, and hardware

Curiosity about emerging AI capabilities and the ability to quickly evaluate and integrate new models and technologies as the landscape evolves

Active US Security clearance, or eligibility and willingness to obtain a US Security clearance is beneficial, but not necessary

What We Require

4+ years of professional software engineering experience building and operating production systems

Engineering background in Computer Science, Mathematics, Software Engineering, Physics, or similar field

Strong coding skills with demonstrated proficiency in programming languages, such as Java, C++, Python, Rust, or similar languages. Familiarity with the Python ML ecosystem is valuable.

Experience with containers, Kubernetes, and deploying backend services in production environments

Strong written and verbal communication skills and ability to iterate quickly with teammates, incorporating feedback and holding a high bar for quality