Sr. Staff Software Engineer, Cribl AI

Join the company that’s building the telemetry infrastructure for the AI era. At Cribl, we partner with IT and Security teams at many of the world’s biggest enterprises, including half of the Fortune 100, to bridge the gap between AI ambition and infrastructure reality. As the AI Platform for Telemetry, we give customers the choice, control, and flexibility to manage and analyze telemetry for both humans and agents, so they can build what’s next.

We’re one of the fastest‑growing private companies and a leading player in a massive, fast‑moving market. With a global workforce, we’re remote‑first and grounded in a simple idea: software is a people business. Cribl is the place where curious, collaborative people can do their best work, grow fast, and bring their full selves to the herd.

Why You'll Love This Role

Join Cribl's AI Systems team and help build the foundational infrastructure powering Generative and Agentic AI capabilities across Cribl's product suite.

You'll work at the intersection of distributed systems, AI inference infrastructure, model routing, and agentic technologies, building scalable platforms that enable our Product and Research teams to develop, evaluate, and bring cutting edge AI capabilities into production.

Our team is responsible for building and operating critical AI infrastructure, including multi provider model routing, inference services, Model Context Protocol (MCP) integrations, AI observability, and SecIT benchmarking and evaluation systems for IT and security workflows.

You'll have the opportunity to solve challenging engineering problems spanning performance, scalability, reliability, and cost efficiency while shaping the architecture of AI systems used across Cribl's products.

You'll collaborate closely with research, and product teams to turn emerging AI capabilities into production ready systems that deliver real customer value.

On top of it all, you'll have fun. Cribl strives to be a great place to work for everyone.

As An Active Member Of Our Team, You Will...

Work closely with Product and Research engineering teams to prototype, experiment with, evaluate, and productionize new AI capabilities.

Design, build, and operate scalable, highly available backend services and distributed systems supporting AI/ML workloads.

Develop and optimize model routing and inference infrastructure across multiple LLM and model providers, balancing latency, throughput, reliability, and cost.

Build infrastructure supporting agentic AI applications, including MCP integrations, tool connectivity, and secure interactions with enterprise systems.

Develop and maintain AI observability, evaluation, and benchmarking systems to measure model quality, inference performance, and production reliability.

Bridge research and production by building scalable AI/ML infrastructure, evaluation frameworks, and data pipelines.

Productionize, launch, and operate AI-based technologies integrated into Cribl's core products, solving real world customer problems.

Drive technical architecture and design decisions, identifying tradeoffs across scalability, performance, operational complexity, and maintainability.

Evaluate emerging AI models, inference technologies, and frameworks for production readiness and adoption.

Collaborate across engineering teams, contribute to technical design reviews, and help establish best practices for building reliable AI powered systems.

Mentor and coach junior engineers, providing technical guidance and support through design reviews, code reviews, and hands-on collaboration to help them grow their technical skills and engineering judgment.

Participate in oncall rotations and support the operational reliability of production AI services.

If You've Got It – We Want It

Bachelor's degree in Computer Science, Engineering, or equivalent practical experience, with 7+ years of software engineering experience.

Strong experience designing, building, and operating distributed systems, scalable backend services, or cloud-native infrastructure.

Proficiency in one or more programming languages, such as Go, Python, or TypeScript.

Experience building, integrating, or operating production AI/ML systems, including NLP, speech/voice or MLOps based platforms.

Experience with AI/ML infrastructure or LLM application architectures, such as model routing, inference platforms, MCP, RAG, or evaluation frameworks.

Experience with cloud infrastructure, API design and production observability.

Understanding of performance optimization, scalability, reliability, and operational troubleshooting in production environments.

Ability to independently drive technical initiatives, make architectural tradeoffs, and collaborate with Product, Research, and Engineering leadership.

Strong problem solving skills and the ability to navigate ambiguity while maintaining focus on customer and business outcomes.

Nice to Have

Hands on experience with model inference runtimes such as ONNX, NVIDIA Triton, vLLM, or TensorRT-LLM.

Experience deploying and optimizing ML models across cloud infrastructure, GPUs, or resource constrained edge devices.

Familiarity with inference optimization techniques, including quantization, dynamic batching, KV cache and hardware aware scheduling to improve latency, throughput, and cost efficiency.

Experience with model serving, inference benchmarking, performance profiling, and detecting model quality or performance regressions.

Experience building or operating high throughput, low latency inference systems at scale.

Familiarity with fine tuning, model optimization, or deploying custom NLP, speech, or other ML models.

Experience with distributed tracing, AI observability and inference optimizations.

Experience designing secure, multi tenant platforms, workload isolation, and resource management.

Contributions to open source AI infrastructure, inference frameworks, or developer tooling.

#LI-AM1

#LI-Remote

The salary for this role is dependent on geographic location and will be based on the individual candidate's job-related knowledge, skills, and experience.

In addition to base salary, for sales and some sales-adjacent roles, employees are eligible to earn incentive compensation (commission). For all other roles, employees are eligible to participate in the Cribl Corporate Bonus Program.

In addition to a competitive salary, Cribl also offers a generous benefits package which includes health, dental, vision, short-term disability, and life insurance, paid holidays and paid time off, a fertility treatment benefit, 401(k), and equity.

Base Salary Range

$250,000—$300,000 USD

Bring Your Whole Self

Diversity drives innovation, enables better decisions to support our customers, and inspires change for the better. We’re building a culture where differences are valued and welcomed, and we work together to bring out the best in each other. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or any other applicable legally protected characteristics in the location in which the candidate is applying.

Interested in joining the Cribl herd? Learn more about the smartest, funniest, most passionate goats you’ll ever meet at cribl.io/about-us.