Sr Lead Software Engineer - AI Agents

We are building a portfolio of agentic operations — production-grade AI agents that plan, act, and continuously improve the everyday experiences of our employees. The mission is to eliminate friction from the moments that matter employees spend more time on work that matters and less time on tickets, forms, and queues.

As Senior Lead Software Engineer, AI Engineering, you will be a hands-on senior individual contributor on this team, reporting to the Executive Director who owns the agentic platform and flagship agents. You will be the technical anchor of one or more agent squads: designing the agent, writing the code, running the evaluation harness, integrating with enterprise tools (ServiceNow, Databricks, identity, endpoint), and taking the pager when it ships.

You will pair with engineers, review PRs, drive design docs, mentor the squad, and be personally accountable for the correctness, safety, cost, and business impact of the agents you build.

Job responsibilities

Design and build agents

• Own the end-to-end design of production agents: planner, tools, memory, retrieval (RAG / GraphRAG), guardrails, human-in-the-loop, evaluators.

• Implement using the team's Model Context Protocol (MCP) stack — write MCP servers/clients, define tool schemas, negotiate capabilities, and integrate securely with enterprise systems.

• Choose the right orchestration pattern (single-agent, planner-executor, supervisor-worker, debate/critique) for each use case; reject unnecessary complexity.

Evaluation, safety, and quality

• Build and maintain the evaluation harness for your agents: gold sets, offline benchmarks, LLM-as-judge with human calibration, online A/B, regression gates in CI/CD.

• Instrument every agent with business KPIs (deflection rate, time-to-resolve, CSAT, cost per interaction) and quality KPIs (task success, groundedness, refusal correctness, safety incidents).

• Partner with cyber, privacy, and model risk to make guardrails concrete: PII handling, prompt-injection & jailbreak defense, tool-use hygiene, secret handling, prompt/model version pinning.

Production reliability (SRE for agents)

• Own SLOs and error budgets for your agents; write runbooks; take part in the on-call rotation.

• Build canaries, circuit breakers, deterministic replay, and rollback paths for prompts, models, and tools.

• Diagnose and fix production issues end-to-end — from token spend anomalies to tool call failures to hallucination regressions.

Platform contribution

• Contribute to the shared agentic platform (orchestration, model gateway, vector/knowledge stores, telemetry, eval harness) — not just consume it. Push improvements upstream so other squads benefit.

• Write design docs, patterns, and internal blog posts. Represent the team in architecture reviews.

Technical leadership without being a manager

• Set the technical bar in your squad through code reviews, pairing, and design reviews.

• Mentor mid-level and junior engineers; grow the next generation of AI engineers.

• Interview candidates and help sustain a strong hiring bar.

Required qualifications, capabilities and skills

• 8+ years of software engineering experience, with 2+ years shipping production LLM- or agent-based systems to real users (not just prototypes or notebooks).

• Deep hands-on expertise in Python (primary) and comfort with at least one other modern language (TypeScript / Java / Go).

• Practical, current experience with the agentic stack: LLM orchestration (e.g. LangGraph, LlamaIndex, Semantic Kernel, custom), tool/function calling, RAG, evaluation harnesses, guardrails, prompt & model versioning, model gateways.

• Hands-on with Model Context Protocol (MCP) architectures — designing servers/clients, tool schemas, capability negotiation, and secure enterprise deployment patterns.

• Track record of building and running production-grade backend / distributed systems (APIs, queues, workers, observability, on-call).

• Strong grounding in data platforms (Databricks / Snowflake / lakehouse patterns) and modern MLOps/LLMOps (CI/CD, feature/prompt/model registry, telemetry).

• Practical experience with responsible AI, security, and controls for enterprise deployments: PII handling, red-teaming, prompt-injection defense, secret & tool-use hygiene.

• Clear technical communication: you can write a crisp design doc, review a PR with substance, and explain a trade-off to a non-technical stakeholder.

• Bachelor's in Computer Science or related discipline, or equivalent industry experience.

Preferred qualifications, capabilities and skills

• Experience building employee-facing / workforce productivity products at scale (IT service desk, HR tech, knowledge management, workplace assistants).

• Experience integrating with ServiceNow, identity providers, endpoint management, and enterprise search.

• Experience with multi-agent patterns (planner-executor, supervisor-worker, debate/critique) and long-horizon workflows.

• Contributions to open source (especially in the MCP / agent / eval ecosystem), published research, patents, or public talks in AI / agents / SRE.

• Prior experience in a highly regulated environment (financial services, healthcare, public sector) with model risk management (SR 11-7 or equivalent).

• Master's or PhD in a quantitative discipline.