Test Automation Lead – AI Testing

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.

Are you passionate about leveraging technology to automate and transform business processes?

Do you have a deep understanding of AI and test automation and a desire to drive innovation in enterprise systems?

If so, the Tech Enablement and Automation team at Tech@Lilly is looking for a Lead Consultant in Test Automation to help scale our automation capabilities and quality practices across the organization.

As a Lead consultant in Test Automation, you will be responsible for driving the quality and reliability of automation solutions across the enterprise. In this role, you will work closely with Engineers and cross-functional teams throughout the full application lifecycle—ensuring that systems are thoroughly tested, resilient, and aligned with business needs from development through to production.

Are you innovative and curious about new technology that can make business processes easy and more efficient? Come join our team!

What You’ll Be Doing:

As a Lead Consultant you will play a pivotal role in shaping and advancing the company’s enterprise-wide quality assurance and engineering strategy for AI powered systems and solutions. You will leverage both your technical expertise and leadership capabilities to oversee the delivery of scalable, high-quality AI and automation solutions. Your responsibilities will include driving resolution of complex testing challenges, leading the implementation of critical enhancements, and spearheading modernization initiatives to ensure our AI and automation frameworks and practices are future-ready and aligned with business objectives.

What You Will Bring:

• Deep expertise in test automation frameworks and strategies, with the ability to extend them to validate AI and ML models, data pipelines, and intelligent applications.

• Hands-on expertise with AI evaluation methodologies, model benchmarking, and quality metrics

• A strong understanding of AI testing methodologies such as model validation, bias detection, explainability testing, and performance benchmarking.

• Ability to design end-to-end automated testing solutions for AI-enabled systems involving APIs, UI, data, and model outputs.

• Proven experience collaborating with data scientists, ML engineers, and software developers to ensure test coverage and model reliability.

How You Will Succeed:

• Lead with influence by promoting a culture of engineering excellence**, responsible AI**, and continuous improvement.

• Develop and maintain automation frameworks tailored for testing AI/ML workflows, including model training, inference, and integration.

• Create and execute automated regression, performance, and data validation tests to ensure the stability and quality of AI-driven systems.

• Implement AI-specific test approaches such as synthetic data generation, result explainability validation, and continuous model monitoring.

• Drive innovation by identifying opportunities to embed automation and intelligence into the testing lifecycle using GenAI tools and frameworks.

• Partner cross-functionally to define and track AI quality metrics (accuracy, precision, recall, fairness) and integrate them into CI/CD pipelines.

What You Should Bring:

• Deep understanding of LLMs, Generative AI, RAG, and Agentic AI architectures

• Hands-on experience in AI evaluation frameworks (RAGAS, DeepEval, LLM-as-a-Judge, LangFuse)

• Expertise in Prompt Validation, Prompt Injection Testing and Adversarial Testing (Red Teaming)

• Knowledge of Data Validation, Data Quality, Data Drift, and Training/Inference Pipeline Testing

• Deep hands-on experience with a broad range of test automation tools and technologies,

  • UI and functional testing: Playwright, Cypress
  • API testing: Postman
  • Performance testing: k6, JMeter

• Strong understanding of supervised/unsupervised learning.

• Experience with ML platforms and tools and cloud AI services.

• Knowledge of DevOps practices and CI/CD tools like Jenkins, GitHub Actions, or Azure DevOps.

• Excellent analytical, debugging, and communication skills with a passion for quality and innovation.

• Define, implement, and own test strategies across enterprise-level projects and programs.

• Lead and mentor global and cross-functional QA teams, promoting best practices and continuous improvement.

• Collaborate with senior leadership, product owners, architects, and development teams to ensure quality is built into all stages of the development lifecycle.

• Drive the implementation of test automation strategies and continuous testing practices.

• Identify test process improvement opportunities and lead initiatives to enhance QA&E maturity and efficiency

• Oversee risk identification, mitigation planning, and resolution related to QA deliverables.

• Excellent communication, and stakeholder management skills.

• Exposure to Power BI, Splunk, or other monitoring/reporting tools for real-time visibility into test execution and system behaviour.

• Champion an “automation-first” mindset in test execution and continuous delivery strategies.

• Provide expert-level consultation on AI and automation tools, frameworks, and best-fit solutions tailored to business and technology needs.

• Align people, tools, and processes to support rapid, reusable, and scalable automation solutions across the enterprise.

• Continuously evaluate emerging tools and technologies to ensure Lilly remains at the forefront of quality engineering innovation.

• Strong understanding of Agile methodologies such as Scrum, Kanban, and SAFe.

Basic Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Data Science, Information Technology, or a related field.

• 7+ years of test automation experience, including at least 2 years focused on AI/ML testing.

• Proven track record in designing and executing automated test suites for complex, distributed, or AI-based systems.

• Strong programming and problem-solving skills with experience in Python, JS/TS, or similar languages.

Preferred Skills:

• Experience with AI quality tools such as DeepChecks, Great Expectations, DataFold.

• Understanding of MLOps and automated model lifecycle management.

• Certification in AI/ML, Test Automation, or Cloud Technologies (e.g., AWS Certified Machine Learning, ISTQB Advanced Test Automation Engineer).

Will this position support sponsorship:

No

Travel percentage for this position:

0-10%

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly