Lead Software Engineer - Data Engg. - Databricks / Snowflake

We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible.

As a Lead Software Engineer at JPMorgan Chase within the (IAM) Identity and Access Management Data team, you will play a crucial role in designing, developing, and maintaining scalable data processing solutions using Databricks, Python, and AWS. You will collaborate with cross-functional teams to deliver high-quality data solutions that support our business objectives.

Job responsibilities

• Execute creative, data-driven software solutions end-to-end (design, development, troubleshooting), thinking beyond routine approaches to solve complex technical problems.

• Design and build a control plane for enterprise data pipelines, standardizing pipeline definition, scheduling, deployment, governance, and run-time management (Databricks today; extensible for future engines).

• Develop self-service APIs/SDKs, templates, and configuration-driven onboarding with consistent guardrails (standards, validation, environment promotion, approvals) and centralized pipeline metadata (ownership, SLAs/SLOs, dependencies, schema/parameter/version tracking).

• Design, develop, and maintain scalable data pipelines and processing workflows using Python, PySpark, SQL, Databricks on AWS; develop fact/dimension models for analytics and reporting.

• Ensure data quality, security, lineage, and operational transparency via standardized observability (logs/metrics/traces), dashboards, alerting, runbooks, and automated remediation patterns (retries/backfills, common-failure automation).

• Lead and participate in the full SDLC (requirements, design, build, test, deploy, maintain), acting as SRE/production support for pipeline and platform services to improve stability and reliability.

• Collaborate with stakeholders to shape data management strategy and translate requirements into scalable, compliant solutions; document data flows, logic, and transformation rules for knowledge sharing.

• Mentor engineers and lead communities of practice to drive adoption of modern engineering practices and tools, fostering an inclusive, high-performing culture; utilize firm-approved AI-assisted development tools to accelerate delivery and testing

• Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.

• Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.

Required qualifications, capabilities, and skills

• Formal training or certification on software engineering concepts and 5+ years applied experience

• Proven experience in data management and ETL/ELT for large-scale processing, including strong SQL, Python, and PySpark with performance tuning and query optimization.

• Hands-on experience with Databricks/Spark and cloud data lake patterns, integrating compute/workflows with AWS services (e.g., S3, ECS, SNS/SQS, Lambda).

• Proven experience building platform services/control planes (or similar orchestration/automation platforms), including API/service design, configuration-driven systems, and versioning/backward compatibility.

• Strong understanding of data quality, security-by-design, and lineage/auditability, including IAM/least privilege and secrets management principles.

• Strong production engineering mindset: observability (logs/metrics/traces), monitoring/alerting, incident response, and operational excellence for always-on services.

• Proficiency in CI/CD and release engineering (quality gates, automated testing, safe deployments/rollbacks) using firm-standard tooling (e.g., Jenkins/Jules, Spinnaker, Sonar).

• Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security.

• Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices

Preferred qualifications, capabilities, and skills

• Experience with orchestration/execution frameworks (Databricks Workflows/Jobs, Airflow, Step Functions) and operational patterns such as dependency graphs (DAGs), replays, and backfills.

• Experience with data governance integrations (e.g., Unity Catalog concepts such as cataloging, permissions, and lineage hooks), where applicable.

• Infrastructure-as-Code experience (Terraform/CloudFormation) and developer-platform “golden path” enablement (internal CLIs, templates, paved roads, onboarding automation).

• Experience with FinOps/cost controls for Spark/Databricks workloads (telemetry, quotas, chargeback/showback) and data formats (Parquet, JSON, CSV, Avro, Delta Lake),

• Knowledge of regulatory reporting and financial data aggregation techniques; Databricks and/or AWS certifications.