Job Description:

Responsibilities:

• Develop & Optimize Data Pipelines

  • Build, test, and maintain ETL/ELT data pipelines using Azure Databricks & Apache Spark (PySpark).
  • Optimize performance and cost-efficiency of Spark jobs.
  • Ensure data quality through validation, monitoring, and alerting mechanisms.
  • Understand cluster types, configuration, and use-case for serverless

• Implement Unity Catalog for Data Governance

  • Design and enforce access control policies using Unity Catalog.
  • Manage data lineage, auditing, and metadata governance.
  • Enable secure data sharing across teams and external stakeholders.

• Integrate with Cloud Data Platforms

  • Work with Azure Data Lake Storage / Azure Blob Storage/ Azure Event Hub to integrate Databricks with cloud-based data lakes, data warehouses, and event streams.
  • Implement Delta Lake for scalable, ACID-compliant storage.

• Automate & Orchestrate Workflows

  • Develop CI/CD pipelines for data workflows using Azure Databricks Workflows or Azure Data Factory.
  • Monitor and troubleshoot failures in job execution and cluster performance.

• Collaborate with Stakeholders

  • Work with Data Analysts, Scientists, and Business Teams to understand requirements.
  • Translate business needs into scalable data engineering solutions.

• API expertise

  • Ability to pull data from a wide variety of APIs using different strategies and methods

Required Skills & Experience:

• Azure Databricks & Apache Spark (PySpark) – Strong experience in building distributed data pipelines.

• Python – Proficiency in writing optimized and maintainable Python code for data engineering.

• Unity Catalog – Hands-on experience implementing data governance, access controls, and lineage tracking.

• SQL – Strong knowledge of SQL for data transformations and optimizations.

• Delta Lake – Understanding of time travel, schema evolution, and performance tuning.

• Workflow Orchestration – Experience with Azure Databricks Jobs or Azure Data Factory.

• CI/CD & Infrastructure as Code (IaC) – Familiarity with Databricks CLI, Databricks DABs, and DevOps principles.

• Security & Compliance – Knowledge of IAM, role-based access control (RBAC), and encryption.

Preferred Qualifications:

• Experience with MLflow for model tracking & deployment in Databricks.

• Familiarity with streaming technologies (Kafka, Delta Live Tables, Azure Event Hub, Azure Event Grid).

• Hands-on experience with dbt (Data Build Tool) for modular ETL development.

• Certification in Databricks, Azure is a plus.

• Experience with Azure Databricks Lakehouse connectors for SalesForce and SQL Server

• Experience with Azure Synapse Link for Dynamics, dataverse

• Familiarity with other data pipeline strategies, like Azure Functions, Fabric, ADF, etc

Soft Skills:

• Strong problem-solving and debugging skills.

• Ability to work independently and in teams.

• Excellent communication and documentation skills.