Job Description

• Hyderabad, Telangana, India

Experience: 5+ years

Must have Skills:

ADF, SQL Server, Databricks Notebooks, Data Pipelines, SPROCs (Stored Procedures), DataWarehouse, Databricks DataLake

Job Description:

The Data Engineer is responsible for designing, developing, and maintaining data pipelines, data warehouses, and data lakes for an organization. They work closely with data scientists, analysts, and other stakeholders to ensure that data is efficiently stored, processed, and accessible for analysis. The Data Engineer must have a strong understanding of various data technologies, including ADF (Azure Data Factory), SQL Server, Databricks Notebooks, and SPROCs (Stored Procedures).

Key Responsibilities:

1. Design and develop data pipelines to extract, transform, and load (ETL) data from various sources into data warehouses or data lakes.

2. Create and maintain data models, schemas, and data structures in SQL Server and Databricks Data Lake.

3. Develop and optimize SPROCs for efficient data retrieval and manipulation.

4. Collaborate with data scientists, analysts, and other team members to understand data requirements and ensure data quality.

5. Monitor and troubleshoot data pipeline issues, and implement improvements to enhance performance and reliability.

6. Develop and maintain data documentation, including data dictionaries, pipeline diagrams, and data quality reports.

7. Stay current with new technologies, tools, and best practices in the data engineering field.

Requirements:

1. Bachelor's degree in Computer Science, Data Science, Engineering, or a related field.

2. Strong programming skills in languages such as Python, R, or Scala.

3. Experience with data engineering tools and platforms, including ADF, SQL Server, Databricks Notebooks, and data pipelines.

4. Familiarity with data warehousing concepts and technologies.

5. Strong understanding of data modeling, ETL processes, and data quality principles.

6. Excellent problem-solving, analytical, and communication skills.

7. Ability to work independently and collaboratively in a fast-paced environment.