Job Description

• Bangalore, India

Your opportunity

We are currently looking for an adept Manager of Data Ingestion to spearhead our initiatives in constructing a scalable and user-friendly data ingestion system customized for various technical teams within New Relic. Your responsibilities will extend to supervising the development of data ingestion applications on this platform as required.

In this capacity, you will play a pivotal role in shaping the overarching strategy for our data ingestion infrastructure and tools, setting standards for constructing systems and datasets utilized throughout the organization. With your extensive technical acumen, you will bolster our environment to establish a resilient data ingestion foundation. We seek an individual who is passionate about driving significant contributions to New Relic.

What you'll do

Team Management:

• Lead and manage a team of software development engineers specializing in building data collection frameworks and tools.
• Utilize Agile methodologies to facilitate efficient team delivery, sprint planning, and retrospectives.
• Mentor and guide engineers in their professional growth and technical skill development.

Framework Development:

• Oversee the design, development, and maintenance of an open-source ingestion framework for both Batch and Streaming use cases.
• Collaborate with team members to build and enhance data ingestion processes and frameworks to meet evolving business needs.

Batch and Streaming Ingestion:

• Manage the implementation of Batch and Streaming data ingestion requests, ensuring adherence to best practices and standards.
• Maintain and administer ingestion jobs for both Batch and Streaming data sources, monitoring performance and resolving issues as needed.

Data Quality Assurance:

• Implement and oversee basic data quality tests such as freshness and completeness checks to maintain the integrity of ingested data.
• Documentation:
• Ensure comprehensive documentation of raw data sources in the Data Lake, including metadata, schema definitions, and ingestion processes.

API Development:

• Lead the design and development of RESTful APIs for data collection, enabling seamless integration with internal and external systems.

Emitter and Consumer Development:

• Drive the development of emitters, Kafka consumers, and integrations with NRDB (New Relic Database) to support data collection and ingestion.

This role requires

• 10+ years of experience
• Minimum of 3 years managing and leading teams
• 3 plus years of professional experience in Python and/or Java development
• 3 plus years of using Javascript 
• Proven experience managing software development teams, with a focus on delivering high-quality solutions within Agile frameworks.
• Strong leadership and mentorship skills, with the ability to inspire and motivate team members to achieve their full potential.
• Proficiency in software development with Python and Java, including experience with data serialization formats such as JSON, Avro, or Protobuf.
• Deep understanding of data collection and ingestion processes, with hands-on experience working with large datasets, Kafka, and other distributed systems.
• Demonstrated ability to design and develop RESTful APIs, ensuring scalability, reliability, and performance.
• Excellent communication skills, with the ability to effectively collaborate with cross-functional teams and stakeholders.
• Strong problem-solving and analytical abilities, with a keen attention to detail and a focus on continuous improvement.
• Prior experience in managing duties such as resource allocation, budget management, and performance evaluations is highly desirable.

Bonus points if you have

If you are passionate about leading data engineering initiatives and driving impactful outcomes in a dynamic environment.