Job Description

About Us

Ai Palette, a SaaS company established in 2018 and headquartered in Singapore, serves Consumer Packaged Goods brands worldwide, such as Nestle, Mondelez, and L’Oreal, by facilitating the launch of highly successful products tailored to their target audience.

Traditionally, brands like Nestle would commission surveys to gauge consumer preferences, a process fraught with challenges including time constraints, data accuracy, and high costs. Ai Palette addresses these issues by harnessing the power of artificial intelligence to analyze vast amounts of internet data, identifying emerging trends across various segments. This approach provides brands with invaluable insights into upcoming trends, enabling them to capitalize on significant growth opportunities efficiently.

Moreover, Ai Palette offers a comprehensive platform where brands can not only access trend data but also develop and test product concepts. Currently, the platform tracks an extensive database of 61 billion data points sourced from over 150 outlets across 12 categories, covering approximately 87% of the world's GDP. Its advanced Natural Language Processing algorithm supports 18 languages, including various Asian languages, distinguishing it as a unique and globally accessible solution.

Ai Palette's success and credibility are further underscored by its partnerships with leading food companies, including Fortune 500 giants like Kellogg's, Nestle, and Olam. The company has garnered support from prominent investors, including SGInnovate, AGFunder, Exfinity Ventures, Pi Ventures, Tinmen, and Orzon Ventures, with the most recent funding injection received in March 2024.

About the job

We are looking for a passionate Data Engineer who will be working in the Data Engineering division and core development of our AI Platform that will ideate consumer products of the future. As the Data Engineer, you will be working hand in hand with the Data Science and Full Stack team on the toughest and most challenging problems in Data Engineering and Cloud Computing handling millions of data points that include social media. You will get the opportunity to work and scale a growing Data Platform.

You don’t want a job that just pays the bills: you want a job to get out of bed for. You take more enjoyment from solving problems than your friends think is normal. If you are looking to find your “ikigai”, then your search stops here right with us!

RESPONSIBILITIES

• Build the data collection pipelines that can acquire and handle millions of public data points from various sources using APIs and web extraction techniques at scale
• Build the data cleaning, quality and integrity pipeline in the platform leveraging the Apache Spark Python and AWS services
• Architect and develop distributed systems that can handle large-scale data processing
• Programming Language: Python/Java, Apache Spark, Apache Flink (Good to have)
• NoSQL Database: Elasticsearch, Dynamo DB/Mongo DB
• Work closely with the Data Science Team for the preprocessing steps required for the AI models in Production Environment
• Scale and automate Data Platform Collection layer to handle the consistently incoming data
• Implement data quality checks and validation processes
• Troubleshoot and resolve performance issues
• Collaborate with cross-functional teams to support data-related initiatives
• Document data pipelines, data models, and other technical processes

REQUIREMENTS

• 3-6 years of experience in building data engineering pipeline
• Have hand ons experience on Apache Spark, PySpark, SQL, Python/Java programming
• Worked with NoSQL database like Elasticsearch, Dynamo DB/Mongo DB, Cassandra - Anyone
• Strong experience in AWS Cloud Platform - S3, EC2, Lambda, Elasticsearch etc
• Quick learner, excellent communication and team player