Involve Asia is seeking a talented Data Engineer who will play a role in empowering our data-driven ecosystem. In this role, you will collaborate closely with data scientists, analysts, and software engineers, contributing to the creation of robust infrastructure and cutting-edge tools to address diverse business challenges. You'll be immersed in a complex data landscape, managing terabytes of data across multiple products and supporting various data stakeholders.
Your primary focus will be on using Python and/or Java to automate data processes, ensuring the efficient handling of data that continues to grow in both scale and complexity. You'll also be tasked with designing and implementing intricate data pipelines and models tailored for analytical consumption, leveraging tools such as EMR, Kubernetes, Airflow, and more.
Key Responsibilities:
- Maintain our data platform with timely and quality data
- Develop and update documentation for the systems.
- Write code that is both maintainable and performs well.
- Design, implement, operate, and improve the analytics platform
- Collaborate with data scientists, business analysts, product managers, software engineers, and other data engineers to develop, implement, and validate deployed data solutions
- Build and maintain data pipelines from internal databases and SaaS applicaions
- Understand and implement data engineering best practicesImprove, manage, and use standards for code maintainability and performance in code review
- Plan and build application to acquire the data
Requirements:
- Good understanding at writing and optimizing SQL or Data queries
- Experience in Python, Java or similar languages
- Familiarity with data warehousing concepts
- Experience in Airflow or other workflow orchestrators
- Familiarity with basic principles of distributed computing
- Experience with big data technologies like Spark, Hudi and Delta Lake
- Demonstrating ability to innovate and leading delivery of a complex solution
- Excellent verbal and written communication - proven ability to communicate with technical teams and summarize complex analyses in business terms
- Ability to work with shifting deadlines in a fast-paced environment
- Ability to work under pressure and have good problem solving skills
Desirable Qualifications:
- Bachelor Degree in quantitative field such as Computer Science (CS)
- Familiarity with AWS, Google Cloud Platform and Microsoft Azure
- Experience in planning day-to-day tasks, knowing how and what to prioritize, and what to execute
- Past experience as a backend or full-stack software engineer is a plus.
- Experience with data governance and security practices