250 Spark jobs in Kuala Lumpur

Overview

We are growing! We are currently looking to hire a Data Processing Specialist to work with us!

Founded in 2006, our story started with two entrepreneurs. Today, we’re proud to be a global business. From Shanghai to Paris, we have 12 offices and operate across four continents in 70 countries. We are home to over 250 professionals from around the world, working together to serve more than 230 luxury clients. At CXG, we love to evolve, elevate, and transform experiences while bringing brand promises to life. We offer strategic solutions that impact performance and elevate the customer experience of some of the world’s most iconic premium and luxury brands.

• Support data analysts in the China region to prepare and process primary data
• Generate key slides for customer experience reports
• Collaborate with data analysts on the creation of analysis
• Manage project timelines and delivery for assigned missions
• Proofreading and data checking research reports to ensure deliverables are error-free
• You will query directly the data from Snowflake with your knowledge in SQL to provide analysis to analysts

• 1 to 2 years’ experience working with data
• Passionate towards the Luxury and Fashion industry
• Detail-orientated, fast learner, good time management and able to work in a fast-paced work environment with a can-do attitude
• Able to interpret large/small amounts of data and multi-task
• Strong communication skills
• Strong analytical mind and critical thinking skills
• Excellent knowledge in MS Office (Excel, Word, PowerPoint)
• Skills in SQL or Python
• Skills in Power BI or automation tool is a plus
• Exceptional written, oral communication abilities in English and Chinese
• Preferably from Statistics, Mathematics or computing background

Overview

We are growing! We are currently looking to hire a Data Processing Specialist to work with us!

Founded in 2006, our story started with two entrepreneurs. Today, we’re proud to be a global business. From Shanghai to Paris, we have 12 offices and operate across four continents in 70 countries. We are home to over 250 professionals from around the world, working together to serve more than 230 luxury clients. At CXG, we love to evolve, elevate, and transform experiences while bringing brand promises to life. We offer strategic solutions that impact performance and elevate the customer experience of some of the world’s most iconic premium and luxury brands.

• Support data analysts in the China region to prepare and process primary data
• Generate key slides for customer experience reports
• Collaborate with data analysts on the creation of analysis
• Manage project timelines and delivery for assigned missions
• Proofreading and data checking research reports to ensure deliverables are error-free
• You will query directly the data from Snowflake with your knowledge in SQL to provide analysis to analysts

• 1 to 2 years’ experience working with data
• Passionate towards the Luxury and Fashion industry
• Detail-orientated, fast learner, good time management and able to work in a fast-paced work environment with a can-do attitude
• Able to interpret large/small amounts of data and multi-task
• Strong communication skills
• Strong analytical mind and critical thinking skills
• Excellent knowledge in MS Office (Excel, Word, PowerPoint)
• Skills in SQL or Python
• Skills in Power BI or automation tool is a plus
• Exceptional written, oral communication abilities in English and Chinese
• Preferably from Statistics, Mathematics or computing background

Company : Web3 & Blockchain-focused Company

Job Advantages

• Cutting-edge technology (Web3 and blockchain)
• Globalized company

Job Responsibilities

As a Senior Java Engineer , you will play a pivotal role in the development and optimization of the company’s data infrastructure, ensuring it supports the evolving needs of our blockchain-focused business. Your primary responsibilities will include:

• Architectural Design & Development : Lead the design and development of the company’s data center, ensuring robust performance and scalability.
• Data Platform Optimization : Provide ongoing solutions for platform architecture and performance optimization to support the company’s rapid growth and business needs.
• Business Enablement : Drive data-driven solutions that accelerate business development and continuously enhance the company's core competitiveness.
• Big Data Iteration : Lead the rapid iteration of big data platforms, ensuring efficiency, cost-effectiveness, and high-quality output.

Job Requirements

We are seeking candidates who are passionate about blockchain technology and possess a strong technical foundation. Ideal candidates will meet the following criteria:

• Education : Bachelor’s degree or above in Computer Science or related majors.
• Experience : More than 3 years of experience in Java development, with a deep understanding of JVM principles and Java development best practices.
• Frameworks : Proficient in frameworks such as Spring, Spring MVC, gRPC, Mybatis , and an ability to understand their underlying principles and mechanisms.
• Core Computer Fundamentals : Strong knowledge of computer operating systems, network architecture, and proficiency in commonly used algorithms, data structures, and design patterns.
• Distributed Systems : Familiarity with distributed systems, caching mechanisms (Redis), messaging systems (Kafka), and big data processing frameworks like Spark, Flink , and Zookeeper . Experience with TiDB is a plus.
• Experience in Microservices : Experience in the design and development of data centers or microservices, with an emphasis on high availability and scalability.
• Tagging Systems/Recommendation Systems : Experience with tagging systems or algorithmic recommendation systems is highly desirable.
• Passion for Blockchain : A strong enthusiasm for the blockchain industry and a commitment to contributing to its growth and development.

Why Join Us?

This role offers the opportunity to work at the forefront of blockchain and Web3 technologies within a global company. You will have the chance to develop and optimize critical infrastructure that powers innovative and scalable solutions in the blockchain space. If you’re ready to work in a fast-paced and cutting-edge environment, this role could be the perfect fit for you.

Apply now to be part of an exciting journey in revolutionizing the Web3 ecosystem!

Job Summary:
We are looking for a Big Data Hadoop Developer to design, develop, and maintain large-scale data processing solutions. The ideal candidate should have strong hands-on experience with the Hadoop ecosystem and integration with relational databases such as MariaDB or Oracle DB for analytics and reporting.

Key Responsibilities:

• Design, develop, and optimize Hadoop-based big data solutions for batch and real-time data processing.
• Work with data ingestion frameworks to integrate data from MariaDB/Oracle DB into Hadoop (Sqoop, Apache Nifi, Kafka).
• Implement Hive, Spark, and MapReduce jobs for data transformation and analytics.
• Optimize Hive queries, Spark jobs, and HDFS usage for performance and cost efficiency.
• Create and maintain ETL pipelines for structured and unstructured data.
• Troubleshoot and resolve issues in Hadoop jobs and database connectivity.
• Collaborate with BI, analytics, and data science teams for data provisioning.
• Ensure data security, governance, and compliance in all solutions.

Technical Skills:

• Big Data Ecosystem: Hadoop (HDFS, YARN), Hive, Spark, Sqoop, MapReduce, Oozie, Flume.
• Databases: MariaDB and/or Oracle DB (SQL, PL/SQL).
• Programming: Java, Scala, or Python for Spark/MapReduce development.
• Data Ingestion: Sqoop, Kafka, Nifi (for integrating RDBMS with Hadoop).
• Query Optimization: Hive tuning, partitioning, bucketing, indexing.
• Tools: Ambari, Cloudera Manager, Git, Jenkins.
• OS & Scripting: Linux/Unix shell scripting.

Soft Skills:

• Strong analytical skills and problem-solving abilities.
• Good communication skills for working with cross-functional teams.
• Ability to manage priorities in a fast-paced environment.

Nice to Have:

• Experience with cloud-based big data platforms (AWS EMR, Azure HDInsight, GCP Dataproc).
• Knowledge of NoSQL databases (HBase, Cassandra).
• Exposure to machine learning integration with Hadoop/Spark.

Senior Big Data Engineer at RAPSYS TECHNOLOGIES PTE LTD. We are seeking an experienced Senior Big Data Engineer to design, develop, and maintain large-scale data processing systems. The ideal candidate will have expertise in big data technologies, data architecture, and analytics to drive data-driven insights and support business objectives.

Kuala Lumpur, Malaysia

Work From Office

Senior Big Data Engineer

• Design and evolve the overall data architecture, ensuring scalability, flexibility, and compliance with enterprise standards.
• Build efficient, secure, and reliable data pipelines using the Bronze-Silver-Gold architecture within EDL.
• Develop and orchestrate scheduled jobs in the EDL environment to support continuous ingestion and transformation.
• Implement Apache Iceberg for data versioning, governance, and optimization.
• Leverage the Medallion framework to standardize data product maturity and delivery.
• Govern metadata, data lineage, and business glossary using tools like Apache Atlas.
• Ensure data security, privacy, and regulatory compliance across all data processes.
• Support Data Mesh principles by collaborating with domain teams to design and implement reusable Data Products.
• Integrate data across structured, semi-structured, and unstructured sources from enterprise systems such as ODS and CRM systems.
• Drive adoption of DataOps/MLOps best practices and mentor peers across units.
• Generate and manage large-scale batch files using Spark and Hive for high-volume data processing.
• Design and implement document-based data models and transform relational models into NoSQL document-oriented structures.

• Bachelor’s, Master’s, or PhD in Computer Science, Data Engineering, or a related discipline.
• 5–7 years of experience in data engineering and distributed data systems.
• Strong hands-on experience with Apache Hive, HBase, Kafka, Solr, Elasticsearch.
• Proficient in data architecture, data modelling, and pipeline scheduling/orchestration.
• Operational experience with Data Mesh, Data Product development, and hybrid cloud data platforms.
• Familiarity with CRM systems, including CRM system, and data sourcing/mapping strategies.
• Proficient in managing metadata, glossary, and lineage tools like Apache Atlas.
• Proven experience in generating large-scale batch files using Spark and Hive.
• Strong understanding of document-based data models and the transformation of relational schemas into document-oriented structures.

• Expertise in data administration, modelling, mapping, collection, and distribution.
• Strong understanding of business workflows to support metadata governance.
• Hands-on experience with analytics and DWH tools (e.g., SAS, Oracle, MS SQL, Python, R Programming).
• Familiarity with data modelling tools (e.g., ERWIN), and enterprise databases (Oracle, IBM DB2, MS SQL, Hadoop, Object Store).
• Experience working across hybrid cloud environments (e.g., AWS, Azure Data Factory).
• In-depth knowledge of ETL/ELT processes and automation frameworks.
• Analytical thinker with strong problem-solving and communication skills.
• Able to collaborate effectively across technical and business teams.

Overview

EPAM Systems Kuala Lumpur, Federal Territory of Kuala Lumpur, Malaysia

Software Engineers at EPAM are the driving force behind strategic initiatives for our clients. As a Senior Software Engineer at EPAM Malaysia, you will use your expertise in Big Data to collaborate with product and engineering teams and combine the functional and technical aspects of Software Development with Big Data technology in the project space of cloud services.

• Design and implement innovative analytical solutions using Hadoop, NoSQL and other Big Data related technologies, evaluating new features and architecture in cloud / on-premise / hybrid solutions
• Build collaborative partnerships with architects, technical leads and key individuals within other functional groups
• Perform detailed analysis of business problems and technical environments to design quality technical solutions
• Participate in code review and test solutions

• At least 6 years of working experience with Big Data technologies and Enterprise Software Development
• Solid skills in enterprise software development, practical experience in infrastructure & bug troubleshooting
• Proven expertise in overseeing and evaluating deliverables produced by team members
• Strong skills in enterprise software development, including infrastructure troubleshooting, incident investigation, performance tuning and root cause analysis
• Hands-on experience in Spark / Pandas, Airflow and Python
• Hands-on experience in cloud experience, preferably AWS, and NoSQL
• Experience with component / integration testing, unit testing, and hands-on experience with GitHub, Kubernetes and Docker

• By choosing EPAM, you're getting a job at one of the most loved workplaces according to Newsweek 2021 & 2022&2023.
• Employee ideas are the main driver of our business. We have a very supportive environment where your voice matters
• You will be challenged while working side-by-side with the best talent globally. We work with top-notch technologies, constantly seeking new industry trends and best practices
• We offer a transparent career path and an individual roadmap to engineer your future & accelerate your journey
• At EPAM, you can find vast opportunities for self-development: online courses and libraries, mentoring programs, partial grants of certification, and experience exchange with colleagues around the world. You will learn, contribute, and grow with us

• EPAM is a leader in the fastest-growing segment (product development/digital platform engineering) of the IT industry. We acquired Just-BI in 2021 to reinforce our leading position as a global Business Intelligence services provider and have been growing rapidly. With a talented multinational team, we provide data and analytics expertise
• We are currently involved in end-to-end BI design and implementation projects in major national and international companies. We are proud of our entrepreneurial start-up culture and are focused on investing in people by creating continuous learning and development opportunities for our employees who deliver engineering excellence for our clients