BigData/Python ETL Senior Engineer #6414

PURPOSE OF THE JOB

This big data engineer will work with the team to deliver automated consumer behavior prediction platform to companies. Our customer is automating and commoditizing cutting edge AI results directly from a client data lake, and your job will be to help platform handling data at the massive scale needed. The data sizes can reach PetaByte ranges, and it is still expected to process rapidly.

MAIN TASKS AND RESPONSIBILITIES

• Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities.
• Implementing ETL process.
• Part of ETL is analyzing and understanding customer's client data well enough to integrate into the API.
• Propose, design and implement Big Data Architecture including infrastructure.
• Monitoring performance and advising any necessary infrastructure changes.
• Defining data retention policies.

EDUCATION, SKILLS AND EXPERIENCE

• 3 years of experience with Hadoop v2 and MapReduce.
• Proficiency with the management of our Hadoop cluster and accompanying services including Hive, Spark, Kafka, Scoop and Oozie.
• Proficiency with Presto.
• 3 years of experience in NoSql databases. Cassandra preferred.
• Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming.
• Experience with integration of data from multiple data sources.
• Knowledge of various ETL techniques.
• 3 years experience in creating Lambda Architecture, along with knowledge of its advantages and drawbacks.
• Experience with Cloudera/MapR/Hortonworks.
• Strong written English skills are required, any internal or external document uses the SCQSR (Situation-Complication-Question-Solution-Resolution) narrative method and must
• Outstanding academic results. Only the smartest people can deliver the results we expect.

LOHIKA BENEFITS

• Friendly and highly professional teams
• Modern office facilities (kitchens, gym, yoga, playroom, coffee and tea points etc.)
• Regular (twice a year) performance reviews
• Fully paid English classes with certified teachers and native speakers
• Internal and external training
• Paid sick-leaves
• Premium health insurance (medication, massage, and doctor in the office etc.)
• 20 working days of annual paid vacation
• Christmas and other state holidays
• Corporate events (corporate parties, sports competitions, What? Where? When? Team etc.)
• Incentives (marriage, childbirth gifts)