Data Engineer, Online Marketplace; Sign-on Bonus: $1000

We offer a $1000 sign-on bonus to the specialist that will accept our offer and join DataArt in this position. $1000 will be included in the first salary.

Our client is a EU-based marketplace. They developed a website and app that allows users to buy and sell wardrobe items or gadgets.

For the first stage the project is aimed at re-thinking and redesigning the data store. The client needs to migrate a large image database, apply a new technology for the data governance system, and to perform a clean-up of useless data. During the next stage it is expected to conduct some R&D and migrate to another data warehouse. We expect long collaboration with the client as the project is at an early stage of development and new challenges and tasks might appear.

The data engineer will work on the migration of all images to another storage from the current S3 bucket, which takes up around 400 TB of data as well as applying governance over the data, switching from the Amazon internal messaging system to Kafka. The task of restructuring the client’s database also provides an opportunity to learn Akamai in practice as it needs to be reconfigured for serving data from the new bucket.

Tech stack and infrastructure: no hard requirements about language used but Scala is preferable as most of engineers work with that language. AWS technologies - data stored in a S3 bucket, Akamai serves data from a bucket.

Responsibilities

• R&D and solution design phase
• Design and implementation of a data migration pipeline
• Integration with the existing data platform

Required Skills and Experience

• Hands-on experience in data engineering (e.g., working with large volumes of data, cloud environments, parallelization)
• Knowledge of AWS technologies (e.g., S3, Aurora)
• Akamai configuration would be a huge plus
• Spoken English, good communication and coordination skills

DataArt offers:

• Professional Development:
  - Experienced colleagues who are ready to share knowledge;
  - The ability to switch projects, technology stacks, try yourself in different roles;
  - Study and practice of English: courses and communication with colleagues from different countries;
  - Support of speakers who make presentations at conferences and meetings of technology communities.
• The ability to focus on your work: a lack of bureaucracy and micromanagement, and convenient corporate services;
• Friendly atmosphere, concern for the comfort of specialists;
• Flexible schedule.