Big Data Engineer

Description:

Big data engineers develop, test, and maintain Big Data solutions for a company. Their job is to gather large amounts of data from multiple sources and ensure that downstream users can access the data quickly and efficiently. Essentially, big data engineers ensure the companys data pipelines are scalable, secure, and able to serve multiple users.

Responsibilities:

Meeting with managers to determine the companys Big Data needs.

Developing Hadoop systems.

Loading disparate data sets and conducting pre-processing services using Hive or Pig.

Finalizing the scope of the system and delivering Big Data solutions.

Managing the communications between the internal system and the survey vendor.

Collaborating with the software research and development teams.

Building cloud platforms for the development of company applications.

Maintaining production systems.

Training staff on data resource management.