Data Engineer

Job description

We are looking for a Data Engineer who will work on the collecting, storing, processing, and analyzing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. The candidate will also be responsible for integrating them with the architecture used across the company. The Data Engineer is tasked with designing and developing scalable ETL packages from the business source systems and the development of ETL routines in order to populate databases from sources and also to create aggregates. It is also the role of the Data Engineer to oversee large-scale data Hadoop platforms and to support the fast-growing data within the business.

Responsibilities & Skills

· Implementing ETL process

· Monitoring performance and advising any necessary infrastructure changes

· Proficient understanding of distributed computing principles

· Proficient in datawarehouse concepts

· Experience with integration of data from multiple data sources

· Proficient in managing RDBMS like MS-SQL, DB2 etc

· Management of Data Lake, Hadoop cluster, with all included services

· Proficiency with Hadoop, MapReduce, HDFS

· Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala

· Experience with Spark

· Experience with NoSQL databases, such as HBase, Cassandra, MongoDB

· Knowledge of various ETL techniques and frameworks, such as Flume

· Experience with Big Data ML toolkits, such as Mahout, SparkML, or H2O

· Good understanding of Lambda Architecture, along with its advantages and drawbacks

 

Qualifications

• The Data Engineer must have a bachelor’s degree in Computer Science, Applied Mathematics, Engineering, or any other technology related field

• A candidate for the position will have as at least 3 years of working experience as a database engineering support personnel or a database engineering administrator within a fast-paced a complex business setting

• Great communication skills

• Data-oriented personality

• Good scripting and programming skills

• 3-5 years of relevant experience

LinkedIN easy apply

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s