Mid Level Data Engineer

Looking for A-player consultants in the Data Engineering space who will be working on fixed-scope, fixed-timeline projects across AWS and GCP for large national and international customers, including banks, airlines, medical aid providers and telecommunications companies.

This position offers fully remote, hybrid and office based (Johannesburg and Cape Town) arrangements.

On offer are also opportunities for upskill:

• We pay for you to take your cloud certifications
• We allow for you to study for cloud certifications on company time
• We run weekly internal knowledge sharing sessions

Required Experience

• Hands-on experience in building ETL pipelines on Big Data, using modern cloud
• Data Engineering tools (1+ years):
• AWS Glue (highly preferred)
• GCP DataFlow (preferred)
• Azure Data Factory
• Object Oriented Python programming (2+ years)
• Python package development
• Python class development
• Pythonic standards across the board
• Working knowledge of PySpark (preferred) or Apache Beam
• Efficient use of the PySpark DataFrame API
• Knowledge of big data file formats (Parquet, ORC, AVRO) and partitioning strategies
• Hands-on experience in writing SQL queries (2+ years)
• ANSI-SQL (preferred)
• Postgresql (preferred)
• MS-SQL
• MySQL
• Oracle
• Database experience
• Amazon Redshift
• Google BigQuery Or equivalent

Infrastructure as code:

• Terraform (preferred)
• Cloudformation
• AWS CDK
• Hands on experience in making use of additional cloud services:
• AWS Lambda (highly important)
• AWS IAM (highly important)
• Cross account roles
• IAM policies
• Amazon S3 (highly important)
• AWS Step Functions (important)
• Amazon SQS
• Amazon SNS
• Amazon DynamoDB
• Amazon Kinesis is a plus
• Architectural thinking is a plus!
• Scalability, security, fault-tolerance
• Required Qualifications and Experience
• BSc Computer Science/Engineering (or equivalent experience)
• AWS or GCP Certifications (Associate and above)