Global leader in workforce solutions for machine learning and business process optimization | Creating meaningful work for talented people in developing nations
Data Engineer – CloudFactory – Roles and Responsibilities
As a member of CloudFactory’s engineering team, the Data Engineer – CloudFactory will be tasked with building and growing a world-class distributed workforce management platform that will connect one million people in developing countries to basic computer work.
- Help to design and develop a data estate that is performant, accessible, secure, scalable, maintainable, and extensible.
- Help to implement true CI/CD using Github Actions, etc.
- Design and develop EDW using Snowflake, DBT.
- Design and develop AWS Data Lake using S3, Athena, Snowflake
- Design and develop data ingestion pipelines using SnowPipe, FiveTran, etc.
- Model EDW entities and ensure all data is complete, accurate, timely, and documented within Data Dictionary and Ubiquitous Terms documents.
- Work towards the implementation of a true Self-Service BI platform.
- Implement good practices to ensure that Git is used in all circumstances, This includes reviewing others’ work and having your work reviewed and approved by colleagues.
Governance & Data Protection
- Ensure that all work follows best security practices and fully adheres to GDPR, PECR, and other data regulations.
- Ensure that all work follows the correct approval and sign-off process before it is pushed into Production.
- Ensure that all work is documented and if needed – has a runbook in order to allow the business to continually support it.
- Where required, ensure that a PIA (Privacy Impact Assessment) is completed.
- Work with others in the team to keep the Data Dictionary and Ubiquitous Language complete and up to date.
- Follow existing processes and work to improve/identify gaps in these processes.
- Ensure the correct SDLC promotion processes are followed.
- Follow the correct sign-off processes to ensure that only approved releases are deployed into Production.
- Ensure that all AWS development follows CI/CD processes and is repeatable.
- Ensure that the AWS “Well-Architected Framework” is adhered to. See https://aws.amazon.com/architecture/well-architected/?wa-lens-whitepapers.sort-by=item.additionalFields.sortDate&wa-lens-whitepapers.sort-order=desc
- Evangelise about the Data Team and estate across the business.
- Build relationships with members of the Data Team and the wider Engineering Team.
- Work closely and collaboratively with all members of the Data Team and wider Engineering Team.
- Work closely with and learn from tech and team leads and challenge proposed solutions with your own ideas.
Data Engineer – CloudFactory – Requirements
- Ability to work across global teams and working with different cultures across different time zones with strong communication and collaboration skills
- Tendency to go above and beyond to make things work; manage own and others work to meet the deadline and assist other team members in their deliverables
- Ability to identify solutions and make the complex simple.
- Ability to breakdown complex problems into simple solutions
- Some experience and knowledge of a coding language such as Python.
- Good experience and knowledge of the SQL query language.
- Some understanding of star schemas and data warehouse concepts..
- Some knowledge of AWS tools and technologies (i.e. Lambda, S3, SQS, SNS, DDB, RDS etc).
- Beneficial – ETL and ELT experience – both batch and microservices-led.
- Beneficial – Some Snowflake experience.
- Snowflake (including SnowPipe, streams, security, integrations etc)
- AWS Data Lake (SNS, SQS, S3, Glue, Athena, DynamoDB etc)
- AWS Data Streams (Kinesis, Elastic Search, LogStash, Lambda, API-G etc)
- Great Mission and Culture
- Meaningful Work
- Growth Opportunities
- Market Competitive Salary
- Health and Medical
Would you like to learn more about Database Administration, CLICK HERE to have a look at the top schools.
For all your IT certification needs, please, click here for information on how to get started