No longer accepting applications
Senior Civil Engineer
DNV Germany Holding GmbHToronto, ON, Canada7 days ago
Job type
- Full-time
Job descriptionCurrent Design document Onboarding sessions explaining existing code structure Access to development environment (Databrick, GCP, Git) (Weekly / monthly...) National security If applicable / Export Classification
If you find this opportunity aligns with your career goals and interests, we kindly request that you send us your documents via E-mail.
Job Title
Data Architect / Engineer
Job Type
Remote
Description
Skills
Data Bricks + Unity Catalog + Terraform + Pyspark Notebook
Technical Activities / Description of Requirements
- N1 Current design review & design updates if required
- N2 Create buckets for each component and environment in Terraform
- N3 Create Meta store and Catalogues in Terraform and assign buckets
- N4 Create users / groups / service principals in Terraform and assign access rights
- N5 Create new DDL and table library for Unity Catalogue structure
- N6 Migrate to shared clusters and refactor / get notebook context info
- N7 Update all notebooks to use new DDL and table library
- N8 Complete validation of Unity Catalogue on Dev
- N9 Data migration workflow with result report
Migrate existing data from existing Delta Tables (saved in Databricks system buckets) to Delta Tables saved in external locations (specific buckets).
Inputs from Thales to perform the activities
Classification
J-18808-Ljbffr