Job Description
Job Description
Hybrid Hadoop Engineer and Hadoop Infrastructure Administrator to build and maintain a scalable and resilient Big Data framework to support Data Scientists.
As an administrator, your responsibility will be to deploy and maintain Hadoop clusters, add and remove nodes using cluster management and monitoring tools like Cloudera Manager, and support performance and scalability requirements, in support of our Data scientist's needs.
Some Relational Database administrator experience will also be desirable to support the general administration of Relational Databases.
Design, build, and maintain Big Data workflows / pipelines to process a continuous stream of data with experience in end-to-end design and build process of Near-Real-Time and Batch Data Pipelines.
Demonstrated work experience in the following with Big Data and distributed programming models and technologies
Knowledge of database structures, theories, principles, and practices (both SQL and NoSQL).
Active development of ETL processes using Spark or other highly parallel technologies, and implementing ETL / data pipelines
Experience with Data technologies and Big Data tools, like Spark, Kafka, Hive
Understanding of Map Reduce and other Data Query and Processing and aggregation models
Understanding of challenges of transforming data across distributed clustered environment
Experience with techniques for consuming, holding, and aging out continuous data streams
Ability to provide quick ingestion tools and corresponding access APIs for continuously changing data schema, working closely with Data Engineers around specific transformation and access needs
Preferred :
Experience as a Database administrator (DBA) will be responsible for keeping critical tools database up and running
Building and managing high-availability environments for databases and HDFS systems
Familiarity with transaction recovery techniques and DB Backup
Skills and Attributes :
Ability to have effective working relationships with all functional units of the organization
Excellent written, verbal, and presentation skills
Excellent interpersonal skills
Ability to work as part of a cross-cultural team
Self-starter and Self-motivated
Ability to work without lots of supervision
Works under pressure and is able to manage competing priorities.
Hide
Software Engineer - Data Infrastructure
The data platform team is a collaborative team that develops a full range of data stores and data technologies, spanning from big data, through NoSQL, cache-layer capabilities, and analytics; all the way to structured SQL engines. The data platform team is responsible for the automation of data plat...
Lead Data Engineer
Active Learning, Agile Methodology, Amazon Redshift, Apache Hadoop, Apache Spark, Application Integrations, AWS Cloud Computing, Data Pipelines, Data Science, Detail-Oriented, Emerging Technologies, Enterprise Application Delivery, Group Problem Solving, Kotlin Programming Language, Machine Learning...
Data Engineer
As a Data Engineer, you will be responsible for handling a large volume of data from back-end apis, mobile applications and IoT devices. Your primary responsibility will be creating meaningful insights from this data, developing and training machine learning models, and ensuring the efficient flow o...
Machine Learning Engineer - EMR Data and NLP Initiatives
Experience with the complexities of EMR data and the regulatory landscape of healthcare data privacy. We are seeking a highly experienced Machine Learning Engineer with a specialized focus on Natural Language Processing (NLP) to join the Peter Munk Cardiac Centre AI Team at UHN. This position will p...
Senior Data Engineer
Ness is a full lifecycle digital engineering firm offering digital advisory through scaled engineering services. As your tech partner, we help engineer your company’s future with cloud and data. Combining our core competence in engineering with the latest in digital strategy and technology, we seaml...
Data Engineer
We're looking for a skilled and enthusiastic Data Engineer to join our growing team! We're a close-knit group building and maintaining mission-critical data pipelines, and we need someone who can hit the ground running. Collaborate with data analysts and stakeholders to define data requirements, and...
Senior Data Engineer
Over 8 years of experience in Data Engineering, with a focus on Data Warehouses and modern data platforms. We are seeking a highly skilled and experienced Senior Data Engineer to join our Data Platform team at OMERS. The ideal candidate will have over 8 years of experience working with Data Warehous...
Senior Consultant, Data Privacy Engineer
Data literacy: finding and managing data, cleansing data, manipulating data. As a Data Privacy Engineer, you will be at the forefront of Deloitte's mission to ensure data privacy and protection. Responsible for assisting with the management of the data privacy, data protection, data usability, perfo...
Data Engineer
We're looking for a skilled and enthusiastic Data Engineer to join our growing team! We're a close-knit group building and maintaining mission-critical data pipelines, and we need someone who can hit the ground running. Collaborate with data analysts and stakeholders to define data requirements, and...
Data Engineer (Large Language Models)
We are currently seeking an experienced Data Engineer with a strong background in setting up scalable data pipelines to process large volumes of data and handle data preparation, including labeling, for fine-tuning and (re)training large language models. Knowledge of data engineering best practices,...