Talent.com
Senior Data Engineer

Senior Data Engineer

CBTW AmericasOntario, Canada
7 days ago
Job type
  • Full-time
Job description

About CBTW

We are a global tech solutions company that believes Collaboration Betters The World .

Leveraging strategy, technology, and design, we partner with organizations worldwide to offer comprehensive solutions from idea conception to product realization. We work with people around the globe to advise, build, run, and support the creation of products with purpose.

The role

The Sr. Data Engineer will be responsible for delivering best-in-class data integration capabilities to WELL’s next-generation platform. This role involves developing bi-directional, stream-based, near real-time synchronization frameworks, enabling EMR teams to implement streaming data pipelines, and developing a data sync chaos monkey to identify data flow issues proactively.

What you will be doing :

  • Develop bi-directional, stream-based, near real-time synchronization framework : Work closely with the Principal Architect to design and implement a system that synchronizes data bi-directionally from EMR systems to a FHIR repository, ensuring resilience against network outages, monitoring production systems, detecting and resolving data conflicts, and auditing data access.
  • Enable EMR teams to implement streaming data pipelines : Provide comprehensive guidance, tools, and training to EMR teams, develop clear documentation, and establish support sessions to ensure effective application of the synchronization framework.
  • Develop a data sync chaos monkey : Implement a system that actively tests the bi-directional sync system to identify and resolve data flow issues.

You have :

  • Change Data Capture (CDC) : Expertise in implementing CDC solutions with tools like Debezium, and familiarity with alternatives such as GoldenGate, SQL Server CDC, and Qlik Replicate.
  • Data Pipeline Development : Experience in designing and maintaining data pipelines for batch and streaming data, with knowledge of stream processing frameworks like Apache Flink and Azure Stream Analytics.
  • Azure Cloud Technologies : Proficiency in Azure services including Azure Kubernetes Service (AKS), Azure Functions, and Azure Event Hubs.
  • Database Management : Advanced knowledge of MySQL, SQL Server / Azure SQL Database, PostgreSQL, and experience with CDC and replication mechanisms.
  • FHIR and Healthcare Interoperability : Understanding of FHIR standards and integration with existing systems.
  • Programming and Scripting : Proficiency in languages such as Java, Python, Go, C#, and scripting skills for automation tasks.
  • Streaming and Messaging Systems : Experience with platforms like Apache Kafka or Azure Event Hubs.
  • Version Control and CI / CD : Expertise in using Git and CI / CD pipelines.
  • Monitoring and Troubleshooting : Experience with tools like Prometheus, Grafana, and Kibana for diagnosing integration issues.