| Requirement ID: | 93438 |
| Job Title: | Principal Databricks Data Engineer |
| Job Type: | Contract |
| Duration: | 6 - 9 months |
| Location: | Toronto, ON |
| Job Description: | Role Descriptions: Job TitlePrincipal Data Engineer Databricks (Finance Risk Cloudera Modernization)LocationToronto| Canada (Hybrid Onsite as required)Experience1218 years overall data engineering experience8 years across enterprise Data Warehouse Data Lake platforms5 years with Databricks Spark at scaleRole SummaryThe Principal Databricks Data Engineer is a senior technical leader responsible for modernizing large scale finance and risk data platforms from legacy Cloudera ecosystems to cloud native Databricks lakehouse architectures.This role demands deep hands on expertise in data warehousing| data lakes| finance risk data models| and semanticconsumption layers| with strong experience supporting regulatory| management reporting| and analytics use cases.The individual will serve as a hands on architect and technical authority| leading platform modernization while partnering closely with Finance| Risk| Analytics| and Governance stakeholders.Key ResponsibilitiesCloudera Databricks ModernizationLead modernization of legacy Cloudera platforms (CDH CDP| Hive| HBase| Impala| Spark) into Databricks Lakehouse.Redesign ingestion| transformation| and consumption patterns from HDFS centric architectures to cloud object storage and Delta Lake.Refactor legacy HiveImpala logic into PySpark Spark SQLbased ELT pipelines.Ensure data parity| reconciliation| and audit integrity during platform migration.Enterprise Data Warehouse Data Lake ArchitectureDesign and govern enterprise Data Warehouse and Data Lake Lakehouse architectures.Implement layered architectures spanning oRaw Landing zonesoCurated conformed layersoSemantic consumption layersModernize traditional EDW patterns into domain aligned| scalable lakehouse https://urldefense.proofpoint.com/v2/url?u=http-3A__designs.Finance&d=DwIGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=uPVyrU0-wRkYMC7tELyR3Z2KXITAdfezxPfpSF43vug&m=x-szm9fPh5qYJV1wFq1ZDray7stJ4KGC6xZuJuQIOe543KSbVqrFiwFDVfTY-KWt&s=NgRlujgyNjmvpDroK_5C-0IY_iW4OYfg_wjnq7SzJ0o&e= Risk Data ModelingSupport implementation of finance and risk data models| including oGeneral Ledger Sub ledger dataoAccounting events and financial hierarchiesoRisk exposure| liquidity| credit| and market risk modelsEnable aggregation| drill down| and drill back from reports to transaction level https://urldefense.proofpoint.com/v2/url?u=http-3A__data.Support&d=DwIGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=uPVyrU0-wRkYMC7tELyR3Z2KXITAdfezxPfpSF43vug&m=x-szm9fPh5qYJV1wFq1ZDray7stJ4KGC6xZuJuQIOe543KSbVqrFiwFDVfTY-KWt&s=rd4Sd1fdS_WYybTUo4wUQhZJIIZi5u_1lhtHMIAbl04&e= regulatory reporting| management reporting| and analytics use cases.Semantic Consumption LayersBuild and manage semantic consumption layers to ensure consistent business logic across oBI and reporting toolsoFinance Risk analyticsoSelf service analytics platformsDefine metrics| dimensions| hierarchies| and KPIs aligned to finance and risk definitions.Implement semantic models using oDatabricks SQLoDelta tablesodbt or equivalent transformation frameworksDatabricks Engineering OptimizationEngineer large scale pipelines using PySpark| Spark SQL| and Delta Lake.Implement medallion architecture (Bronze Silver Gold) aligned to business domains.Optimize Databricks workloads for cost| performance| and reliability (Z ORDER| OPTIMIZE| caching| cluster policies).Data Governance| Quality LineageImplement data quality frameworks| reconciliation co |
| Apply Now |