Architecture & Governance
Data Quality Engineering
Beyond standard data movement, I engineer custom, cost-optimized data quality engines and enforce rigorous enterprise governance protocols to ensure absolute reporting integrity.
Custom DQM Engines (Cost-Optimized)
Proprietary logic engineered to bypass expensive third-party MDM software.
Entity Resolution & Deduplication Index
The Problem: Legacy legal databases contained severely repeating MatterId records, threatening reporting integrity.
The Engine: Engineered a zero-cost deduplication engine within Azure SQL staging using advanced T-SQL Window Functions to isolate 'Rank 1' records, avoiding expensive MDM software.
Metric Tracked: Strict Uniqueness (100% duplicate elimination prior to API injection).
Temporal SLA Consistency Parser
The Problem: Jira Service Management ticket data was trapped in heavily nested JSON payloads, making SLA breach reporting impossible natively.
The Engine: Built a serverless parser using Power Automate to intercept REST API JSON payloads, flattening deeply nested timestamp arrays into a standardized tabular model.
Metric Tracked: Temporal Consistency (Ensuring 100% accuracy of Incident SLA violation tracking).
Referential Integrity Orphan Bridging
The Problem: Migrating 24 months of legacy AS400/DB2 data into a Fabric Lakehouse resulted in orphan fact records missing parent dimensions.
The Engine: Wrote PySpark logic in the Silver Layer to detect missing keys and dynamically generate 'Structural Dummy Records' to bridge the gap without dropping financial transactions.
Metric Tracked: Referential Integrity (Maintained 100% financial transaction completeness).
Point-of-Entry Telemetry Validation
The Problem: Factory floor operational data was plagued by typos and schema mismatches due to manual Excel entry, crashing ETL pipelines.
The Engine: Scrapped Excel and built a custom Power Apps front-end validation layer enforcing strict Regex constraints and dynamic type-casting before database insertion.
Metric Tracked: Schema Conformity & Accuracy (Reduced downstream ETL failures by 99%).
API Payload Truncation & Throttling
The Problem: Legacy on-premise string data exceeded Dynamics Dataverse schema limits, causing ADF API timeouts.
The Engine: Built a metadata-driven data profiler in Azure SQL staging to identify string violations, applying controlled truncation and tuning ADF Data Integration Units (DIUs).
Metric Tracked: Format Conformity & Pipeline SLA (Zero API timeout failures).
Core Enterprise DQM Standards
Standardized operational metrics implemented across enterprise pipelines.
Incremental Load Freshness
Measured refresh SLA latency using dynamic Watermark Tables, ensuring point-in-time financial data syncs via Power BI Direct Lake mode.
Source-to-Target Reconciliation
Automated row-count checksums between 600+ legacy SQL Servers and the Cloud Data Warehouse, guaranteeing zero data loss.
PII Anonymization Rate
Leveraged Azure SQL Column/Row-Level Security and Dynamic Data Masking to hide sensitive citizen data, ensuring 100% privacy compliance.
Cross-Company Unit Standardization
Monitored the conversion accuracy of differing units of measure (UOM) and currencies into standardized global benchmarks within SSAS OLAP cubes.
Critical Dimension Null-Tolerance
Established a <0.01% tolerance for NULLs in critical fields, triggering automated ETL pipeline halts to protect executive dashboard integrity.