Reflections from 20 years of building enterprise data platforms and scaling cloud migrations
The most sophisticated data architecture in the world will fail if the end-users and business teams find it too cumbersome to adopt.
Making adoption seamless is just as important as writing optimized ETL code. If the business cannot easily interact with the semantic layers or dashboards we build, the architecture has failed its ultimate test.
As organizations plan architectures for both traditional analytics and future AI agents, the comparison between Microsoft Fabric and Databricks is the most critical decision an architect makes.
“Choosing between Fabric and Databricks isn't about finding a universal winner, it's about matching platform strengths to your enterprise scale, team capabilities, and AI roadmap.”
Where Databricks Still Outperforms Fabric: Advanced AI & Machine Learning Scale remains unmatched for heavy custom model training, LLM fine-tuning, and MLflow governance. Multi-Cloud Flexibility runs natively across AWS, Azure, and GCP versus Fabric's strict Azure confinement, and Fine-Grained Compute Control allows granular tuning of node types, autoscaling policies, and spot instances compared to pooled capacity.
When building datasets that will feed Retrieval-Augmented Generation (RAG) pipelines and autonomous AI agents in the future, speed without governance leads to failure.
“Approximately 80% of enterprise AI projects fail to deliver their promised business value due to inadequate data foundations.”
When technical teams build pipelines without defining clear semantic rules, different departments use conflicting definitions for the same metric (e.g., 'Revenue'). AI agents inherit these conflicts, rendering automated multi-system decisions unreliable.
To bridge this gap, modern platforms like Microsoft Fabric introduce a business context layer (Fabric IQ and Ontology features) that provides standardized entity types, explicit relationship graphs, and OneLake data bindings so AI agents operate under governed guardrails.
Early in my career, I witnessed firsthand how enterprise resource planning (ERP) implementations can completely derail when technology choices become disconnected from actual business realities.
“Gartner predicts that over 70% of ERP initiatives fail to fully meet their original business case goals, with a quarter failing catastrophically.”
That failure fundamentally shaped my architectural philosophy today: Never design a data or enterprise platform in a vacuum. Every pipeline, lakehouse, or warehouse must trace a direct, measurable line back to a business objective.
The root cause is rarely the syntax or the cloud infrastructure. It happens when projects skip deep discovery phases, ignore human behavior, and fail to align software capabilities with day-to-day operations. True enterprise architecture isn't about deploying a platform; it's about building a bridge between technical execution and measurable business outcomes.
Moving legacy SQL databases straight to the cloud without restructuring data models is one of the fastest ways to stall a modernization initiative.
Without re-architecting, organizations end up paying premium cloud compute costs while maintaining the same underlying performance bottlenecks they had on-premises.