Skip to main content

Schema on write

Here's a short story about schema on read vs schema on write in markdown format:

The Tale of Two Data Architects

A Story of Schema Decisions​

Sarah and Mike were two data architects working at competing e-commerce companies. Both were tasked with designing their company's next-generation data platform, but they chose different paths.

Mike's Schema-on-Read Adventure​

Mike, excited about the flexibility of modern data lakes, chose a schema-on-read approach:

"We'll store everything raw and define the schema when we query. It's so flexible!" he proclaimed proudly.

Initially, things went smoothly. His team stored JSON, CSV, and log files in their data lake. They saved time on data ingestion, and developers loved the freedom.

But then the problems began:

  • Data scientists complained about inconsistent data formats
  • Query performance suffered as each read operation needed to parse and interpret the data
  • Data quality issues weren't caught until late in the process
  • Multiple teams interpreted the same data differently
  • Processing costs increased with each read

Sarah's Schema-on-Write Success​

Sarah, meanwhile, took a more traditional schema-on-write approach:

"We'll define our schemas upfront and validate data at ingestion," she explained to her team.

Her initial setup took longer:

  • Careful schema design
  • Data validation rules
  • ETL pipeline development
  • Documentation of data structures

But as months passed, Sarah's decision proved valuable:

✅ Data quality was consistently high ✅ Query performance was blazing fast ✅ Analytics teams worked with confidence ✅ Storage costs were optimized ✅ Data governance was straightforward

The Outcome​

One year later, Mike's team was drowning in technical debt, spending more time cleaning and interpreting data than generating insights. They began a costly migration to a schema-on-write approach.

Sarah's team, though moving slower at first, had built a robust, reliable data platform that became a competitive advantage for their company.

Key Takeaways​

Schema-on-write advantages:

  • Better data quality
  • Improved query performance
  • Clear data governance
  • Lower long-term maintenance costs
  • Reduced data inconsistencies

While schema-on-read offers flexibility, the long-term benefits of schema-on-write often outweigh the initial investment, especially for enterprise-scale applications where data quality and performance are crucial.