Lakehouse & Apache Iceberg on Redshift

Not every byte belongs inside the warehouse. Cold history, raw event archives, data shared with partners, and tables that Spark, Athena and Redshift all need to read are better kept in S3 in an open format. Redshift has three ways to reach that data, and the choice between them shapes cost, performance and governance for years. We design the lakehouse layer, set up the permissions, and migrate what is already there.

Spectrum, Iceberg or S3 Tables

Redshift Spectrum queries files in S3 through external tables registered in the AWS Glue Data Catalog. It is mature and works with plain Parquet, ORC, CSV and JSON. It has no notion of transactions, schema evolution or row-level updates, so tables are append-only by convention and partition management is yours.

Apache Iceberg tables add ACID transactions, schema and partition evolution, time travel and hidden partitioning on top of the same S3 files. Redshift queries Iceberg tables through the Glue Data Catalog, and the same tables are written and read by Spark on EMR or Glue, Athena and third-party engines. This is the format we recommend for any new shared dataset.

Amazon S3 Tables are Iceberg tables with S3 doing the maintenance: compaction, snapshot expiry and unreferenced-file cleanup run automatically, and the tables are registered in the catalog for you. For teams without a Spark maintenance job, S3 Tables remove the operational cost that made Iceberg daunting.

Which one fits depends on who writes the data, how often it changes, and what engines need it. We write that down per dataset rather than picking one for everything.

Catalog and permissions

A lakehouse is only as good as its permission model. We set up the Glue Data Catalog structure (databases, naming, ownership), register Redshift to use it, and move access control into AWS Lake Formation so that table, column and row-level permissions apply consistently whether a user comes through Redshift, Athena or Spark. Cross-account sharing is done with Lake Formation resource links and Redshift data sharing, not by copying buckets.

Cost model: query in place or load

Scanning Parquet in S3 from Redshift is billed per terabyte scanned (Spectrum) or in RPU time (Serverless). Loading the same data into Redshift Managed Storage costs storage per month and makes every query faster. The crossover depends on query frequency and how much of the table each query touches. Our assessment models both for your real queries, and the usual answer is a split: hot, frequently joined data inside Redshift; cold, wide or shared data in Iceberg.

Migrating Spectrum external tables to Iceberg

Many customers have years of Spectrum external tables with hand-managed partitions. We migrate them in place: convert or rewrite the Parquet into Iceberg tables (or S3 Tables), register them, repoint Redshift external schemas, and retire the partition-management scripts. Queries keep working during the migration because both the old and new tables are readable from the same external schema pattern.

Deliverables

• Lakehouse design document: per-dataset format decision, catalog layout, permission model
• Configured Glue Data Catalog, Lake Formation permissions and Redshift external schemas
• Migration of existing Spectrum tables to Iceberg or S3 Tables with validation queries
• Cost comparison of query-in-place vs load for your top queries

See also Redshift Data Sharing Across Accounts for the warehouse-side sharing pattern that complements the lakehouse.

Talk to us

Tell us what your cluster looks like today and what it costs, and we will come back with a written assessment and a fixed-scope proposal. Contact us or call +1 (726) 227-3497.