Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
Iceberg has gained a huge amount of popularity in recent years, but why is this table format now finding such widespread adoption? There are a number of reasons and I shall attempt to explain them in this post.
Before Iceberg, Data Lakes were a mess because of the way they stored data (here is an example using parquet):
data-lake/
├── sales/
│ ├── year=2023/
│ │ ├── month=01/
│ │ │ ├── part-0001.parquet
│ │ │ ├── part-0002.parquet
│ │ │ └── part-0003.parquet
│ │ └── month=02/
│ └── year=2024/
- No schema evolution (add a column = rewrite everything)
- No ACID transactions (concurrent writes corrupt data)
- Slow queries (scan millions of files to find relevant data)
- No time travel (can’t view yesterday’s data)
- Manual partition management (error-prone)
- Expensive operations (DELETE/UPDATE require full rewrites)
- No hidden partitioning (users must know partition structure)
With Iceberg, these problems have been solved.
ACID
Atomicity, Consistency, Isolation and Durability. To understand more about what ACID really means, please read the following post:
With Iceberg being ACID compliant, it is now able to handle consurrent transactions.
-- Multiple writers can safely update the same table
-- Writer 1 (consurrent)
INSERT INTO sale VALUES (1001, 'Product A', 50.00);
-- Writer 2 (concurrent)
INSERT INTO sale VALUES (1002, 'Product B', 75.00);
-- Both succeed! No corruption.
-- Iceberg uses optimistic concurrency controlTime Travel
Iceberg provides datasets with snapshots. It means we now have the ability to revert to previous points in time:
-- select using timestamp
SELECT * FROM iceberg_table FOR TIMESTAMP AS OF TIMESTAMP '2020-01-01 10:00:00 UTC'
-- select using version
SELECT * FROM iceberg_table FOR VERSION AS OF 949530903748831860;Before Iceberg once data was overwritten, it was gone forever. Now we can:
- Reproduce reports from the past
- Audit and compliance
- Rollback bad data loads
- Compare versions
Schema Evolution
Before Iceberg, schema, changes were difficult and any change required rewriting the entire dataset. Now we are able to update the schema without any major downtime.
-- Add a column (instant operation, no data rewrite)
ALTER TABLE sale ADD COLUMN discount_amount DECIMAL(10,2);
-- Rename a column (metadata-only operation)
ALTER TABLE sale RENAME COLUMN customer_id TO client_id;
-- Change column type (with data evolution)
ALTER TABLE sale ALTER COLUMN price TYPE DECIMAL(12,2);- Add: Add a new column to the table or to a nested struct
- Drop: Remove an existing column from the table or a nested struct
- Rename: Rename an existing column or field in a nested struct
- Update: Widen the type of a column, struct field, map key, map value, or list element
- Reorder: Change the order of columns or fields in a nested struct
Iceberg schema updates are metadata changes, so no data files need to be rewritten to perform the update.
Hidden Partitioning
Before Iceberg, users had to know and specify partition columns manually which was an art in and of itself. Partitioning is a way to make queries faster by grouping similar rows together when writing. For example, in SQL, querying flights between 12pm and 1pm would result in a query as shown:
SELECT level, message
FROM flight
WHERE flight_time BETWEEN '2022-01-01 12:00:00' AND '2022-01-01 13:00:00';In traditional Hive-based data lakes, partitions were grouped by a particular column into physical folders. For example, data might be partitioned by year, month, and day, with each partition containing the relevant Parquet files.
This approach lacked flexibility as the partition structure was coupled to the physical storage. If a partition needed to be changed, it would require the entire dataset to be restructured which would be both resource and time consuming. The same applies to removing a partition column.
Iceberg uses metadata files to store the partition information and does not rely on the physical storage structure. This approach makes partition evolutyion flexible. We could start with a yearly partition, but as the data volumes increase we might want to move to a monthly partition. Using Iceberg, we alter the table adding the month column as a partition column. This means new rows will be grouped by year and month.
Efficient Updates and Deletes
Before Iceberg, DELETE and UPDATE operations meant rewriting the entire dataset.
- Copy-on-write (CoW) or merge-on-read (MoR) strategies
- Only affected files are rewritten
- Position deletes for MoR
- Much faster than full table rewrites
File-Level Metadata and Pruning
Before Iceberg we had to scan every file to check if it contained relevant data. For example, consider the following query.
SELECT * FROM sales
WHERE order_date = '2024-10-15'
AND amount > 1000;Iceberg scans only files where the order_date range includes 2024-10-15 and amount > 1000. This potentially skips 99% of files. This is discussed against a broader context in this post.
Multiple Engine Support
Iceberg now works with all major query engines.
This means swapping between engines is now trivial and does not involve complex re-writes and costly development cycles.
Performance
The performance improvements are dramatic in certain use cases.
Before Iceberg:
-- Query 100 TB table
-- Scans: 500,000 files
-- Time: 45 minutes
SELECT COUNT(*) FROM sales
WHERE order_date = '2024-10-15';With Iceberg:
-- Same query
-- Scans: 150 files (min/max pruning)
-- Time: 12 seconds
SELECT COUNT(*) FROM sales
WHERE order_date = '2024-10-15';
**Real companies report:**
-- 10-100x faster queries
-- 50-90% reduction in storage costs (through compaction)As you can see, Iceberg offers many advantages over previous tble formats. Its adoption will grow and more and more improvements will make this one of the most important tools in any data engineers toolbox. There are some alternatives you should be aware of that give a different approach that may better suit your solution, for example DuckLake:
You must be logged in to post a comment.