PARQUET File
Think of a Parquet file as a very well-organized goods warehouse.
The key idea is simple:
A traditional row-based file stores everything about one item together.
Parquet stores similar attributes together in columns.
That makes Parquet especially fast when you only need a few columns from a very large dataset.
1. Imagine a warehouse
Suppose you have 1 million products with this data:
Product_IDProduct_NameCategoryQuantityPriceWarehouse1001LaptopElectronics201200Dallas1002ChairFurniture50150Austin1003MonitorElectronics30400Dallas1004DeskFurniture10700Houston
A normal row-oriented format such as CSV roughly stores it like this:
1001, Laptop, Electronics, 20, 1200, Dallas
1002, Chair, Furniture, 50, 150, Austin
1003, Monitor, Electronics, 30, 400, Dallas
1004, Desk, Furniture, 10, 700, Houston
Visually:
ROW-BASED WAREHOUSE
Shelf 1
┌─────────────────────────────────────────────┐
│ ID │ Name │ Category │ Qty │ Price │ City │
└─────────────────────────────────────────────┘
Shelf 2
┌─────────────────────────────────────────────┐
│ ID │ Name │ Category │ Qty │ Price │ City │
└─────────────────────────────────────────────┘
Shelf 3
┌─────────────────────────────────────────────┐
│ ID │ Name │ Category │ Qty │ Price │ City │
└─────────────────────────────────────────────┘
Every box contains everything about that product.
2. Parquet organizes the warehouse differently
Parquet says:
"Instead of putting all information about one product together, put similar information together."
So the warehouse becomes:
PARQUET WAREHOUSE
PRODUCT ID AISLE
┌──────┐
│ 1001 │
│ 1002 │
│ 1003 │
│ 1004 │
└──────┘
PRODUCT NAME AISLE
┌─────────┐
│ Laptop │
│ Chair │
│ Monitor │
│ Desk │
└─────────┘
CATEGORY AISLE
┌─────────────┐
│ Electronics │
│ Furniture │
│ Electronics │
│ Furniture │
└─────────────┘
PRICE AISLE
┌──────┐
│ 1200 │
│ 150 │
│ 400 │
│ 700 │
└──────┘
WAREHOUSE AISLE
┌─────────┐
│ Dallas │
│ Austin │
│ Dallas │
│ Houston │
└─────────┘
This is called columnar storage.
3. Why does this matter?
Imagine the manager asks:
"What is the average price of all products?"
With CSV, the system may need to read:
Product_ID
Product_Name
Category
Quantity
Price
Warehouse
even though you only asked for:
Price
Parquet can walk directly to the Price aisle.
Query:
SELECT AVG(price)
FROM products
↓
Parquet opens only:
┌───────────────┐
│ PRICE COLUMN │
├───────────────┤
│ 1200 │
│ 150 │
│ 400 │
│ 700 │
└───────────────┘
Everything else can be skipped.
That means:
less data read → less network traffic → less CPU → faster query → lower cloud cost
4. Parquet architecture
The internal structure is roughly:
PARQUET FILE
│
├── Row Group 1
│ │
│ ├── Product_ID Column Chunk
│ ├── Category Column Chunk
│ ├── Quantity Column Chunk
│ └── Price Column Chunk
│
├── Row Group 2
│ │
│ ├── Product_ID Column Chunk
│ ├── Category Column Chunk
│ ├── Quantity Column Chunk
│ └── Price Column Chunk
│
└── Footer Metadata
│
├── Schema
├── Row-group locations
├── Min values
├── Max values
├── Null counts
└── Encoding / compression details
Three concepts matter most.
File
The entire warehouse.
products.parquet
Row group
A large section of the warehouse.
For example:
Row Group 1 = Products 1–100,000
Row Group 2 = Products 100,001–200,000
Row Group 3 = Products 200,001–300,000
Column chunk
Inside each row group, each column gets its own storage area.
Row Group 1
│
├── Product_ID
├── Product_Name
├── Category
├── Quantity
├── Price
└── Warehouse
This combination is important.
Parquet is column-oriented inside row groups.
5. The really powerful part: metadata
Imagine every warehouse section has a sign outside:
SECTION 1
Products: 1 - 100,000
PRICE
Minimum: $5
Maximum: $2,000
WAREHOUSE
Dallas
Austin
Houston
Another section says:
SECTION 2
Products: 100,001 - 200,000
PRICE
Minimum: $2
Maximum: $900
Now you ask:
SELECT *
FROM products
WHERE price > 1500;
Parquet looks at Section 2:
Maximum price = $900
So Parquet knows:
"There cannot possibly be a product costing more than $1,500 here."
It skips that whole section.
This is called data skipping or predicate pushdown.
Conceptually:
Query:
Price > $1,500
│
▼
┌──────────────┐
│ Row Group 1 │
│ Min = $5 │
│ Max = $2000 │
└──────┬───────┘
│
│ READ
▼
┌──────────────┐
│ Row Group 2 │
│ Min = $2 │
│ Max = $900 │
└──────┬───────┘
│
└────── SKIP
That is one reason Parquet can be extremely fast.
6. Compression is much better
Imagine this Category column:
Electronics
Electronics
Electronics
Electronics
Electronics
Furniture
Furniture
Furniture
Furniture
Because similar data sits together, Parquet can compress it very efficiently.
Conceptually it might represent this as:
Electronics × 5
Furniture × 4
instead of repeating the full values.
The actual encodings are more sophisticated, but this is the basic idea.
Parquet commonly uses techniques such as:
Dictionary encoding
Run-length encoding
Bit packing
Compression
So a huge dataset might conceptually shrink like:
CSV
████████████████████████████████████████
10 GB
↓
PARQUET
████████████
3 GB
The actual compression ratio depends heavily on the data.
7. Why Parquet works so well for analytics
Suppose your warehouse contains:
1 BILLION PRODUCTS / TRANSACTIONS
Columns:
transaction_id
customer_id
merchant
category
amount
country
device
timestamp
channel
status
risk_score
A business analyst asks:
SELECT
country,
SUM(amount)
FROM transactions
GROUP BY country;
Parquet may only need:
country
amount
Instead of reading all 11 columns.
Visually:
11-column dataset
ID ❌
Customer ❌
Merchant ❌
Category ❌
Country ✅
Amount ✅
Device ❌
Time ❌
Channel ❌
Status ❌
Risk ❌
That is a major advantage when datasets are hundreds of gigabytes or terabytes.
8. Parquet versus CSV using the warehouse analogy
FeatureCSV warehouseParquet warehouseOrganizationProduct-by-productAttribute-by-attributeStorageRow basedColumn basedCompressionLimitedVery strongSchemaWeak / externalStored in fileQuery selected columnsReads lots of unnecessary dataReads required columnsLarge analyticsSlowerVery efficientMetadataMinimalRich metadataData skippingLimitedYesSpark/DatabricksSupportedHighly optimized
9. Important distinction: Parquet is not a database
Parquet is a file format.
Think:
ADLS Gen2
│
└── products/
│
├── part-00001.parquet
├── part-00002.parquet
├── part-00003.parquet
└── part-00004.parquet
Spark, Databricks, Trino, Snowflake, Fabric and other engines can read those files.
So:
Storage
│
▼
ADLS Gen2
│
▼
Parquet Files
│
▼
Databricks / Spark
│
▼
SQL / PySpark
│
▼
Analytics
10. Where Delta Lake fits in
This is especially important for your Databricks learning.
Parquet is the physical file format.
Delta Lake adds a transaction layer around Parquet.
Think of it this way:
PARQUET
=
warehouse shelves + boxes
Delta Lake adds:
Warehouse management system
+
inventory history
+
transaction log
+
change tracking
+
ACID guarantees
Architecture:
DELTA TABLE
products/
│
├── part-00001.parquet
├── part-00002.parquet
├── part-00003.parquet
│
└── deltalog/
├── 000000000000000.json
├── 000000000000001.json
└── ...
So:
Delta Lake
│
├── Parquet = stores the actual DATA
│
└── Delta Log = tracks what happened to the data
That distinction is worth remembering for DP-750.
The mental model I recommend
Memorize this picture:
GOODS WAREHOUSE
CSV / ROW STORAGE
────────────────────────────────
Product 1 → ID + Name + Qty + Price + City
Product 2 → ID + Name + Qty + Price + City
Product 3 → ID + Name + Qty + Price + City
PARQUET / COLUMN STORAGE
────────────────────────────────
ID AISLE
1001
1002
1003
NAME AISLE
Laptop
Chair
Monitor
PRICE AISLE
1200
150
400
CITY AISLE
Dallas
Austin
Dallas
Query:
"Give me average PRICE"
│
▼
PRICE AISLE
│
▼
Read only this column
The most important sentence is:
Parquet stores data by column so analytical engines can read only the data they need, compress it efficiently, and skip sections that cannot match the query.
For your Azure Databricks architecture, the relationship you want to remember is:
ADLS Gen2
↓
Parquet files
↓
Delta Lake
↓
Databricks / Spark
↓
Bronze → Silver → Gold
↓
SQL / Power BI
Where Parquet is the efficient storage format, while Delta Lake makes those Parquet files behave more like a reliable enterprise table.