Extensible File Allocation Table (exFAT)

Zoran Iliev 2010

History
2006, on Windows Hardware Engineering Conference (WinHEC), Microsoft presented Personal Storage: Opportunities and challenges for pocket-sized storage devices in the Windows world. Same Conference Vishal Ghotge - Program Manager, Core File Services, Microsoft Corporation presents exFAT

The design goals of exFat

Be as similar to FAT32 as possible Overcome FAT32s limitations Design for the future Make it extensible Scale to large volume sizes Manage large files Designed to optimize for flash media exFAT A simple, extensible, rich filesystem for next generation devices and flash media.

Some Key Features Of exFAT

Free space bitmap Identifies free and available clusters on the volume allowing Faster look-up for free space allocation, Faster deletes Support for large files File size field is 8 bytes, Theoretical maximum file size of 264 bytes Large cluster size The on-disk structure limit for cluster size is 2255, Implementation limit is 32 MB
Source: download.microsoft.com/download/5/b/9/5b97017b.../sto072_wh06.ppt

Some Key Features Of exFAT

Extensibility for the future - Directory structure is flexible and extensible - Support for more than 216 (65,536) files in a single directory - Timestamps can be in UTC rather than only local time (starting with Vista SP2). - Easy to add new features and can add some features without breaking compatibility Name hashes - 2 byte checksum of up-cased file name, allows for faster checks of filenames OEM specific parameters and types - Feature to store OEM specific volume information Support for access control lists (not supported yet ) Support for Transaction-Safe FAT File System (TFAT) (optionally WinCE activated function)

exFAT vs FAT 32
exFAT is not backward compatible with FAT32 There are no short names in exFAT

Disk Space Overhead

Default cluster sizes for exFAT

VOLUME Structure

BOOT Regions Main and Backup

Boot Sector On Disk Hex View

Boot Sector On Disk Hex View

JumpBoot (0, 3) Contains the jump instruction for CPUs common in personal computers, which, when executed, "jumps" the CPU to execute the bootstrapping instructions in the BootCode field

Boot Sector On Disk Hex View

File System Name (3, 8) The FileSystemName field contains the name of the file system on the volume, "EXFAT in ASCII characters, which includes three trailing white spaces.

Boot Sector On Disk Hex View

Must Be Zero (11, 53) The MustBeZero field directly corresponds with the range of bytes the packed BIOS parameter block consumes on FAT12/16/32 volumes. The valid value for this field is 0, which helps to prevent FAT12/16/32 implementations from mistakenly mounting an exFAT volume.

Boot Sector On Disk Hex View

Partition Offset (64, 8) 02 00 = 8192 The PartitionOffset field describes the media-relative sector offset of the partition which hosts the given exFAT volume. This field aids boot-strapping from the volume using extended INT 13h** on personal computers.

**Extended Int13h functions address sectors using Logical Block Addressing

Boot Sector On Disk Hex View

Volume Length (72, 8) F4 40 00= 16007168 * 512= 8195670016 7.6GB The VolumeLength field describes the size of the given exFAT volume in sectors.

Boot Sector On Disk Hex View

FAT Offset (80, 4) 08 00 = 2048 The FatOffset field describes the volume-relative sector offset of the First FAT.

Boot Sector On Disk Hex View

FAT Length (84, 4) 07 C0 = 1984 The FatLength field describes the length, in sectors, of each FAT table (the volume may contain up to two FATs, not implemented yet).

Boot Sector On Disk Hex View

Cluster Heap Offset (88, 4) 10 00 = 4096 The ClusterHeapOffset field describes the volume-relative sector offset of the Cluster Heap. This field enables implementations to align the Cluster Heap to the characteristics of the underlying storage media and it is the Physical Sector Location of the Bitmap.

Boot Sector On Disk Hex View

CLUSTER COUNT (92, 4) 03 D0 C0 = 250048 The ClusterCount field describes the number of clusters the Cluster Heap contains. The value of the ClusterCount field determines the minimum size of a FAT.

Boot Sector On Disk Hex View

First Cluster of Root Directory (96, 4) 04 = 4 The FirstClusterOfRootDir ectory field contains the cluster index of the first cluster of the root directory. The FirstClusterOfRootDir ectory field contains the cluster index of the first cluster of the root directory.

Boot Sector On Disk Hex View

Volume Serial Number (100, 4) The Volume Serial Number field contains a unique serial number.

Boot Sector On Disk Hex View

File System Revision (100, 4) 1.0 The high-order byte is the major revision number and the loworder byte is the minor revision number. For example, if the highorder byte contains the value 01h and if the low-order byte contains the value 05h, then the FileSystemRevision field describes the revision number 1.05.

Boot Sector On Disk Hex View

Volume FLAGS (106, 2) The VolumeFlags field contains flags which indicate the status of various file system structures on the exFAT volume.

Boot Sector On Disk Hex View

Bytes Per Sector Shift (108, 1) 29 = 512 Bytes per Sector The BytesPerSectorShift field describes the bytes per sector expressed as binary logarithm log2(N), where N is the number of bytes per sector.

Boot Sector On Disk Hex View

Sectors Per Cluster Shift (109, 1) 26 = 64 Sectors per Cluster The Sectors PerClusterShift field describes the bytes per sector expressed as binary logarithm log2(N), where N is the number of bytes per sector.

Boot Sector On Disk Hex View

Number of Fats (110, 1) The NumberOfFats field describes the number of FATs and Allocation Bitmaps the volume contains. Can be 1 or 2.

Boot Sector On Disk Hex View

Drive Select (111, 1) The DriveSelect field contains the extended INT 13h drive number, which aids boot-strapping from this volume using extended INT 13h on personal computers.

Boot Sector On Disk Hex View

Percent In Use (112, 1) 0% - no files written The PercentInUse field describes the percentage of clusters in the Cluster Heap which are allocated. FFh-indicates the percentage of allocated clusters in the Cluster Heap is not available; 00 indicates backup of the Main Boot region.

Boot Sector On Disk Hex View

Percent In Use (112, 1) 9% in use

Boot Sector On Disk Hex View

Reserved (113, 7)

Boot Sector On Disk Hex View

Boot Code (120, 390) The BootCode field contains bootstrapping instructions.

Boot Sector On Disk Hex View

Boot Signature (510, 2)

Boot Sector Contd

First 9 sectors (0 to 8) are marked with the x55 AA signature Sector 11 contains Boot Checksum, 4 bytes value is repeated from the beginning to the end of the sector. This value, generated during format and is copied in the backup VBR, is a checksum of the other areas of the boot region, not including the volume flags or the percent-in-use fields. Sectors 12-23 containing a complete backup of the first 12 sectors of the volume

Sector 10

Sector 11 Boot Checksum

Sector 12

FAT Region
A FAT describes cluster chains in the Cluster Heap. A cluster chain is a series of clusters which provides space for recording the contents of files, directories, and other file system structures. A FAT represents a cluster chain as a singly-linked list of clusters indices. With the exception of the first two entries, every entry in a FAT represents exactly one cluster and each entry is 4 bytes in size.

FAT Region

Data Region
The Data region contains the Cluster Heap, which provides managed space for file system structures, directories, and files. The Cluster Heap's structure is very simple; each consecutive series of sectors describes one cluster, as the SectorsPerClusterShift field defines. Importantly, the first cluster of the Cluster Heap has index two, which directly corresponds to the index of Fat Entry 2.

Allocation Bitmap
In an exFAT volume, an Allocation Bitmap maintains the record of the allocation state of all clusters. This is a significant difference from exFAT's predecessors (FAT12, FAT16, and FAT32), in which a FAT maintained a record of the allocation state of all clusters in the Cluster Heap.

Allocation Bitmap
An Allocation Bitmap records the allocation state of the clusters in the Cluster Heap. Each bit in an Allocation Bitmap indicates whether its corresponding cluster is available for allocation or not. An Allocation Bitmap represents clusters from lowest to highest index. For historical reasons, the first cluster has index 2. Note: the first bit in the bitmap is the lowest-order bit of the first byte.

Directory Structure
The exFAT file system uses a directory tree approach to manage the file system structures and files which exist in the Cluster Heap. Directories have a one-to-many relationship between parent and child in the directory tree. The directory to which the FirstClusterOfRootDirectory (96, 4) field refers is the root of the directory tree. All other directories descend from the root directory in a singly-linked fashion. Each directory consists of a series of directory entries.

Example: Calculating the first sector of the Root entry

Directory Structure
One or more directory entries combine into a directory entry set which describes something of interest, such as a file system structure, sub-directory, or file.

Volume Label Directory Entry

The Volume Label is a Unicode string which enables end users to distinguish their storage volumes. In the exFAT file system, the Volume Label exists as a critical primary directory entry in the root directory .

Volume Label Directory Entry

Entry Type (0, 1)

Volume Label Directory Entry

Character Count (1, 1)

The CharacterCount field contains the length of the Unicode string the VolumeLabel field contains.

Volume Label Directory Entry

Volume Label (2, 22)

The VolumeLabel field contains a Unicode string, which is the user-friendly name of the volume. Maximum 11 characters Volume Label Name.

Volume Label Directory Entry

Reserved (24, 8)

Volume Label Directory Entry

If the volume was not given a label, the first value for the first entry will be: 0x03 instead of 0x83. Incidentally, renaming of the Volume name changes the inUse flag to zero to the previous Volume Label Directory Entry and creates new Volume Label Directory Entry. Nothing else is changed within the previous Volume Label Directory Entry.

Allocation Bitmap Directory Entry

In the exFAT file system, a FAT does not describe allocation state of clusters; rather, an Allocation Bitmap does. Allocation Bitmap exist in the Cluster Heap and have corresponding critical primary directory entries in the root directory. The NumberOfFats field determines the number of valid Allocation Bitmap directory entries in the root directory.

Allocation Bitmap Directory Entry

If the NumberOfFats field contains the value 1, then the only valid number of Allocation Bitmap directory entries is 1. If the NumberOfFats field contains the value 2, then the only valid number of Allocation Bitmap directory entries is 2.

Allocation Bitmap Directory Entry

Entry Type (0, 1)

Allocation Bitmap Directory Entry

Bitmap Flags (1, 1)

0, which means the given directory entry describes the First Allocation Bitmap 1, which means the given directory entry describes the Second Allocation Bitmap and is possible only when NumberOfFats contains the value 2

Allocation Bitmap Directory Entry

Reserved (2, 18)

Allocation Bitmap Directory Entry

First Cluster (20, 4)

This field contains the index of the first cluster of the cluster chain, as the FAT describes, which hosts the Allocation Bitmap.

Allocation Bitmap Directory Entry

Data Length (24, 4) 7A 18 = 31256 = 30.5 Kb

Up-Case Table Directory Entry

The Up-case Table defines the conversion from lower-case to upper-case characters. This is important due to the File Name directory entry using Unicode characters and the exFAT file system being case insensitive and case preserving. The Up-case Table exists in the Cluster Heap and has a corresponding critical primary directory entry in the root directory.

Up-Case Table Directory Entry

Entry Type (0, 1)

Entry Type 82

Up-Case Table Directory Entry

Reserved (1, 3)

Up-Case Table Directory Entry

Table Checksum (4,4)

The TableChecksum field contains the checksum of the Up-case Table (which the FirstCluster and DataLength fields describe). Implementations shall verify the contents of this field are valid prior to using the Up-case Table.

Up-Case Table Directory Entry

Reserved (8, 12)

Up-Case Table Directory Entry

First Cluster (20, 4)

This field contains the index of the first cluster of the cluster chain, as the FAT describes, which hosts the Up-case Table.

Up-Case Table Directory Entry

Data Length (20, 4) 16 CC = 5836 = 5.699 Kb

Volume Guid Directory Entry

The Volume GUID directory entry contains a GUID which enables implementations to uniquely and programmatically distinguish volumes. The Volume GUID exists as a benign primary directory entry in the root directory.

File Directory Entry

File directory entries describe files and directories. They are critical primary directory entries and any directory may contain zero or more File directory entries. For a File directory entry to be valid, at most one Stream Extension Directory entry and at least one File Name Directory entry immediately follow the File directory entry.

File Directory Entry

Entry Type (0, 1)

Entry Type 85

File Directory Entry

Secondary Count (1, 1)

The SecondaryCount field describes the number of secondary directory entries which immediately follow the given primary directory entry. These secondary directory entries, along with the given primary directory entry, comprise the directory entry set.

File Directory Entry

Set Checksum (2, 2)

The SetChecksum field contains the checksum of all directory entries in the given directory entry set. The checksum excludes this field

File Directory Entry

File Attributes (4, 2) 20h = 32 (25)-> Archive

File Directory Entry

Reserved 1 (6, 2)

File Directory Entry

Create 10ms Increment1 (20, 1)
Create Timestamp 1 (8, 4)

In combination, the CreateTimestamp and CreateTime10msIncrement fields describe the date and time the given file/directory was created. These two fields conform to the definitions of the Timestamp and 10msIncrement fields.

File Directory Entry

Timestamp Field Structure :

0 4 Seconds divided by 2 (0 = 0 sec., 29 = 58 sec.) 5 10 Minutes 0 59 11 15 Hours 0 23 16 20 Days 1 31 21 24 Months (01 = January 02 = February etc. ) 25 31 Years since 1980

File Directory Entry

10ms Increment Fields Structure :

The valid range of values for these fields is:

At least 0, which represents 0 milliseconds At most 199, which represents 1990 milliseconds

File Directory Entry

Create 10ms Increment1 (20, 1) 50h = 80 (* 10) = 800ms Create Timestamp 1 (8, 4) 25/02/2010 16:39:54

File Directory Entry

Last Modified 10ms Increment1 (21, 1) 00 ?? Last Modified Timestamp 1 (12, 4) 25/02/2010 16:20:52

File Directory Entry

Time Zone Created (22, 1) Time Zone Modified (23, 1)

Time Zone Accessed (24, 1)

Time Zone Calculation

There is one byte for each Time Stamp (Created, Modified and Accessed). The bit assignment is: - 1 bit reserved (the most significant bit) - the other 7 bits are representing a signed integer that is recording the number of 15 minutes intervals.

Time Zone Calculation

A8h = 10101000 -> 101100 = 40 X 15 = 600 min => Time Zone +10 (Canberra, Sydney)

File Directory Entry

Reserved 2 (25, 7)

Stream Extension Directory Entry

The Stream Extension directory entry is a critical secondary directory entry in File directory entry sets. This directory entry is valid only if it immediately follows the File directory entry.

Stream Extension Directory Entry

Entry Type (0, 1)

Entry Type C0

Stream Extension Directory Entry

Secondary Count (1, 1)

Stream Extension Directory Entry

Reserved 1 (2, 1)

Stream Extension Directory Entry

Name Length 1 (3, 1)

The NameLength field contains the length of the Unicode string the subsequent File Name directory entries collectively contain.

Stream Extension Directory Entry

Name Hash 1 (4, 2)

The NameHash field contains a 2-byte hash of the upcased file name. This enables implementations to perform a quick comparison when searching for a file by name. Importantly, it provides a sure verification of a mismatch. Implementations shall verify all NameHash matches with a comparison of the up-cased file name.

Stream Extension Directory Entry

Reserved 2 (6, 2)

Stream Extension Directory Entry

Valid Data Length (8, 8)

The ValidDataLength field describes how far into the data stream user data has been written. Implementations shall update this field as they write data further out into the data stream.

Stream Extension Directory Entry

Reserved3 (16, 4)

Stream Extension Directory Entry

First Cluster (20, 4)

Stream Extension Directory Entry

Data Length (28, 4)

File Name Directory Entry

File Name directory entries are critical secondary directory entries in File directory entry sets. The valid number of File Name directory entries in a File directory entry set is NameLength/15, rounded up to the nearest integer. File Name directory entries are valid only if they immediately follow the Stream Extension directory entry as a consecutive series. File Name directory entries combine to form the file name for the File directory entry set.

File Name Directory Entry

Entry Type (0, 1)

Entry Type C1

File Name Directory Entry

General Secondary Flags Field (1, 1)

File Name Directory Entry

File Name(2, 30)

File Name Directory Entry

The file names "." and ".." have the special meaning of "this directory" and "containing directory", respectively. Implementations shall not record either file name in the FileName field. However, implementations may generate these two file names in directory listings to refer to the directory being listed and the containing directory.

File Creation

By Examining the FAT entries we can find the allocated clusters. We know that the First cluster is 78h, so we go to the beginning of the FAT, at sector 128, which is defined in the boot sector:

So, if we do the math, we have: First Fragment: 4999-119 = 4880 Second Fragment: 7135-5607 = 1528 FILE SIZE: = 6408 CLUSTERS

Deleted Files
Directory Entry after file deletion: As we already mentioned, The InUSE bit describes whether the given directory entry in use or not. The valid values for this field are: 0, which means the given directory entry is not in use; this means the given structure actually is an unused directory entry 1, which means the given directory entry is in use; this means the given structure is a regular directory entry Accordingly, when a file or directory is deleted, the flag changes to 0, indicating that this directory entry is free or unused directory entry.

Deleted Files
FAT after file deletion No change until re-used for the next fragmented file. BITMAP after file deletion The bits that were showing in use (1) are immediately changed to not in use (0)

Recovery of Contiguous files in exFAT

Questions?
Special Thank you to my colleague and friend Steve Ilett. Thank you all for attending this session!

Bibliography
Ravisankar V. Pudipeddi and Vishal Ghotge and Ravinder Thind 2009: Contiguous Allocation in an Extensible File System, US Patent Application Publication, Pub. No.: US 2009/0164539 A1. Ravisankar V. Pudipeddi and Vishal Ghotge and Ravinder Thind 2009: Quick Filename Lookup Using Name Hash, US Patent Application Publication, Pub. No.: US 2009/0164440 A1. Jeff Hamm, Extended FAT File System, 2009 Article ID: 955704 - Last Review: September 29, 2009 - Revision: 3.1: Description of the exFAT file system driver update package, http://support.microsoft.com/kb/955704, accessed February 2009. Raymond Chen 2006: Windows Confidential, A Brief and Incomplete History of FAT32, TechNet Magazine. Vishal Ghotge, (Program Manager, Core File Services Microsoft Corporation), exFAT (PPT), Win HEC 2006