10/4/2016

TheFourStagesofNTFSFileGrowth,Part2|AsktheCoreTeam

Server & Tools Blogs > Server & Management Blogs > Ask the Core Team
Sign in

Ask the Core Team

Microsoft Enterprise Platforms Support: Windows Server Core Team

The Four Stages of NTFS File Growth, Part 2

March 12, 2015 by John Marlin [MSFT] // 11 Comments

0

A few years ago I wrote a blog entry entitled, The Four Stages of NTFS File Growth.
This attempted to explain what happens to a file as it gains complexity. Complexity being akin to fragmentation.
If you have not read the above mentioned blog entry, please do so now. This information will not make the
slightest bit of sense unless you read my earlier post. Ill wait.

http://blogs.technet.com/b/askcore/archive/2009/10/16/the
fourstagesofntfsfilegrowth.aspx
Welcome back.
Since its posting, I have answered a number of questions, mostly about the structure called the attribute list. So
today I want to cover this a little more indepth to hopefully address some of these said questions.
In the previous blog entry, I explained how very complex files had the potential of creating an attribute list
shown below.

https://blogs.technet.microsoft.com/askcore/2015/03/12/thefourstagesofntfsfilegrowthpart2/

1/10

10/4/2016

TheFourStagesofNTFSFileGrowth,Part2|AsktheCoreTeam

The base record and all the child records are each 1kb in size. Each child record keeps track of a portion of the
files data stream. The more fragmented the data stream, the more mapping pairs are required to track the
fragments, and thus the more child records will be created. Each child record must be tracked in the attribute list.
Keep in mind that the child records can hold much more than just two mapping pairs. This is just simplified to
keep the diagram from being completely unreadable.
The problem with this is that the attribute list itself. It is NOT a child record, it is created using free space outside
the Master File Table MFT. A files attribute list has a hard limit of how large it can grow. This cannot be changed.
If it were, it would break backwards compatibility with older versions of NTFS that wouldnt know how to deal
with a larger attribute list.
NOTE: The diagram shows the attribute list as being smaller than the 1kb file record. And while it is true that it
starts out that way, the upper limitation of the attribute list is 256kb.

So it is possible to hit a point where a file cannot add on any additional fragments. This is often the case when
the following error messages are encountered.
https://blogs.technet.microsoft.com/askcore/2015/03/12/thefourstagesofntfsfilegrowthpart2/

2/10

10/4/2016

TheFourStagesofNTFSFileGrowth,Part2|AsktheCoreTeam

Insufficient system resources exist to complete the requested service

The requested operation could not be completed due to a file system limitation
What these messages are trying to tell us is that the attribute list has grown to its maximum size and additional
file fragments cannot be created.
To put this into perspective, this isnt simply about file SIZE. It has to do with how fragmented the file is. In fact it
is very hard to MAKE happen. There are really only two scenarios where it is somewhat common.
Compressing very large files, like virtual hard disks VHD
Very large SQL snapshots, which are sparse
Both compressed and sparse files introduce high levels of fragmentation because of how they are stored. So very
large files that are also sparse or compressed run the risk of hitting this limitation. To add to the problem, you
cannot clear this up by running defragmentation/optimization. Sparse and compressed files are going to be
fragmented.
The good news is that we figured out a way around this. The bad news is that it isnt really well understood.
It really starts with this hotfix.

http://support.microsoft.com/kb/967351/
Installing the hotfix doesnt resolve the issue by itself. What this hotfix does is that it gives us the ability to create
instances of NTFS that use file records that are 4kb in size, rather than the 1kb that NTFS has used for the longest
time.
How is this possible? If we cant change the size of the attribute list, how can we change the size of file records?
The attribute list is a hard coded limitation. Microsoft made the decision, for performance reasons, that we really
should keep a lid on how big the attribute list should grow. On the other hand, file record size is selfdefined. By
default, the size is defined as 1kb, but records could be other sizes, as long as all the records in a volume are the
same size.
This was put to the test when 4kb sector hard drives started to become popular. Since you wouldnt want a file
record to be smaller than a sector, these 4kb sector drives were formatted to utilize a file record size of 4kb. Thats
where the hotfix comes into the picture. In addition to being able to use 4kb file records on 4kb sector hard
drives, an option was added to the FORMAT.EXE command to force it to create an instance of NTFS with 4kb file
records, regardless of sector size.
So why should we care about the size of the file records? Look at the diagram again.
https://blogs.technet.microsoft.com/askcore/2015/03/12/thefourstagesofntfsfilegrowthpart2/

3/10

10/4/2016

TheFourStagesofNTFSFileGrowth,Part2|AsktheCoreTeam

If the records are bigger, they can store more mapping pairs, and thus track more fragments. In theory, a file
could have FOUR TIMES the number of fragments before running into the same issue.
The catch is that the size of file records is set at the time of formatting. So if you have a volume that is running
into this issue, you will need to do the following.
1. Copy off your files
2. Reformat the drive using the switch Format /L
3. Copy the files back
You cant change the size of file records after the fact. It has to be set when formatting. But without an
understanding of just what it is that we are changing.
This solves the problem in the short term. For the long term, other solutions were implemented to prevent
fragmentation past a certain point. In the newer versions of Windows, NTFS will stop fragmenting compressed
and sparse files before the attribute list reaches 100% of its maximum size.
This should put the issue to rest once and for all. However, until everyone gets to Windows 8.1 or Windows
Server 2012 R2, we will still run into this issue from time to time.
For more information about 4kb sector drives, check out my article on Windows IT Pro.

http://windowsitpro.com/windows/promiseadvanced
formatharddrives
https://blogs.technet.microsoft.com/askcore/2015/03/12/thefourstagesofntfsfilegrowthpart2/

4/10

10/4/2016

TheFourStagesofNTFSFileGrowth,Part2|AsktheCoreTeam

Robert Mitchell
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

Search MSDN with Bing

Search this blog

Search all blogs

Top Server & Tools Blogs

ScottGu's Blog
Brad Andersons "In the Cloud" Blog
Brian Harry's Blog
Steve "Guggs" Guggenheimer's Blog

Share This Post

EPS Team Blogs

ASK Perf Blog
Memory Management, Performance, Printing, Terminal Server
CPR / Escalation Services
Debugging, Hangs, Tools
Directory Services
DFSR, Certificates, Group Policy
LATAM [Portugese & Spanish]
Espanol, Portugues
Manageability
SMS MOM
Networking
Scalable Networking, OCS, Communications Server
Small Business Server
Windows Essential Business Server
SoftGrid
Microsoft Application Virtualization, SoftGrid, Softricity
https://blogs.technet.microsoft.com/askcore/2015/03/12/thefourstagesofntfsfilegrowthpart2/

5/10