MFT Zone using NTFS in 2k over 650MB!

[videobruce]

I have 2 boxes both with 2 bootable HDD's all running 2k that I just converted over from FAT32 to NTFS.
After doing so I did a disc to disc copy using Drive Image from a bootable floppy (actually a partition to partition copy) and found the MFT Zone grew from less than 70 MB to over 600MB! To be acurate, 650MB and 1GB!

Any one else have this experiance or has copied a converted drive over to another?

There are no errors reported by chkdsk or chkntfs and all seems to work. BTW, the defrag program I use is from vCom (formerly Ontrack) called System Suite V4 Jet Defrag. It has the ability I just found out to display the MFT and the MFT Zone prior to doing a defrag that Norton doesn't/can't! It shows a one huge block after the first block of data (.exe's)

I looked at the M$'s KB and it doesn't tell you a whole lot and doesn't answer the question. I'm new to NTFS and didn't heard of this even existing untill now. I do have the latest updates to the program, though this new outfit hasn't done a thing to it since it was spun off 9 months ago. I doubt it is a JetDefrag issue, but it might be a DI issue.

 

[Nothinman]

NTFS stores small files (less than 2K I think) inside the MFT, so if there's a lot of small files that may account for it. I've personally never looked at how big the MFT was on any boxes so I can't even say if that's normal.

 

[videobruce]

Than why isn't it in the orginal active drive?

 

[stevewm]

If the volumes have alot of free space then the MFT Zone will grow rather large. As the space becomes needed however Windows will reduce the MFT Zone size.

On my system volume, which is 37.2GB the MFT reserved space (MFT Zone) is 450MB. The MFT itself is only 42.6MB in size. This volume is about 40% full.

On my MUCH bigger storage volume, which is 95% full, my MFT is 71MB in size and the MFT Zone is only 90MB. When this drive was only 10% full the MFT Zone was near 1GB, while the MFT itself wasn't over 20MB.

Windows handles the MFT Zone size, and it will dynamically adjust it whenever it needs to be. It just decided in this particular instance to increase the MFT Zone size. This is a perfectly normal operation of the NTFS filesystem. Don't worry about it. As I said above you are not out 700MB of hardrive space, the MFT Zone can still be used by files when needed.

 

[videobruce]

Stupid question: volume=partition or drive?

The next day I went to run JetDefrag (mostly to look at the size of this MFT Zone) and it reported only 36 MB. The MFT is 19 MB. Nothing was done to the drive as far as I remember. How could it go from 1 GB down to 36 MB overnight?

I do understand that it is dynamically resized depending upon file number and sizes, but that radical of a change?

 

[bsobel]

Originally posted by: videobruce
Stupid question: volume=partition or drive?
The next day I went to run JetDefrag (mostly to look at the size of this MFT Zone) and it reported only 36 MB. The MFT is 19 MB. Nothing was done to the drive as far as I remember. How could it go from 1 GB down to 36 MB overnight?
I do understand that it is dynamically resized depending upon file number and sizes, but that radical of a change?

The MFT zone is the area the MFT can use, it doesn't mean that much space is in use. From your description I suspect you had 512byte clusters (since you converted from fat32 to ntfs), and now have 4k clusters (after using drive image). The net result is that alot more files probably fit into their MFT entry that used to.

Bill

 

[Lord Evermore]

The default setting for NTFS is 1/8th of the total volume space being reserved for the MFT. The reserved zone is where the MFT CAN go if it needs to. The space gets reserved when you create the partition, and is just saved so it gets used only if the entire rest of the free space is used up. Of course, on a 20GB partition, that means 2.5GB is reserved.

I would guess that when you converted to NTFS, no space was properly reserved. For some reason when the partition was copied, the reserved space was created to comply with proper NTFS rules. I think what actually happened is when you first checked the MFT size after the conversion, Windows hadn't yet reset the MFT reservation size. After you copied the drive, and booted, Windows then reserved the normal amount.

Don't worry about it. The space isn't "used" and is completely available for your files, however it won't get used unless you fill up the rest of the drive. Unfortunately that usually means 2.5GB or more of the prime (quick access) space at the beginning of the drive is reserved. Unfortunately 1/8th is the lowest setting possible, Microsoft doesn't seem to feel a need to modify the standard in this regard to account for the hundredfold increase in drive sizes in the past several years.

I would not expect Drive Image to convert from 512k clusters to 4k clusters, as it is only copying the existing partition format. But I also can't explain the MFT reserved zone growing suddenly. 512k clusters is definitely the worst configuration possible, and if you can get a utility to convert them to 4k that's a good thing to do. You can see what the current size is simply by running chkdsk.

Also, the "size" of the MFT entry doesn't change based on the cluster size. NTFS simply allocates a specific amount of space in the MFT for each file or directory entry. However a file that is only 1.5KB or so can fit entirely in the NTFS entry space because there is a specific "data/index" portion of the entry. A 1KB file will always fit into the entry space because the size of an MFT entry isn't determined by the cluster size.

dir /a $mft can show you the MFT size in Win2k without a special defragger. WinXP's defrag program shows it in the report when you Analyze the drive.

http://support.microsoft.com/default.aspx?scid=kb;en-us;Q174619

 

[bsobel]

oops, double post, see comments below

 

[bsobel]

Also, the "size" of the MFT entry doesn't change based on the cluster size. NTFS simply allocates a specific amount of space in the MFT for each file or directory entry. However a file that is only 1.5KB or so can fit entirely in the NTFS entry space because there is a specific "data/index" portion of the entry. A 1KB file will always fit into the entry space because the size of an MFT entry isn't determined by the cluster size.

Yes it is determined by the cluster size (not a direct mapping, however), all MFT entries will be the same size, the can be 1-4k depending on what cluster size is used during the format. The default size is 1k. The record size used will never be less than 1k or greater than 4k. A quick qoogling of the subject gives this decent overview.

Unfortunately 1/8th is the lowest setting possible, Microsoft doesn't seem to feel a need to modify the standard in this regard to account for the hundredfold increase in drive sizes in the past several years.

And they shouldn't. Remember, the first extend of every file is stored in the MFT anyhow. MS does allow you to pre-reserve a larger area via a registry setting. Also remember that MS will automatically decrease the MFT zone when the 'non mft zone' is filled (this repeats until the drive it actually full).

Bill

 

[Lord Evermore]

Yes, but they shouldn't be reserving such a large section of the drive by default. My MFT is only under 40MB in size (35MB used, the actual size is 65MB after having many thousands of files for a short time; I may never fill the MFT again). I bet 99% of people are about the same or at least under 100MB. What's the point of reserving 10GB of an 80GB volume, with the reserved space located in the prime speed area of the drive, when it will never need to be used? The only way that much space would be taken up is if you had 10GB of files all under 1K. That's 10 million files (plus or minus the space for any indexes for files that don't fit).

They should adjust it so that the default is much lower to reflect both the massively increased drive sizes today, the usage of the average home and workstation system, and the fact that file sizes in general have gone far beyond 1K on average.

On my system, 5919 files and directories are under 2K. The other 27000 are larger. So at most, the files that fit in the MFT entirely are taking up a paltry 11MB, while just the indexes in the MFT for the other files takes up 24MB.

I assume that when the reserved zone needs to be used, the zone is shrunk from the end first, meaning that the fastest of the reserved area truly is never used until the drive is completely full. It doesn't seem that the reserved space is actually reserved at the beginning of the partition though. I formatted my drive on an XP machine before doing a fresh install on it, and the reserved space is located quite a ways in according to DiskKeeper. But that's still forcing actual user data toward the inner, slow part of the drive, simply leaving a large blank gap in the middle of the drive. Certainly the MFT needs to be on a quick portion of the disk, but it doesn't need to reserve so much space anymore. On a 1GB drive, 1/8th of the space was only 120MB, a perfectly sensible reservation given the smaller average file size (although most people didn't store nearly as many files back then; certainly reserving anything more than that by default would be stupid).

I was wrong about the entry size changing. But it actually changes my argument somewhat. I use an 8KB cluster size (never really noticed it being any better or worse than 4K for my uses). There are 11850 files on my system under 8KB. That means about one-third of my files fit into the MFT entirely (presumably, assuming the actual available space is something close to 8KB). Obviously not that many actually do, Since my MFT isn't nearly the 88MB that would imply, but it does indicate that even with a very large number of files able to fit in the MFT, it won't nearly grow large enough to use the reserved space. Even the 2.5GB reserved on mine is probably 10 times more than I'd ever need.

An FTP server with many users, or a web server, could probably use a much larger amount of the reserved space, but even then, those will have even larger volumes, meaning dozens of GBs of reserved space which could never be filled.

 

[bsobel]

I assume that when the reserved zone needs to be used, the zone is shrunk from the end first, meaning that the fastest of the reserved area truly is never used until the drive is completely full.

I haven't looked at the code in a long time, but as I recall yes, it does shrink from the end.

It doesn't seem that the reserved space is actually reserved at the beginning of the partition though. I formatted my drive on an XP machine before doing a fresh install on it, and the reserved space is located quite a ways in according to DiskKeeper. But that's still forcing actual user data toward the inner, slow part of the drive, simply leaving a large blank gap in the middle of the drive. Certainly the MFT needs to be on a quick portion of the disk, but it doesn't need to reserve so much space anymore. On a 1GB drive, 1/8th of the space was only 120MB, a perfectly sensible reservation given the smaller average file size (although most people didn't store nearly as many files back then; certainly reserving anything more than that by default would be stupid).

The reverse argument is that the initial 'finding' of the directory entry and it's associated information faster is a bigger performance enhancer than then reading the associated extents from the slower part of the drive. Your argument about the size adjusting down with larger drives does (at the surface) seem to make sense, just as MS slowly lowered the page file defaults when system suddenly had 512/1g of memory instead of 8meg That said, I still doubt the performance implications of what your suggesting are as big as you think. But without running some tests I'm just calling that my gut feeling.

I was wrong about the entry size changing. But it actually changes my argument somewhat. I use an 8KB cluster size (never really noticed it being any better or worse than 4K for my uses).

Only issues are that you can't use compression/encryption on the files if your using > 4k clusters (stupid restriction they should eventually address but keep putting off).

There are 11850 files on my system under 8KB.

The max record size is 4k, but even with 8k clusters I think MS is still using 1k or 2k records. Also, even if you had 4k records a full 4k of data wouldn't fit as their is record overhead, run information index information, etc taking some of that space.

An FTP server with many users, or a web server, could probably use a much larger amount of the reserved space, but even then, those will have even larger volumes, meaning dozens of GBs of reserved space which could never be filled.

It doesn't mean it won't be filled at all, it means it won't be filled until the drive is almost full.

Your making me remember things I haven't had to play with in a bit, shame on you

Bill

 

[Lord Evermore]

Doh, forgot about the 4k max record size regarding the 8k clusters. Oh well, still applies mostly. I'm not saying it's a HUGE performance loss, I'm just saying it needs to be addressed because it is rather wasteful. If you go with a single platter 80GB drive, 12% of the area is reserved for the MFT, which is a significant chunk of the outer tracks though not 12% due to different track lengths.

The last bit about the servers, I was referring to the reserved space not possibly ever being filled by the MFT, so there's no reason to reserve that much.

 

[videobruce]

Here is the text from NTFSINFO from Wininternals for both boxes and all 8 drives:

NTFS Information Dump
http://www.ntinternals.com
-----------
Main Computer:
Drive C
Volume Size
Volume size : 8307 MB
Total sectors : 17012771
Total clusters : 17012771
Free clusters : 12404036
Free space : 6056 MB (72% of drive)
-----------
Allocation Size
Bytes per sector : 512
Bytes per cluster : 512
Bytes per MFT record : 1024
Clusters per MFT record: 2
-----------
MFT Information
MFT size : 20 MB (0% of drive)
MFT start cluster : 16646
MFT zone clusters : 42880 - 113568
MFT zone size : 34 MB (0% of drive)
MFT mirror start : 8776406
-----------
Drive D
Volume Size
Volume size : 5726 MB
Total sectors : 11727386
Total clusters : 1465923
Free clusters : 1113050
Free space : 4347 MB (75% of drive)
-----------
Allocation Size
Bytes per sector : 512
Bytes per cluster : 4096
Bytes per MFT record : 1024
Clusters per MFT record: 0
-----------
MFT Information
MFT size : 6 MB (0% of drive)
MFT start cluster : 4
MFT zone clusters : 1632 - 183264
MFT zone size : 709 MB (12% of drive)
MFT mirror start : 732961
-----------
Drive E
Volume Size
Volume size : 69852 MB
Total sectors : 143058760
Total clusters : 17882345
Free clusters : 17347338
Free space : 67763 MB (97% of drive)
-----------
Allocation Size
Bytes per sector : 512
Bytes per cluster : 4096
Bytes per MFT record : 1024
Clusters per MFT record: 0

MFT Information
MFT size : 0 MB (0% of drive)
MFT start cluster : 4
MFT zone clusters : 32 - 524288
MFT zone size : 2047 MB (2% of drive)
MFT mirror start : 524288

Drive F
Volume Size
Volume size : 22881 MB
Total sectors : 46861541
Total clusters : 46861541
Free clusters : 27871423
Free space : 13609 MB (59% of drive)
-----------
Allocation Size
Bytes per sector : 512
Bytes per cluster : 512
Bytes per MFT record : 1024
Clusters per MFT record: 2
-----------
MFT Information
MFT size : 35 MB (0% of drive)
MFT start cluster : 22942
MFT zone clusters : 23573600 - 29357632
MFT zone size : 2824 MB (12% of drive)
MFT mirror start : 23430770

Backup Computer
Drive C
Volume Size
Volume size : 5114 MB
Total sectors : 10474316
Total clusters : 10474316
Free clusters : 6267468
Free space : 3060 MB (59% of drive)
----------------
Allocation Size
Bytes per sector : 512
Bytes per cluster : 512
Bytes per MFT record : 1024
Clusters per MFT record: 2
----------------
MFT Information
MFT size : 17 MB (0% of drive)
MFT start cluster : 29554
MFT zone clusters : 3256800 - 4529760
MFT zone size : 621 MB (12% of drive)
MFT mirror start : 6321545

Drive D
Volume Size
Volume size : 6173 MB
Total sectors : 12643091
Total clusters : 12643091
Free clusters : 8435948
Free space : 4119 MB (66% of drive)
----------------
Allocation Size
Bytes per sector : 512
Bytes per cluster : 512
Bytes per MFT record : 1024
Clusters per MFT record: 2
----------------
MFT Information
MFT size : 17 MB (0% of drive)
MFT start cluster : 29554
MFT zone clusters : 1561248 - 1691616
MFT zone size : 63 MB (1% of drive)
MFT mirror start : 6321545

Drive E
Volume Size
Volume size : 14480 MB
Total sectors : 29655926
Total clusters : 29655926
Free clusters : 9658565
Free space : 4716 MB (32% of drive)
----------------
Allocation Size
Bytes per sector : 512
Bytes per cluster : 512
Bytes per MFT record : 1024
Clusters per MFT record: 2
----------------
MFT Information
MFT size : 52 MB (0% of drive)
MFT start cluster : 14481392
MFT zone clusters : 15084160 - 15446080
MFT zone size : 176 MB (1% of drive)
MFT mirror start : 15872960

Drive F
Volume Size
Volume size : 31996 MB
Total sectors : 65529071
Total clusters : 65529071
Free clusters : 49929245
Free space : 24379 MB (76% of drive)
----------------
Allocation Size
Bytes per sector : 512
Bytes per cluster : 512
Bytes per MFT record : 1024
Clusters per MFT record: 2
----------------
MFT Information
MFT size : 32 MB (0% of drive)
MFT start cluster : 3621270
MFT zone clusters : 21086240 - 25197888
MFT zone size : 2007 MB (6% of drive)
MFT mirror start : 32764535