Sparse file support seems to have problems

[nickdu]

I don't believe there is any built in support for sparse files in .NET. So I
created some pinvoke calls to get sparse file support within my managed
application. However, there seems to be some other resource limit I'm
running up against. When I run my sample application, which when the same
unmanaged application runs it creates a 16TB file (which only consumes a gig
or so), it fails early with an "insufficient resources" error when seeking.
Below is my sample application:

using System;
using System.IO;
using System.ComponentModel;
using System.Runtime.InteropServices;

namespace Win32
{

public class File
{
public const uint FILE_DEVICE_BEEP = 0x00000001;
public const uint FILE_DEVICE_CD_ROM = 0x00000002;
public const uint FILE_DEVICE_CD_ROM_FILE_SYSTEM = 0x00000003;
public const uint FILE_DEVICE_CONTROLLER = 0x00000004;
public const uint FILE_DEVICE_DATALINK = 0x00000005;
public const uint FILE_DEVICE_DFS = 0x00000006;
public const uint FILE_DEVICE_DISK = 0x00000007;
public const uint FILE_DEVICE_DISK_FILE_SYSTEM = 0x00000008;
public const uint FILE_DEVICE_FILE_SYSTEM = 0x00000009;
public const uint FILE_DEVICE_INPORT_PORT = 0x0000000a;
public const uint FILE_DEVICE_KEYBOARD = 0x0000000b;
public const uint FILE_DEVICE_MAILSLOT = 0x0000000c;
public const uint FILE_DEVICE_MIDI_IN = 0x0000000d;
public const uint FILE_DEVICE_MIDI_OUT = 0x0000000e;
public const uint FILE_DEVICE_MOUSE = 0x0000000f;
public const uint FILE_DEVICE_MULTI_UNC_PROVIDER = 0x00000010;
public const uint FILE_DEVICE_NAMED_PIPE = 0x00000011;
public const uint FILE_DEVICE_NETWORK = 0x00000012;
public const uint FILE_DEVICE_NETWORK_BROWSER = 0x00000013;
public const uint FILE_DEVICE_NETWORK_FILE_SYSTEM = 0x00000014;
public const uint FILE_DEVICE_NULL = 0x00000015;
public const uint FILE_DEVICE_PARALLEL_PORT = 0x00000016;
public const uint FILE_DEVICE_PHYSICAL_NETCARD = 0x00000017;
public const uint FILE_DEVICE_PRINTER = 0x00000018;
public const uint FILE_DEVICE_SCANNER = 0x00000019;
public const uint FILE_DEVICE_SERIAL_MOUSE_PORT = 0x0000001a;
public const uint FILE_DEVICE_SERIAL_PORT = 0x0000001b;
public const uint FILE_DEVICE_SCREEN = 0x0000001c;
public const uint FILE_DEVICE_SOUND = 0x0000001d;
public const uint FILE_DEVICE_STREAMS = 0x0000001e;
public const uint FILE_DEVICE_TAPE = 0x0000001f;
public const uint FILE_DEVICE_TAPE_FILE_SYSTEM = 0x00000020;
public const uint FILE_DEVICE_TRANSPORT = 0x00000021;
public const uint FILE_DEVICE_UNKNOWN = 0x00000022;
public const uint FILE_DEVICE_VIDEO = 0x00000023;
public const uint FILE_DEVICE_VIRTUAL_DISK = 0x00000024;
public const uint FILE_DEVICE_WAVE_IN = 0x00000025;
public const uint FILE_DEVICE_WAVE_OUT = 0x00000026;
public const uint FILE_DEVICE_8042_PORT = 0x00000027;
public const uint FILE_DEVICE_NETWORK_REDIRECTOR = 0x00000028;
public const uint FILE_DEVICE_BATTERY = 0x00000029;
public const uint FILE_DEVICE_BUS_EXTENDER = 0x0000002a;
public const uint FILE_DEVICE_MODEM = 0x0000002b;
public const uint FILE_DEVICE_VDM = 0x0000002c;
public const uint FILE_DEVICE_MASS_STORAGE = 0x0000002d;
public const uint FILE_DEVICE_SMB = 0x0000002e;
public const uint FILE_DEVICE_KS = 0x0000002f;
public const uint FILE_DEVICE_CHANGER = 0x00000030;
public const uint FILE_DEVICE_SMARTCARD = 0x00000031;
public const uint FILE_DEVICE_ACPI = 0x00000032;
public const uint FILE_DEVICE_DVD = 0x00000033;
public const uint FILE_DEVICE_FULLSCREEN_VIDEO = 0x00000034;
public const uint FILE_DEVICE_DFS_FILE_SYSTEM = 0x00000035;
public const uint FILE_DEVICE_DFS_VOLUME = 0x00000036;
public const uint FILE_DEVICE_SERENUM = 0x00000037;
public const uint FILE_DEVICE_TERMSRV = 0x00000038;
public const uint FILE_DEVICE_KSEC = 0x00000039;
public const uint FILE_DEVICE_FIPS = 0x0000003A;
public const uint FILE_DEVICE_INFINIBAND = 0x0000003B;

public const uint FILE_ANY_ACCESS = 0;
public const uint FILE_SPECIAL_ACCESS = (FILE_ANY_ACCESS);
public const uint FILE_READ_ACCESS = ( 0x0001 ); // file & pipe
public const uint FILE_WRITE_ACCESS = ( 0x0002 ); // file & pipe

public const uint METHOD_BUFFERED = 0;
public const uint METHOD_IN_DIRECT = 1;
public const uint METHOD_OUT_DIRECT = 2;
public const uint METHOD_NEITHER = 3;

public const uint FSCTL_SET_SPARSE = ((FILE_DEVICE_FILE_SYSTEM << 16) |
(FILE_SPECIAL_ACCESS << 14) | (49 << 2) | METHOD_BUFFERED);

[DllImport("kernel32.dll", SetLastError = true)]
public static extern int DeviceIoControl(IntPtr device, uint code, IntPtr
inBuffer,
uint inBufferSize, IntPtr outBuffer, uint outBufferSize,
out uint bytesReturned, IntPtr overlapped);
}

}

public class Application
{
public static void Main(string[] args)
{
if (args.Length != 1)
Console.WriteLine("Error, expecting file name.");
else
{
FileStream file = File.Create(args[0]);
using(file)
{
uint bytesReturned;

if (Win32.File.DeviceIoControl(file.SafeFileHandle.DangerousGetHandle(),
Win32.File.FSCTL_SET_SPARSE, IntPtr.Zero, 0, IntPtr.Zero, 0,
out bytesReturned, IntPtr.Zero) == 0)
{
int result = Marshal.GetLastWin32Error();
throw(new Win32Exception(result, string.Format(
"Failed setting file {0} to sparse.", args[0])));
}

byte[] bytes = new byte[1024];
for (long i = 0; i < bytes.Length; ++i)
{
bytes[i] = (byte) 'A';
}

for (long i = 0; i < 1024 * 1024; ++i)
{
file.Seek(i * 1024 * 1024 * 1024, SeekOrigin.Begin);
file.Write(bytes, 0, bytes.Length);
}
}
}
}
}

[Jialiang Ge [MSFT]]

Hello Nick

In my test, when 'long i' grows to about 16384 or 16385, I see the error.

16384 * 1024 * 1024 * 1024 bytes = 16 TB.

The exception was thrown because you hit the sparse file size limit. See
the section "Can They Really Be That Large?" in
http://www.flexhex.com/docs/articles/sparse-files.phtml.

Please let me know whether it answers the question.

Regards,
Jialiang Ge (***@online.microsoft.com, remove 'online.')
Microsoft Online Community Support

=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
***@microsoft.com.

This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================

[nickdu]

Yes, 16384 is how far I get with my C++ application. However, the similar
.NET application (the one I included in the post) only gets to about 112.
Why might your .NET application be getting farther than mine?
--
Thanks,
Nick

***@community.nospam
remove "nospam" change community. to msn.com

Post by Jialiang Ge [MSFT]
Hello Nick
In my test, when 'long i' grows to about 16384 or 16385, I see the error.
16384 * 1024 * 1024 * 1024 bytes = 16 TB.
The exception was thrown because you hit the sparse file size limit. See
the section "Can They Really Be That Large?" in
http://www.flexhex.com/docs/articles/sparse-files.phtml.
Please let me know whether it answers the question.
Regards,
Microsoft Online Community Support
=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================

[Jialiang Ge [MSFT]]

Hi Nick

I tested the exactly same code on both my x86 and x64 machines, and it got
to 16384.

1. Does it consistently get to 112 in multiple tests on your side? Or does
it get to a random number around 112?

2. Can you be double sure that your drive is formatted as NTFS, and does
have enough physical space for the file considering Size on Disk?

3. http://www.flexhex.com/docs/articles/sparse-files.phtml

<quote>
You can create a largest possible 16 terabyte sparse file if and only if it
consists of a single sparse zero area, no data at all. Writing even a
single data byte drops the size limit far, far below. The exact limit
depends on many factors: availability of system resources, the layout of
the real data areas, even the order of the I/O requests; *in some cases the
limit may be as low as several hundreds gigabytes.* If you don't mind
experimenting, you can use FlexHEX to find your system limit.
</quote>

I will ask the product group for the condition in which the limit may be as
low as several hundred gigabytes.

4. You may consider debugging into the line file.Seek(i * 1024 * 1024 *
1024, SeekOrigin.Begin); with BCL source code:
http://referencesource.microsoft.com/downloadsetup.aspx

I debugged into it, and when it hits the limit (16385), I got the HRESULT
0x00000057 from WriteFileNative. You may check whether it's the same error
code on your side.

Regards,
Jialiang Ge (***@online.microsoft.com, remove 'online.')
Microsoft Online Community Support

=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
***@microsoft.com.

This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================

[nickdu]

1. It was somewhat random around 112. However, when I just ran it again it
got to 430.

2. I believe my drive is fine as the C program can successfully get to 16384
running on the same machine.

The managed application is .NET 2.0. What version of .NET are you using?
--
Thanks,
Nick

***@community.nospam
remove "nospam" change community. to msn.com

Post by Jialiang Ge [MSFT]
Hi Nick
I tested the exactly same code on both my x86 and x64 machines, and it got
to 16384.
1. Does it consistently get to 112 in multiple tests on your side? Or does
it get to a random number around 112?
2. Can you be double sure that your drive is formatted as NTFS, and does
have enough physical space for the file considering Size on Disk?
3. http://www.flexhex.com/docs/articles/sparse-files.phtml
<quote>
You can create a largest possible 16 terabyte sparse file if and only if it
consists of a single sparse zero area, no data at all. Writing even a
single data byte drops the size limit far, far below. The exact limit
depends on many factors: availability of system resources, the layout of
the real data areas, even the order of the I/O requests; *in some cases the
limit may be as low as several hundreds gigabytes.* If you don't mind
experimenting, you can use FlexHEX to find your system limit.
</quote>
I will ask the product group for the condition in which the limit may be as
low as several hundred gigabytes.
4. You may consider debugging into the line file.Seek(i * 1024 * 1024 *
http://referencesource.microsoft.com/downloadsetup.aspx
I debugged into it, and when it hits the limit (16385), I got the HRESULT
0x00000057 from WriteFileNative. You may check whether it's the same error
code on your side.
Regards,
Microsoft Online Community Support
=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================

[Jialiang Ge [MSFT]]

Hi Nick

I was using .NET 2.0 too.

According to PG's comments, there is a limit to the number of extents a
file can have. The limit you see at 112gb is in the range.

A heavily fragmented file in an NTFS volume may not grow beyond a certain
size
http://support.microsoft.com/default.aspx?scid=kb;en-us;967351

But why can the corresponding C/C++ code run well?

Here is an explanation. Please help to check whether it is the case:

You haven't provided the C code that has the same code logic, it's possible
that the C/C++ code only appears to be succeeding. If the code uses a
'long' data type for i, then computing the seek offset as i *1024 * 1024 *
1024 will just result in an overflowed long, i.e. some offset < 2GB. To
seek to large offsets properly requires using a LARGE_INTEGER and
SetFilePointerEx (or messing with two LONGs and SetFilePointer).

In C#, the 'long' data type is 64 bits
(http://msdn.microsoft.com/en-us/library/yyaad03b(VS.71).aspx), so the C#
program is the one that is accurately demonstrating NTFS behavior.

Regards,
Jialiang Ge (***@online.microsoft.com, remove 'online.')
Microsoft Online Community Support

=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
***@microsoft.com.

This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================

[nickdu]

I don't think my C program is appearing to succeed because of overflow. Here
are a couple lines of the code:

for (i = 0; i < 1024 * 1024; ++i)
{
LARGE_INTEGER li;

li.QuadPart = (unsigned __int64) i * (unsigned __int64) 1024 * (unsigned
__int64) 1024 * (unsigned __int64) 1024;


plus, after the program finishes the OS shows the file as 16 TB.
--
Thanks,
Nick

***@community.nospam
remove "nospam" change community. to msn.com

Post by Jialiang Ge [MSFT]
Hi Nick
I was using .NET 2.0 too.
According to PG's comments, there is a limit to the number of extents a
file can have. The limit you see at 112gb is in the range.
A heavily fragmented file in an NTFS volume may not grow beyond a certain
size
http://support.microsoft.com/default.aspx?scid=kb;en-us;967351
But why can the corresponding C/C++ code run well?
You haven't provided the C code that has the same code logic, it's possible
that the C/C++ code only appears to be succeeding. If the code uses a
'long' data type for i, then computing the seek offset as i *1024 * 1024 *
1024 will just result in an overflowed long, i.e. some offset < 2GB. To
seek to large offsets properly requires using a LARGE_INTEGER and
SetFilePointerEx (or messing with two LONGs and SetFilePointer).
In C#, the 'long' data type is 64 bits
(http://msdn.microsoft.com/en-us/library/yyaad03b(VS.71).aspx), so the C#
program is the one that is accurately demonstrating NTFS behavior.
Regards,
Microsoft Online Community Support
=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================

[Jialiang Ge [MSFT]]

Hello Nick

A product group member tells me that you are more likely running into a
system level insufficient resource. Would you please try increasing your
page file max size?

Regards,
Jialiang Ge (***@online.microsoft.com, remove 'online.')
Microsoft Online Community Support

=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
***@microsoft.com.

This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================

[nickdu]

I will certainly try that, but how come it works fine in the C program? The
difference I can see between the two is that the C# program might be
buffering the file while the C one is not as I'm using the
FILE_FLAG_NO_BUFFERING flag. Could this be the issue? Is there a way to
turn buffering off in the managed application? Otherwise I guess I could
just call CreateFile() via interop from C#.
--
Thanks,
Nick

***@community.nospam
remove "nospam" change community. to msn.com

Post by Jialiang Ge [MSFT]
Hello Nick
A product group member tells me that you are more likely running into a
system level insufficient resource. Would you please try increasing your
page file max size?
Regards,
Microsoft Online Community Support
=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================

[Jialiang Ge [MSFT]]

Hello Nick

I can't be sure that FILE_FLAG_NO_BUFFERING is related to the problem, but
it deserves your test. I'm going to email the event tracing team and listen
to their comments on the FILE_FLAG_NO_BUFFERING flag.

To pass FILE_FLAG_NO_BUFFERING flag, you have to P/Invoke CreateFile
manually and give the handle (IntPtr) to FileStream's constructor.

Regards,
Jialiang Ge (***@online.microsoft.com, remove 'online.')
Microsoft Online Community Support

=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
***@microsoft.com.

This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================

[Jialiang Ge [MSFT]]

In addition, would you please tell me on what arch & OS the code is running
into the problem?

Regards,
Jialiang Ge (***@online.microsoft.com, remove 'online.')
Microsoft Online Community Support

=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
***@microsoft.com.

This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================

[Jialiang Ge [MSFT]]

Hello Nick,

Do you have any progress about this issue? Is FILE_FLAG_NO_BUFFERING the
culprit? Please inform us about the test result and we will do further
investigations of the cause. If you need further assistance, feel free to
let me know. I will be more than happy to be of assistance.

Have a great day!

Regards,
Jialiang Ge (***@online.microsoft.com, remove 'online.')
Microsoft Online Community Support

=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
***@microsoft.com.

This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================