interop callback stack overflow

[nickdu]

My application is making use of ETW (Event Trace for Windows) API for
tracing. We've noticed that sometimes our application crashes and it appears
this intermittantly happens when our managed application's ETW callback is
executed. I had the debugger (cdb) attached to IIS the other day and it did
capture a crash dump when this occurred. The exception was a stack overflow.
Below is the faulting stack:

0:000> kb 50
ChildEBP RetAddr Args to Child
050cf65c 7c90de7a 7c801e3a ffffffff 800703e9 ntdll!KiFastSystemCallRet
050cf660 7c801e3a ffffffff 800703e9 050cf6d4 ntdll!NtTerminateProcess+0xc
050cf670 7a0960b1 ffffffff 800703e9 c9b72903 kernel32!TerminateProcess+0x20
050cf6d4 7a0961e1 050cf67c 0023f398 050cf814
mscorwks!EEPolicy::HandleFatalStackOverflow+0xf2
050cf6f8 7a00b617 00000001 00000000 00000000
mscorwks!EEPolicy::HandleStackOverflow+0x173
050cf714 7c9032a8 050cf800 050cfd00 050cf814
mscorwks!COMPlusFrameHandler+0x10b
050cf738 7c90327a 050cf800 050cfd00 050cf814 ntdll!ExecuteHandler2+0x26
050cf7e8 7c90e48a 00000000 050cf814 050cf800 ntdll!ExecuteHandler+0x24
050cf7e8 7c812afb 00000000 050cf814 050cf800
ntdll!KiUserExceptionDispatcher+0xe
050cfb34 7a0363d7 e053534f 00000000 00000000 kernel32!RaiseException+0x53
050cfb50 7a0c86ae 0023f398 0023f398 050cfba0 mscorwks!ReportStackOverflow+0x61
050cfb60 79e7c5d2 00000050 00100000 00080000 mscorwks!Alloc+0x3b
050cfba0 79e7c697 79304f88 c9b72393 050cfc74 mscorwks!FastAllocateObject+0x38
050cfc44 04a709d9 01875fa4 00000000 00000000 mscorwks!JIT_NewFast+0x9e
WARNING: Frame IP not in any known module. Following frames may be wrong.
050cfd0c 79e71e5f 00000004 00000000 050cfef4 0x4a709d9
050cfd24 7a0c9692 050cfed0 00000010 009531b8
mscorwks!UM2MThunk_WrapperHelper+0x1f
050cfd84 7a0c97c9 050cfe6c 0023f398 0023f398
mscorwks!UM2MThunk_WrapperWorker+0x60
050cfd98 79fd9799 050cfe6c 050cfe3c 79f7762b mscorwks!UM2MThunk_Wrapper+0x14
050cfe48 7a0c984d 00000005 7a0c97b5 050cfe6c
mscorwks!Thread::DoADCallBack+0xda
050cfea0 009531a2 ffffffff 0023f398 050cfed0 mscorwks!UM2MDoADCallBack+0x7c
050cfec8 77e05f9e 00000004 00000000 050cfef4 0x9531a2
050cff1c 77e2fa38 00ba4028 00000000 00000000
advapi32!WmipInternalNotification+0x17d
050cff60 77e2fbee 00ba4028 00000040 050cffe8
advapi32!WmipDeliverAllEvents+0x33
050cff70 77e061b1 00ba4028 00000040 050cffe4
advapi32!WmipProcessEventBuffer+0x82
050cffe8 77e2f9ee 00000000 00000004 00000000 advapi32!WmipEventPump+0x28c
050cfff4 00000000 1f1f1f1f 00000000 00000000
advapi32!WmipEventPumpFromKernel+0x42

Is this something that has been identified as a potential problem, e.g.
callbacks into managed code from unmanaged threads? I'm guessing the
unmanaged thread's stack is not that large and when the managed code executes
it ends up doing a bunch of stuff which blows out the stack, again just a
guess of mine. If this is the case what might be a solution? Should I just
set some flag in the callback and somehow get the code to run in a managed
thread? If so, what's the best way to do this? Async delegate? A dedicated
worker thread which I manage? etc.

[Jie Wang [MSFT]]

Hi Nick,

Are you using the EventRegister Function to make the ETW callback to your
code?

If that is the case, could you also show me the lines that you used to
register the callback?

And is it happening everytime the callback happens or with a pattern or
just randomly?

Thanks,
Jie Wang

Microsoft Online Community Support

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
***@microsoft.com.

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.

Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business days is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

[nickdu]

We're calling RegisterTraceGuids() which takes a callback function. Here is
the code snippet which calls that method (or the RegisterTraceGuids() method
we expose from our interop assembly):

etwCallbackReference = new Win32.Etw.WMIDPREQUEST(TraceControlCallback);
Win32.Etw.TRACE_GUID_REGISTRATION[] traceguids = new
Win32.Etw.TRACE_GUID_REGISTRATION[1];
traceguids[0] = new Win32.Etw.TRACE_GUID_REGISTRATION();
traceguids[0].Guid = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(Guid)));
try
{
Marshal.StructureToPtr(sourceGuid, traceguids[0].Guid, false);
this._registration = Win32.Etw.RegisterTraceGuids(etwCallbackReference,
IntPtr.Zero,
ref sourceGuid, ref traceguids, null, null);
}
finally
{
Marshal.FreeHGlobal(traceguids[0].Guid);
}

The callback, etwCallbackReference, is an instance member and the instance
of the class to which this belongs is referenced throughout the entire
lifetime of the application.

I don't know yet whether the crash occurs all the time or intermittently. I
believe it's intermittent.
--
Thanks,
Nick

***@community.nospam
remove "nospam" change community. to msn.com

Post by Jie Wang [MSFT]
Hi Nick,
Are you using the EventRegister Function to make the ETW callback to your
code?
If that is the case, could you also show me the lines that you used to
register the callback?
And is it happening everytime the callback happens or with a pattern or
just randomly?
Thanks,
Jie Wang
Microsoft Online Community Support
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.
Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business days is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

[Jie Wang [MSFT]]

Hi Nick,

After doing some research, I found your original guess could be right -
this could be caused by the default stack size in IIS.

Here is the KB article describing the fact:
http://support.microsoft.com/default.aspx/kb/932909

Normally the default 1MB stack size is sufficient for a .NET application;
however, starting from Windows Sever 2003, the default stack size of IIS is
changed to 256KB.

You may want to change it back to 1MB default. The workaround is described
in the post "Stack sizes in IIS - affects ASP.NET" by Tom:
http://blogs.msdn.com/tom/archive/2008/03/31/stack-sizes-in-iis-affects-asp-
net.aspx

Hope this helps. Any updates please also kindly let me know.

Thanks,
Jie Wang

Microsoft Online Community Support

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
***@microsoft.com.

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.

Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business days is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

[nickdu]

Thanks. But if the thread is an unmanaged thread created by the ETW api's,
then doesn't the unmanaged code that created the thread set the stack size
for the thread? Just wondering why this is an IIS issue if IIS is not
managing the thread.
--
Thanks,
Nick

***@community.nospam
remove "nospam" change community. to msn.com

Post by Jie Wang [MSFT]
Hi Nick,
After doing some research, I found your original guess could be right -
this could be caused by the default stack size in IIS.
http://support.microsoft.com/default.aspx/kb/932909
Normally the default 1MB stack size is sufficient for a .NET application;
however, starting from Windows Sever 2003, the default stack size of IIS is
changed to 256KB.
You may want to change it back to 1MB default. The workaround is described
http://blogs.msdn.com/tom/archive/2008/03/31/stack-sizes-in-iis-affects-asp-
net.aspx
Hope this helps. Any updates please also kindly let me know.
Thanks,
Jie Wang
Microsoft Online Community Support
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.
Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business days is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

[Ben Voigt [C++ MVP]]

There's a process-wide default stack size which is used if zero is passed to
CreateThread (as is usually done).

Post by nickdu
Thanks. But if the thread is an unmanaged thread created by the ETW api's,
then doesn't the unmanaged code that created the thread set the stack size
for the thread? Just wondering why this is an IIS issue if IIS is not
managing the thread.
--
Thanks,
Nick
remove "nospam" change community. to msn.com

Post by Jie Wang [MSFT]
Hi Nick,
After doing some research, I found your original guess could be right -
this could be caused by the default stack size in IIS.
http://support.microsoft.com/default.aspx/kb/932909
Normally the default 1MB stack size is sufficient for a .NET application;
however, starting from Windows Sever 2003, the default stack size of IIS is
changed to 256KB.
You may want to change it back to 1MB default. The workaround is described
http://blogs.msdn.com/tom/archive/2008/03/31/stack-sizes-in-iis-affects-asp-
net.aspx
Hope this helps. Any updates please also kindly let me know.
Thanks,
Jie Wang
Microsoft Online Community Support
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.
Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business days is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

[Jie Wang [MSFT]]

Post by Ben Voigt [C++ MVP]
There's a process-wide default stack size which is used if zero is
passed to CreateThread (as is usually done).

Ben is right. And the default size is stored in the PE header. That's why
we need to modify the w3wp.exe to enlarge the default size.

Also there is a "disableStackOverflowProbing" setting that might help (I
haven't tested it in ASP.NET though):

"If you specify a very small stack size, you might need to disable
stack-overflow probing. When the stack is severely constrained, the probing
can itself cause a stack overflow. To disable stack overflow probing, add
the following to your application configuration file."

http://msdn.microsoft.com/en-us/library/5cykbwz4.aspx

Regards,
Jie Wang

Microsoft Online Community Support

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
***@microsoft.com.

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.

Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business days is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

[nickdu]

Thanks, that clears up my confusion about the stack size setting.

I'm guessing we're not going to want to change the default setting of IIS.
I assume we can work around this issue by having the code we want to run
execute on a managed thread? Would an alternate solution be to wrap the
existing callback code up into another routine and call that asynchronously
via an async delegate?
--
Thanks,
Nick

***@community.nospam
remove "nospam" change community. to msn.com

Post by Jie Wang [MSFT]

Post by Ben Voigt [C++ MVP]
There's a process-wide default stack size which is used if zero is
passed to CreateThread (as is usually done).

Ben is right. And the default size is stored in the PE header. That's why
we need to modify the w3wp.exe to enlarge the default size.
Also there is a "disableStackOverflowProbing" setting that might help (I
"If you specify a very small stack size, you might need to disable
stack-overflow probing. When the stack is severely constrained, the probing
can itself cause a stack overflow. To disable stack overflow probing, add
the following to your application configuration file."
http://msdn.microsoft.com/en-us/library/5cykbwz4.aspx
Regards,
Jie Wang
Microsoft Online Community Support
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.
Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business days is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

[Jie Wang [MSFT]]

This post might be inappropriate. Click to display it.

[nickdu]

So you're saying the managed ASP.NET threads have the same stack size as
defaulted by IIS (e.g. 256K with Windows 2003)?
--
Thanks,
Nick

***@community.nospam
remove "nospam" change community. to msn.com

Post by Jie Wang [MSFT]
Hi Nick,
As long as the "managed" thread's underlying implementation uses the OS
thread, it doesn't make much differences here.
One thing I can think of is, if it is your code controls when the callback
happens - in this case, like calling EnableTrace function - it happens on a
thread that you created with a larger thread stack.
I'm not sure if my assumption about the control of the callback is correct
or not. Please let me know.
Thanks,
Jie Wang
Microsoft Online Community Support
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.
Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business days is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

[Jie Wang [MSFT]]

Yes. Your ASP.NET application is hosted in the w3wp.exe (IIS Worker
Process), so all threads created in that process without explicitly
specifying a new stack size will use the default size (256K) defined in the
w3wp.exe header.

Regards,
Jie Wang

Microsoft Online Community Support

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
***@microsoft.com.

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.

Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business days is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

[nickdu]

I'm curious, how would one use up 256K of stack space? That seems like an
awful lot of space. I can see how an unmanaged app might easily use up a
large amount of stack space, though probably an unlikely thing to do, by
declaring large buffers on the stack, but I can't see how my managed app can
consume a bunch of stack space other than creating an infinite recursive loop.

I'm trying to make sense of this hypothesis of mine and possible easily
reproduce it.
--
Thanks,
Nick

***@community.nospam
remove "nospam" change community. to msn.com

Post by Jie Wang [MSFT]
Yes. Your ASP.NET application is hosted in the w3wp.exe (IIS Worker
Process), so all threads created in that process without explicitly
specifying a new stack size will use the default size (256K) defined in the
w3wp.exe header.
Regards,
Jie Wang
Microsoft Online Community Support
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.
Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business days is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

[Jie Wang [MSFT]]

Hi Nick,

I don't want to make any assumptions here about why you got a stack
overflow. But I believe by analyzing the dump would tell something.
Although doing a direct dump analysis is out of the newgroup support scope,
you can still do it yourself and in case you got any questions, you can ask
them in the microsoft.public.windbg newgroup, the experts there is a great
resource.

Best regards,
Jie Wang

Microsoft Online Community Support

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
***@microsoft.com.

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.

Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business days is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.