How should I PInvoke an ANSI multibyte string to a varargs paramet

[opellegr]

I have to use PInvoke to call a C library function that has the following
signature:

foo(const char *, const char *, EnumType, ...)

The function configures a resource in a way that varies by StructType; the
case I'm interested in configures something that expects the varargs to be a
single ANSI multibyte string. I used this PInvoke signature to call the
function:

[DllImport(DllName, CharSet = CharSet.Ansi, EntryPoint = "foo")]
public static extern int Foo(string s1, string s2, EnumType e1, params
string[] s3);

The params string[] seemed odd to me, but previous signatures that called
this function were using it for other types such as bool so I followed the
pattern.

I wrap this with a friendlier .NET method:

public void Foo(string s1, string s2, string s3)
{
int error = Dll.Foo(s1, s2, EnumType.Foo, s3);
// handle errors
}

Recently I changed the signature to include "BestFitMapping = false,
ThrowOnUnmappableChar = true" in the DLLImport attribute to comply with
FxCop's suggestions. This is a red herring as I will describe later.

What I expect out of making this change is limited support for Unicode. For
example, on a machine with an English code page an exception will be thrown
if you pass a string that contains a Japanese character. On a Japanese
machine, I'd expect to be able to pass a Japanese string to the function.
The English test worked as expected, but the Japanese test throws a
System.Runtime.InteropServices.COMException with HRESULT 0x8007007A. This
happens even without the BestFitMapping and ThrowOnUnmappableChar settings.

I've done a little looking around and saw some sites that suggested you
could PInvoke varargs by just specifying normal arguments, so I tried this
signature:

[DllImport(DllName, CharSet = CharSet.Ansi, EntryPoint = "foo")]
public static extern int Foo(string s1, string s2, EnumType st1, string
s3);

This throws an AccessViolationException when I use it on either the English
or Japanese machine.

After some digging around, I found out that you have to use the CDecl If I
specify a calling convention of CDecl and take normal parameters like this:

[DllImport(DllName, CharSet = CharSet.Ansi, EntryPoint = "foo",
CallingConvention = CallingConvention.Cdecl)]
public static extern int Foo(string s1, string s2, EnumType e1, string
s3)

When I use this specification, I get an AccessViolationException.

The only thing I've found that works consistently is this:

[DllImport(DllName, CharSet = CharSet.Ansi, EntryPoint = "foo",
BestFitMapping = false, ThrowOnUnmappableChar = true)]
public static extern int Foo(string s1, string s2, EnumType e1, params
IntPtr[] s3)

To make this work, I use Marshal.StringToHGlobalAnsi() and pass that
pointer. This works for English and Japanese. I'm not comfortable with not
understanding why this is the solution, but it works.

I cannot use UTF-8 or another Unicode encoding because this C library
expects and handles only multibyte ANSI. I have found that specifying
CharSet.Unicode for this function works, but I'm worried it's only by
coincidence and not something upon which I should rely.

What's going on? What is wrong with my PInvoke signatures? Why is it that
manually marshaling the string seems to be the only thing that works?

[opellegr]

I'd like to add I found something interesting that I hadn't noticed when
fooling with this.

There are two versions of the function, one that uses "..." in the parameter
list, and another that takes a va_list parameter. The PInvoke declaration
I'm using references the va_list version (which is the only exported one, I'm
talking to the developer about that and will experiment with it.)

I'm still very curious why params arrays seem to work properly for
everything but strings. Is this indicative that I'm doing something wrong?

Post by opellegr
I have to use PInvoke to call a C library function that has the following
foo(const char *, const char *, EnumType, ...)
The function configures a resource in a way that varies by StructType; the
case I'm interested in configures something that expects the varargs to be a
single ANSI multibyte string. I used this PInvoke signature to call the
[DllImport(DllName, CharSet = CharSet.Ansi, EntryPoint = "foo")]
public static extern int Foo(string s1, string s2, EnumType e1, params
string[] s3);
The params string[] seemed odd to me, but previous signatures that called
this function were using it for other types such as bool so I followed the
pattern.
public void Foo(string s1, string s2, string s3)
{
int error = Dll.Foo(s1, s2, EnumType.Foo, s3);
// handle errors
}
Recently I changed the signature to include "BestFitMapping = false,
ThrowOnUnmappableChar = true" in the DLLImport attribute to comply with
FxCop's suggestions. This is a red herring as I will describe later.
What I expect out of making this change is limited support for Unicode. For
example, on a machine with an English code page an exception will be thrown
if you pass a string that contains a Japanese character. On a Japanese
machine, I'd expect to be able to pass a Japanese string to the function.
The English test worked as expected, but the Japanese test throws a
System.Runtime.InteropServices.COMException with HRESULT 0x8007007A. This
happens even without the BestFitMapping and ThrowOnUnmappableChar settings.
I've done a little looking around and saw some sites that suggested you
could PInvoke varargs by just specifying normal arguments, so I tried this
[DllImport(DllName, CharSet = CharSet.Ansi, EntryPoint = "foo")]
public static extern int Foo(string s1, string s2, EnumType st1, string
s3);
This throws an AccessViolationException when I use it on either the English
or Japanese machine.
After some digging around, I found out that you have to use the CDecl If I
[DllImport(DllName, CharSet = CharSet.Ansi, EntryPoint = "foo",
CallingConvention = CallingConvention.Cdecl)]
public static extern int Foo(string s1, string s2, EnumType e1, string
s3)
When I use this specification, I get an AccessViolationException.
[DllImport(DllName, CharSet = CharSet.Ansi, EntryPoint = "foo",
BestFitMapping = false, ThrowOnUnmappableChar = true)]
public static extern int Foo(string s1, string s2, EnumType e1, params
IntPtr[] s3)
To make this work, I use Marshal.StringToHGlobalAnsi() and pass that
pointer. This works for English and Japanese. I'm not comfortable with not
understanding why this is the solution, but it works.
I cannot use UTF-8 or another Unicode encoding because this C library
expects and handles only multibyte ANSI. I have found that specifying
CharSet.Unicode for this function works, but I'm worried it's only by
coincidence and not something upon which I should rely.
What's going on? What is wrong with my PInvoke signatures? Why is it that
manually marshaling the string seems to be the only thing that works?