DirectX HLSL compiler chokes on unicode sources

In a week or so I’ve created two HLSL effects in two different .fx files. While they were really simple and straightforward, one of them failed to load at runtime in my managed DirectX 3D application (yet both worked in NVidia FX Composer). Why would one fail while another is working? The error information was really non existent (i.e. “there was an error in the application” – DirectX, thanks so much for all of the details…). So I tried the fxc compiler that comes with DirectX SDK. The problematic effect didn’t compile, of course. This time the error was more verbose but highly misleading:

ID3DXEffectCompiler: There were no techniques

The problem with this message was that I had a technique defined (remember, the thing was working in FX Composer). Hm. Google didn’t help me at all. Then I went with my standard problem solving approach (SPSA) – eliminate pieces step by step (take a working sample and modify it slowly step by step to pinpoint the problem). Ok, so I copied the working effect content into non-working file opened in editor (I did copy&paste of the code, not the file). I tried fxc again and to my surprise it yelled the same error at me. What? How could it be? The content was the same. Perhaps I’ve drank too much coffee today? So I used KDiff3, a file comparison tool (free), to compare both files. On opening it warned me that while the content is the same the binary content isn’t. This info finally triggered my brains – it has to be the unicode signature prefix. To be sure I fired yet another free utility, this time a hex editor called HxD. And there is was a hex prefix of EF BB BF meaning an UTF-8 encoded text. We live in an unicode universe for a bunch of years now and I guess fxc should work with non-ASCII files as well. Or at least it should complain that file isn’t an ASCII one.