Fixing combination of NuGet and Team Foundation in workgroup configuration: 401 Unauthorized

The problem

A lot of users of Visual Studio 2010 (SP1), Team Foundation Server in workgroup and NuGet faced a very annoying problem – often we’d get 401-Unauthorized when installing/uninstalling/updating a NuGet package. Apparently it happens only in this combination (not sure if my host OS – Windows 7 plays any role in it) and not consistently. But when it starts the only way to get rid of errors is to restart Visual Studio.

The only workaround so far was to:

1. Go Offline with TFS
2. Manually make files writable (csproj, packages configuration, etc.) or uncheck them before #1
3. Close Visual Studio
4. Open Visual Studio
5. Do NuGet
6. Close Visual Studio
7. Open Visual Studio
8. Go Online with TFS

The steps above were mandatory for every batch of NuGet operations. Which is a huge pain and absurdly great annoyance with, otherwise excellent, NuGet. Needless to say I was among the people facing this issue. And I get so annoyed that I decided to make a choice at that point: either ditch NuGet or fix it myself (NuGet is an open source project).

Being a developer I opted for second choice of course. Was there really a choice? Anyway, here is how it went my 24hrs of debugging and 15s fixing. If you just want to see the solution feel free to skip to the Solution below.

Debugging

1. I downloaded NuGet sources.

2. When opening NuGet solution I quickly find out that I was missing Visual Studio 2010 SDK (because NuGet is an Visual Studio extension) so I downloaded the one I’ve found on the Internet. And it didn’t install saying something about prerequisites not installed. Ah, one needs Visual Studio 2010 SP1 SDK. Get it here. Somebody, please let know Visual Studio Extensibility Developer Center that they are listing the outdated SDK.

3. I set NuGet.VsExtension as Start Up project and fired up the debugger. Which opens another instance of Visual Studio 2010 where I’ve crafted up a sample solution used for reproducing the dreadful 401. I was able to reproduce it often but not always.

4. It took me some time to get familiar with NuGet sources. After that I invested some time to speed up the problem detection (as soon as possible the better) by modifying pieces of NuGet sources and after many trials and errors I’ve found that I have to dig deeper, into the bowels of Team Foundation System Client code.

5. I fired up my preferred tool for debugging assemblies for which I don’t have sources – .net reflector. It works better than using MS source symbols and it works for every assembly. It doesn’t work perfectly but that’s due to assembly optimizations and other black magic issues but it works well enough. Armed with decompiled TFS client assemblies I dug deeper and deeper. But couldn’t find an obvious fault.

6. I brought up a new weapon: Microsoft Network Monitor to analyse the network traffic. After all TFS communication is through HTTP/SOAP. There I’ve found the first clue to the root of the problem. Normally TFS client would send a request that would be refused by server with response saying that NTLM authentication is required. The client would re-send request with NTLM authentication and everything would work. But when the problem occurs the client just doesn’t respond to NTLM challenge – instead it just throws 401 unauthorized exception without even trying to authenticate against the server. I had no idea why it sometimes work and sometimes not.

Successful communication

Unsuccessful communication

7. At this point I was thinking of enabling System.Net tracing to get more useful info if possible. Immediately I faced a problem. The only way to enable System.Net is through app.config file but not in code. See, I couldn’t use app.config file because I was debugging a library and library’s app.config file is simply ignored. I’ve looked for a way to enable tracing programmatically in code, which is possible for user tracing scenarios, but not for System.Net. Bad luck, but there is nothing that can’t be fixed with a bit of reflection, like this:

private static void InitLogging()
{
    TextWriterTraceListener listener = new TextWriterTraceListener(@"D:\temp\ts.log");
    Type type = typeof(HttpWebRequest).Assembly.GetType("System.Net.Logging");
    MethodInfo initl = type.GetMethod("InitializeLogging", System.Reflection.BindingFlags.Static | System.Reflection.BindingFlags.NonPublic);
    initl.Invoke(null, null);

    foreach (string s in new string[] { "s_WebTraceSource", "s_HttpListenerTraceSource", "s_SocketsTraceSource", "s_CacheTraceSource" })
    {
        FieldInfo webTsFi = type.GetField(s, BindingFlags.Static | BindingFlags.NonPublic);
        TraceSource webTs = (TraceSource)webTsFi.GetValue(null);
        webTs.Switch.Level = SourceLevels.Verbose;
        webTs.Listeners.Add(listener);
    }
    FieldInfo le = type.GetField("s_LoggingEnabled", BindingFlags.Static | BindingFlags.NonPublic);
    le.SetValue(null, true);
}

And voila, the thing started to spit a ton of information into file D:\temp\ts.log. But again, it only showed the symptom but not the cause (trace parts after first request, note that unsuccessful one doesn’t even try to NTLM authenticate):

System.Net Information: 0 : [10488] Associating HttpWebRequest#51488348 with ConnectStream#13361802
System.Net Information: 0 : [10488] Associating HttpWebRequest#51488348 with HttpWebResponse#7364733
System.Net Information: 0 : [10488] AcquireDefaultCredential(package = NTLM, intent  = Outbound)
System.Net Information: 0 : [10488] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = (null), targetName = HTTP/TFS, inFlags = Delegate, MutualAuth, Connection)
System.Net Information: 0 : [10488] InitializeSecurityContext(In-Buffers count=0, Out-Buffer length=40, returned code=ContinueNeeded).
System.Net Warning: 0 : [10488] HttpWebRequest#51488348::() - Resubmitting request.

Successful communication

System.Net Information: 0 : [10488] Associating HttpWebRequest#51488348 with ConnectStream#13361802
System.Net Information: 0 : [10488] Associating HttpWebRequest#51488348 with HttpWebResponse#7364733

Unsuccessful communication

8. At this point I concentrated on debugging System.Net.HttpWebRequest class as re-submitting is not done at TFS client level. After even more trial and errors I was finally able to pinpoint the root of the evil.

The root of the problem

The decision whether to or not to try NTLM authentication is based on which internet zone OS thinks the request target is. In other words if OS says that your TFS server is outside intranet then HttpWebRequest won’t bother with NTLM authentication at all. It is that simple. The decision lies within PresentationCore’s (!) internal CustomCredentialPolicy.InternetSecurityManager class which delegates the question about the internet zone to OS and returns the result to HttpWebRequest. For some reason at some point it starts to return Internet instead of Intranet. I am not sure exactly why, but I have a remedy. A dramatically simple one which doesn’t even involve modifications to NuGet (no need to wait for a NuGet fix!).

The solution

Open Internet Explorer browser, go to Internet Options/Security, select Local Intranet icon, click Sites button

On Local Intranet dialog click Advanced

and add your TFS server to the Websites list, like I did with mine (replace TFS with the name of your server)

Restart Visual Studio any enjoy NuGet from a new perspective!

This solution apparently solves all of the issues I had with the dreaded 401. Let me know if it works for you as well.

Considerations

The problem might not be related to NuGet at all but rather to PresentationCore (NuGet is a WPF application) which gets confusing results from OS through some interop. NuGet/Visual Studio is just a combination that triggers the otherwise sleeping problem.