Fixing combination of NuGet and Team Foundation in workgroup configuration: 401 Unauthorized

The problem

A lot of users of Visual Studio 2010 (SP1), Team Foundation Server in workgroup and NuGet faced a very annoying problem – often we’d get 401-Unauthorized when installing/uninstalling/updating a NuGet package. Apparently it happens only in this combination (not sure if my host OS – Windows 7 plays any role in it) and not consistently. But when it starts the only way to get rid of errors is to restart Visual Studio.

The only workaround so far was to:

  1. Go Offline with TFS
  2. Manually make files writable (csproj, packages configuration, etc.) or uncheck them before #1
  3. Close Visual Studio
  4. Open Visual Studio
  5. Do NuGet
  6. Close Visual Studio
  7. Open Visual Studio
  8. Go Online with TFS

The steps above were mandatory for every batch of NuGet operations. Which is a huge pain and absurdly great annoyance with, otherwise excellent, NuGet. Needless to say I was among the people facing this issue. And I get so annoyed that I decided to make a choice at that point: either ditch NuGet or fix it myself (NuGet is an open source project).

Being a developer I opted for second choice of course. Was there really a choice? Anyway, here is how it went my 24hrs of debugging and 15s fixing. If you just want to see the solution feel free to skip to the Solution below.

Debugging

1. I downloaded NuGet sources.

2. When opening NuGet solution I quickly find out that I was missing Visual Studio 2010 SDK (because NuGet is an Visual Studio extension) so I downloaded the one I’ve found on the Internet. And it didn’t install saying something about prerequisites not installed. Ah, one needs Visual Studio 2010 SP1 SDK. Get it here. Somebody, please let know Visual Studio Extensibility Developer Center that they are listing the outdated SDK.

3. I set NuGet.VsExtension as Start Up project and fired up the debugger. Which opens another instance of Visual Studio 2010 where I’ve crafted up a sample solution used for reproducing the dreadful 401. I was able to reproduce it often but not always.

4. It took me some time to get familiar with NuGet sources. After that I invested some time to speed up the problem detection (as soon as possible the better) by modifying pieces of NuGet sources and after many trials and errors I’ve found that I have to dig deeper, into the bowels of Team Foundation System Client code.

5. I fired up my preferred tool for debugging assemblies for which I don’t have sources – .net reflector. It works better than using MS source symbols and it works for every assembly. It doesn’t work perfectly but that’s due to assembly optimizations and other black magic issues but it works well enough. Armed with decompiled TFS client assemblies I dug deeper and deeper. But couldn’t find an obvious fault.

6. I brought up a new weapon: Microsoft Network Monitor to analyse the network traffic. After all TFS communication is through HTTP/SOAP. There I’ve found the first clue to the root of the problem. Normally TFS client would send a request that would be refused by server with response saying that NTLM authentication is required. The client would re-send request with NTLM authentication and everything would work. But when the problem occurs the client just doesn’t respond to NTLM challenge – instead it just throws 401 unauthorized exception without even trying to authenticate against the server. I had no idea why it sometimes work and sometimes not.

Successful communication
Successful communication

Unsuccessful communication
Unsuccessful communication

7. At this point I was thinking of enabling System.Net tracing to get more useful info if possible. Immediately I faced a problem. The only way to enable System.Net is through app.config file but not in code. See, I couldn’t use app.config file because I was debugging a library and library’s app.config file is simply ignored. I’ve looked for a way to enable tracing programmatically in code, which is possible for user tracing scenarios, but not for System.Net. Bad luck, but there is nothing that can’t be fixed with a bit of reflection, like this:

private static void InitLogging()
{
    TextWriterTraceListener listener = new TextWriterTraceListener(@"D:\temp\ts.log");
    Type type = typeof(HttpWebRequest).Assembly.GetType("System.Net.Logging");
    MethodInfo initl = type.GetMethod("InitializeLogging", System.Reflection.BindingFlags.Static | System.Reflection.BindingFlags.NonPublic);
    initl.Invoke(null, null);

    foreach (string s in new string[] { "s_WebTraceSource", "s_HttpListenerTraceSource", "s_SocketsTraceSource", "s_CacheTraceSource" })
    {
        FieldInfo webTsFi = type.GetField(s, BindingFlags.Static | BindingFlags.NonPublic);
        TraceSource webTs = (TraceSource)webTsFi.GetValue(null);
        webTs.Switch.Level = SourceLevels.Verbose;
        webTs.Listeners.Add(listener);
    }
    FieldInfo le = type.GetField("s_LoggingEnabled", BindingFlags.Static | BindingFlags.NonPublic);
    le.SetValue(null, true);
}

And voila, the thing started to spit a ton of information into file D:\temp\ts.log. But again, it only showed the symptom but not the cause (trace parts after first request, note that unsuccessful one doesn’t even try to NTLM authenticate):

System.Net Information: 0 : [10488] Associating HttpWebRequest#51488348 with ConnectStream#13361802
System.Net Information: 0 : [10488] Associating HttpWebRequest#51488348 with HttpWebResponse#7364733
System.Net Information: 0 : [10488] AcquireDefaultCredential(package = NTLM, intent  = Outbound)
System.Net Information: 0 : [10488] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = (null), targetName = HTTP/TFS, inFlags = Delegate, MutualAuth, Connection)
System.Net Information: 0 : [10488] InitializeSecurityContext(In-Buffers count=0, Out-Buffer length=40, returned code=ContinueNeeded).
System.Net Warning: 0 : [10488] HttpWebRequest#51488348::() - Resubmitting request.

Successful communication

System.Net Information: 0 : [10488] Associating HttpWebRequest#51488348 with ConnectStream#13361802
System.Net Information: 0 : [10488] Associating HttpWebRequest#51488348 with HttpWebResponse#7364733

Unsuccessful communication

8. At this point I concentrated on debugging System.Net.HttpWebRequest class as re-submitting is not done at TFS client level. After even more trial and errors I was finally able to pinpoint the root of the evil.

The root of the problem

The decision whether to or not to try NTLM authentication is based on which internet zone OS thinks the request target is. In other words if OS says that your TFS server is outside intranet then HttpWebRequest won’t bother with NTLM authentication at all. It is that simple. The decision lies within PresentationCore’s (!) internal CustomCredentialPolicy.InternetSecurityManager class which delegates the question about the internet zone to OS and returns the result to HttpWebRequest. For some reason at some point it starts to return Internet instead of Intranet. I am not sure exactly why, but I have a remedy. A dramatically simple one which doesn’t even involve modifications to NuGet (no need to wait for a NuGet fix!).

The solution

Open Internet Explorer browser, go to Internet Options/Security, select Local Intranet icon, click Sites button

image

On Local Intranet dialog click Advanced

image

and add your TFS server to the Websites list, like I did with mine (replace TFS with the name of your server)

image

Restart Visual Studio any enjoy NuGet from a new perspective!

This solution apparently solves all of the issues I had with the dreaded 401. Let me know if it works for you as well.

Considerations

The problem might not be related to NuGet at all but rather to PresentationCore (NuGet is a WPF application) which gets confusing results from OS through some interop. NuGet/Visual Studio is just a combination that triggers the otherwise sleeping problem.

Comments (34) -

  • Rudi Larno

    9/2/2011 11:02:45 AM | Reply

    Miha, great post. Unfortunately this does not always seem to solve the problem. I already had my tfs server url in the Intranet Zone set, and still get the dreaded TF30063 error while using NuGet. Fortunately, simply restarting VS2010 is enough to get NuGet with TFS working again.

    • Miha Markic

      9/2/2011 11:04:37 AM | Reply

      Hi Rudi,

      Did you check what's going on on the network? That'd be my first step.

  • Daniel

    9/4/2011 2:02:07 AM | Reply

    Seemed to fix my problem. Great post! Thanks!

  • Anthony

    9/7/2011 12:18:15 AM | Reply

    Fixed my problem, too. Thanks!

  • Roman

    10/19/2011 4:06:42 AM | Reply

    Fixed my problem too! Thanks!

  • Jaime

    11/8/2011 11:32:39 PM | Reply

    I had the same problem here. At the begining I suspected about the recently installed Builder service (there are some webpages that relate this problem with the associated builder user).
    My TFS is connected to an Active Directory, not a workgroup but presented the same problem.
    This blog fully solved my problem!!!!
    Just added my server name to the Intranet list: http://*.mydomain.com/tfs

  • Erik Nordin

    11/14/2011 4:29:12 PM | Reply

    Thanks, been searching all day for a solution.

  • Ryan Haney

    12/4/2011 6:25:39 AM | Reply

    Changing the settings in IE worked for me, without restarting visual studio.

    THANKS!

  • JYL

    12/18/2011 9:09:45 PM | Reply

    Awesome ! Thanks !

  • Tim B

    1/27/2012 8:24:30 PM | Reply

    Thank you your diligent effort, which I expect has helped hundreds of people, if not more--fewer than 1% of the people helped bothered to make a comment.

    I was running Visual Studio in Admin mode and had been running the Azure emulator. I wanted to mention that I did nothing more than restart Visual Studio, but not in admin mode, and it seemed to fix the error. So, people may wish to try this approach as well.

  • Pawel

    2/3/2012 12:47:12 PM | Reply

    Thanks a lot!!!
    It worked for me as well.

  • Bojan

    2/3/2012 3:59:02 PM | Reply

    Looks like this helps. Thank you.

  • Tomer

    2/9/2012 1:59:47 PM | Reply

    Thanks! This solves the problem.

  • Mike

    3/4/2012 12:21:13 AM | Reply

    Thank you for this workaround. This was a lifesaver!

  • Peter

    3/12/2012 10:07:10 PM | Reply

    Thanks it worked for me.  If you mention the fix first it would have saved me more time.  I had to go offline and have TFS complain about things when I reconnected.

  • Vitalit

    5/16/2012 12:00:49 PM | Reply

    Thanks you very much. Works as magic Smile

  • Robin

    6/13/2012 11:12:35 AM | Reply

    What settings should be applied when using codeplex?

    • Miha Markic

      6/13/2012 11:27:33 AM | Reply

      Do you have a particular problem with codeplex? I think codeplex settings are, well, on codeplex site.

  • shrike

    6/22/2012 6:04:50 PM | Reply

    Госпади, вот оно что!

    You've saved my mind, man. Thanx.

  • Björn

    6/27/2012 12:04:44 PM | Reply

    Good stuff! Thanks!

  • Rob

    7/3/2012 10:42:16 AM | Reply

    awesome seems to slove it, been stalking our tech support for some time now without any good answers.

  • jony

    7/23/2012 10:08:12 AM | Reply

    Thanks! Helped me fix all sorts of problems me and my team were experiencing Smile

  • Mariusz.W

    7/31/2012 4:01:16 PM | Reply

    Ha! Thank you very much!

  • jungleleo

    8/21/2012 3:30:54 AM | Reply

    Thanks you ,help me a lot

  • tkrause

    8/22/2012 4:34:51 PM | Reply

    Our domain policy overrides the sites stored in local intranet. I added the TFS url to the trusted sites. SEEMS to work also! Thank you!

  • Paul K

    9/4/2012 11:40:24 PM | Reply

    Worked for me as well! Appreciate all the debugging you put in to find this solution! Thanks!!

  • BossHogg6

    9/13/2012 4:34:13 PM | Reply

    Still having the same issue.

  • Diogo

    10/3/2012 2:32:35 AM | Reply

    Thank you very much for sharing! It worked for me.

  • Malone

    2/13/2013 11:31:53 AM | Reply

    Great article. It helped me solve the same issue with another VS 2010 extension.

  • Sharpguru

    3/13/2013 5:30:04 PM | Reply

    Works for me, Thankyou very much!!!

  • Filip Stanek

    3/22/2013 3:30:46 PM | Reply

    I had the same issue after we updated the password to the TFS server. I had to update the password on VS 2012 and the problem went away.

  • Manav

    4/26/2013 3:18:54 PM | Reply

    thanks. it helped to resolve the issue in TFS Integration Tool.

  • Prasad Kopanati

    7/10/2013 11:06:25 PM | Reply

    Thanks.. Worked like a charm. I had the TFS error every time I installed a package using Nuget. After using your tip, it is gone.. Thanks so much.

  • Andrew Bennett

    8/9/2013 1:36:06 PM | Reply

    Thanks for this fix, saved me a lot of time! Unfortunately, it highlights the rather flakey nature of the TFS product. You just don't get this with SVN.

Loading