Thursday, June 5, 2014

MSTest testsettings annoyance in automated builds

When you run a test using the MSTest command-line, it outputs a note that it used the default test settings. When searching my PC, I couldn't find anything that seemed to be a default testsettings file (extension .testsettings), so I gave up and assumed if one isn't specified then defaults are hardcoded or come from some location other than a .testsettings file... ok, fine, no big deal.

The problem is when we changed everything over from LINQ-to-SQL to Entity Framework, it caused 10 of our unit tests to start failing in the automated build, but when run locally in Visual Studio, they were succeeding. Turns out the failures also manifested themselves locally through the MSTest command-line. But if you create a dummy .testsettings file (just the shell of the XML with nothing really set), then it works.

Somehow the unspecified-testsettings behavior is different than the behavior when specifying a dummy .testsettings file... Some DLL's weren't getting copied to the right place, and I know you can configure various deployment settings in the testsettings (and configuring deployment and out folders has always been a nuisance in MSTest), but I didn't even have to do any of that -- it just worked with a dummy file. And all our tests now magically pass again on the build server.

Not to mention the nuisance that Microsoft can't seem to settle on a test settings file format or UI for modifying them... They had testrunconfig (which had a GUI editor as part of Visual Studio) and then they changed it to testsettings, and it had a GUI editor so it was fine... Now they (sort of? apparently? it's hard to tell...) deprecated those two and have runsettings instead, which has no GUI editor and doesn't do all the same things as testsettings and testrunconfig.

By the way, where are my trx files now when testing through Visual Studio 2012? The testing interface -- test list, test results, etc. -- has gone way backwards from Visual Studio 2010, and it's quite frustrating. But let's not get started on VS2012 and the ALL CAPS MENU BAR, badly redesigned pending-changes window, various workflow annoyances, etc... That's another bloog post that I'd rather not do.

Now, Microsoft, in your defense, you never claimed any such "Don't be evil" policy, so carry on doing what you do best. :)

P.S.: It's going to be great having Steve Ballmer as an NBA owner -- maybe they can let him give sweaty pre-game pep talks sometimes. "BALLPLAYERS BALLPLAYERS BALLPLAYERS, AAAH!"

No comments:

Post a Comment