Death by a thousand cuts

Every year I decide to spend some time refreshing OpenCover i.e. upgrading to the latest tools such as Visual Studio, upgrading all the packages that OpenCover depends on etc. etc. and it is never a good time for me.

I don't know why I do this to myself, I know it is going to hurt and it's always the tiny things that somehow take ages to remedy. However I need to do this so that I can uninstall old versions of Visual Studio before I move on to addressing some of the latest features and issues within OpenCover.

Thing is, as projects go that I have to deal with on a regular basis, OpenCover is one of the simplest, it has around 20 projects and there are around 30 external packages and the majority of those projects and packages are there for testing purposes. I am quite happy about the level of testing (it is not perfect - but when is it ever?) and the number of serious defects are quite low; as of time of writing, the only defects in the issues are the ones we raised ourselves from the error logs we received and an issue relating to support for the latest Fakes that comes with Visual Studio 2015.

I've decided this time that I am going to list the steps I go through over the next few days (or weeks depending on whatever time I can find) as I do this again for 2016.

Upgrade the project from Visual Studio 2013 to Visual Studio 2015

[Note: that I didn't go for Visual Studio "15" because it isn't supported by AppVeyor yet and I am not a masochist.]
Installer project failed to upgrade - upgrade to latest Wix in the 3.X series and try again.
No errors during upgrade this time.

Building

Fix C++ issues due to std::hash_map no longer being supported.
Some C# projects no longer compile due to some Roslyn related intellisense error, what the ...?
- [external package Mono.Gendarme can no longer be found even though it is in the references, another developer also found this and reported something similar on our gitter channel.]
- This makes no sense, remove it, add it again - nope, remove it and get ReSharper to add it - nope.
- Check the signing process [the package comes unsigned so I have to manually sign it; is it really that hard to add a strong name key everyone?] - no change, still works in Visual Studio 2013 though. Why me?
- Upgrade to latest ReSharper [thanks again for the freebie JetBrains.], wait....
- Still doesn't find the reference (yes, I know, it was wishful thinking.) It's there, I can see it, why don't you look harder Visual Studio - Roslyn - whatever...!
- ... have few beers whilst going round in circles and my google-fu is failing me.
- New tack, use new ReSharper to decompile library into code and pull the bits I want into a new file, compile, move on - bite me!
- Fix installers and other packaging steps due to assembly removal.

Testing
- Seems changes to the compiler behaviour mean that some of the tests that use the output of the compiler to test how OpenCover works now fail, at least it is consistent in release and debug builds. Fix them up.
- SpecFlow integration tests fail on command line. Have to install new plugin that supports Visual Studio 2015 to run them in Visual Studio using ReSharper.
- Find I can't debug (use breakpoints that is) tests that have been built as Release even though full PDBs exist. Get latest updates of Visual Studio 2015 i.e. Service pack 2, wait... [I am sure I installed Visual Studio 2015 quicker than it took to service pack it.]. What have they done to the logo..? Never mind, let's just move on.
- Breakpoints are now working and, okay, fix the parsing [how did we get away with that for so long] and the tests are now passing again.
Building

Build on the command line again... Success!
Build release build on the command line... Success!
Push to GitHub and run a test build on AppVeyor... Success!

Upgrading the Packages

First undo the hack from what now seems weeks away as I can now once again reference Mono.Gendarme; It seems the Service Pack fixed many things, I promise I am not bitter about this, much.
I am now on familiar ground, so let's have a look at what packages need upgrading using Visual Studio, okay 9 updates, 8 of them are used only for testing and 6 of them have gone through major release number changes; I am worried about these because if there is going to be a breaking change then it'll be those 6; it's not showing the solution only packages so I'll deal with them manually.
First, upgrade the 3 packages with the potentially least impact i.e. no major version number changes. [By the way, the UI for Nuget package management is far, far better than before]. Build, test... Success!
Now try the other packages one by one [this allows me to rollback or fix issues with just a single package at a time]. My biggest concern is Nunit and Specflow as these are dependent on each other and my "spider-sense" is tingling, so I am going to leave these to last.
- Unity (used for testing only). Build, test... Success!
- xUnit (used for testing only). Build, test... Fail. Update xunit.runners package, find I also need to update nuget.exe itself [at least that is just a one liner], only to find it is empty except for a readme file informing me to download another package. Download that package. Update scripts, build, test, ... Success!
- Specflow (used for testing only). There are a number of packages all interlinked on versions and dependencies. Now running in a strange universe with two versions of NUnit packages in my solution; I know from past experience they will not play nicely with each in the same folder so will need to take the leap on NUnit soon. Run tests using ReSharper, tests fail due to execution folder no longer the same, fix them up and try again... Success!
- NUnit (used for testing only). I've heard some war stories about this upgrade... Well that went well only one build issue (obsolete method) and easily corrected. Build, test... Fail. It seems more than just obsoleted types have changed, though not sure if it is a NUnit thing or a ReSharper integration thing. Ah, more tests failing due to execution folder has changed, lets solve them first as we have a solution for that. And, now we are left with a change in how we handle fields used for ValueSource, easy to remedy. Now I need to manually update the runner for the command line build. Build, test... Success! Because I've upgraded the test runner I'll push the build to AppVeyor again just in case something serious has changed.
- log4net (used in multiple projects). Build, test... Success!

And we're done, well hopefully, nothing serious has been caught and we are now running on Visual Studio 2015 with the ability to, at last, start using the latest syntax available. Now this upgrade took about 3 days overall with a lot of elapsed time in between whilst I did other things (life) and waiting on stuff to install.