VAR&D

I had an interesting discussion with a client a few days ago that was centered around code levels on devices. We’re updating some code on a pair of Nexus 7010’s in a few weeks and we spent some time poring over the release notes, upgrade/downgrade procedures, and known bugs as it relates to the code version we are moving to. We are also going over to the local Cisco office to use their lab gear to verify these procedures and that the code bump won’t break anything.

That led to a broader discussion around how we in the VAR world can verify that all the moving pieces work together and all potential problems are identified before any implementation or upgrade. This particular engineer had just come from a much larger environment where he had Spirent testing gear and plenty of spare hardware to test things before deployment. His contention was that you could really add more to the “value added” part of VAR if you could offer additional assurance around deployments and upgrades.

What Usually Happens?

If a client needs to upgrade their code on certain hardware, they typically have to rely on release notes and upgrade instructions from the vendor. Maybe there is a known bug list available for the version they are upgrading to. Maybe not. It depends on the vendor. They also might just update the code after the first update beyond the major release has been released. For example, platform X gets upgraded to ver 2.1 since it is the first update since ver 2.0. I’ve been in quite a few environments where the rule of thumb was not to upgrade any software until the first service pack or major patch had been released.

No matter the approach you take from the above choices, you are taking a risk that all will go well. If you are a large enough organization, or possibly tech focused, you might have a lab that has the same hardware as your production network. If so, you can actually do comprehensive testing, provided you actually have time to get that done. After all, you have meetings and conference calls to sit through right?

Current Testing State

From my sloppy and lazy research, which consisted of asking a question or two on Twitter and also reflecting on past experiences, I was able to determine a few things:

1) Vendors do extensive testing on their products. This may sometimes include other vendor hardware, but will probably not encompass anything more than the most common scenarios.

2) VARs will do testing on a case by case basis, but only the big ones are going to have the right gear to do that.

3) If you REALLY need to make sure, and you are a good enough customer, you can go out to a customer proof of concept center run by the vendor whose gear you are using and they can test various scenarios for you.

4) Just do the upgrade, call support if it breaks, and let the vendors slug it out with each other. After all, that’s why you pay them for maintenance and support.

What Could Happen?

Imagine a world where you could go to a VAR and ask them about potential problems with a code upgrade or a multi-vendor implementation project and they could tell you what you could realistically expect. I’m not talking about the “we’ve done this before and never had a major problem”. I’m talking about them being able to take your present network state and future network state and give you some concrete information around what your experience will be post-upgrade.

My initial thoughts were that a VAR could sink a pile of money into a lot of lab gear and start building out various tests and have engineers break things and fix them. Of course, that means those engineers aren’t out getting those ever so important services dollars from post-sales efforts, or they aren’t smooth-talking customers over fancy meals during pre-sales engagements. You have to be willing to take the loss on the testing engineers with the hope that you make the cost of their salaries back in fees from clients.

The particular client who filled me with this idea suggested that it might be better for a company to do this testing and then sell that information to VARs in some type of package deal. Perhaps along the lines of a subscription service similar to what a company may do with Gartner to gain access to their reports and analysts.

Plausible Scenario?

Customer ABC goes to VAR XYZ to get some advice on a planned code upgrade of their distribution switches. These switches run OSPF and have neighbor relationships with their firewalls as well as some routers. The switches, firewalls, and routers are from different manufacturers. Customer ABC just wants to make sure they can perform the upgrade without their network exploding. VAR XYZ has consulted the testing company, and they are able to provide them with information regarding those particular products and the code levels they are running and are going to run after the upgrade. VAR XYZ then passes this information along to the customer, who decides whether or not to proceed based on the test results.

Closing Thoughts

Is that a realistic endeavor? Could it be done and have credibility? I think so, provided there is no money coming from vendors. That just ruins it in terms of credibility. That isn’t to say that you can’t extract value from vendor sponsored tests. You can, but you must take that with a grain of salt.

Of course, any type of company doing the testing would require about a billion pages of legal mumbo jumbo to avoid getting sued. They would also have to have some pretty precise testing methodologies to ensure valid results. There’s also the issue of not being able to test every possible scenario and pre-package the results. You could develop the most popular configurations over time and test one-offs when requested.

What do you think about something like this being a reality in the IT industry? Would companies pay for that kind of data or is this not realistic?