I had an interesting discussion with a client a few days ago that was centered around code levels on devices. We’re updating some code on a pair of Nexus 7010’s in a few weeks and we spent some time poring over the release notes, upgrade/downgrade procedures, and known bugs as it relates to the code version we are moving to. We are also going over to the local Cisco office to use their lab gear to verify these procedures and that the code bump won’t break anything.
That led to a broader discussion around how we in the VAR world can verify that all the moving pieces work together and all potential problems are identified before any implementation or upgrade. This particular engineer had just come from a much larger environment where he had Spirent testing gear and plenty of spare hardware to test things before deployment. His contention was that you could really add more to the “value added” part of VAR if you could offer additional assurance around deployments and upgrades.
What Usually Happens?
If a client needs to upgrade their code on certain hardware, they typically have to rely on release notes and upgrade instructions from the vendor. Maybe there is a known bug list available for the version they are upgrading to. Maybe not. It depends on the vendor. They also might just update the code after the first update beyond the major release has been released. For example, platform X gets upgraded to ver 2.1 since it is the first update since ver 2.0. I’ve been in quite a few environments where the rule of thumb was not to upgrade any software until the first service pack or major patch had been released.
No matter the approach you take from the above choices, you are taking a risk that all will go well. If you are a large enough organization, or possibly tech focused, you might have a lab that has the same hardware as your production network. If so, you can actually do comprehensive testing, provided you actually have time to get that done. After all, you have meetings and conference calls to sit through right?
Current Testing State
From my sloppy and lazy research, which consisted of asking a question or two on Twitter and also reflecting on past experiences, I was able to determine a few things:
1) Vendors do extensive testing on their products. This may sometimes include other vendor hardware, but will probably not encompass anything more than the most common scenarios.
2) VARs will do testing on a case by case basis, but only the big ones are going to have the right gear to do that.
3) If you REALLY need to make sure, and you are a good enough customer, you can go out to a customer proof of concept center run by the vendor whose gear you are using and they can test various scenarios for you.
4) Just do the upgrade, call support if it breaks, and let the vendors slug it out with each other. After all, that’s why you pay them for maintenance and support.
What Could Happen?
Imagine a world where you could go to a VAR and ask them about potential problems with a code upgrade or a multi-vendor implementation project and they could tell you what you could realistically expect. I’m not talking about the “we’ve done this before and never had a major problem”. I’m talking about them being able to take your present network state and future network state and give you some concrete information around what your experience will be post-upgrade.
My initial thoughts were that a VAR could sink a pile of money into a lot of lab gear and start building out various tests and have engineers break things and fix them. Of course, that means those engineers aren’t out getting those ever so important services dollars from post-sales efforts, or they aren’t smooth-talking customers over fancy meals during pre-sales engagements. You have to be willing to take the loss on the testing engineers with the hope that you make the cost of their salaries back in fees from clients.
The particular client who filled me with this idea suggested that it might be better for a company to do this testing and then sell that information to VARs in some type of package deal. Perhaps along the lines of a subscription service similar to what a company may do with Gartner to gain access to their reports and analysts.
Plausible Scenario?
Customer ABC goes to VAR XYZ to get some advice on a planned code upgrade of their distribution switches. These switches run OSPF and have neighbor relationships with their firewalls as well as some routers. The switches, firewalls, and routers are from different manufacturers. Customer ABC just wants to make sure they can perform the upgrade without their network exploding. VAR XYZ has consulted the testing company, and they are able to provide them with information regarding those particular products and the code levels they are running and are going to run after the upgrade. VAR XYZ then passes this information along to the customer, who decides whether or not to proceed based on the test results.
Closing Thoughts
Is that a realistic endeavor? Could it be done and have credibility? I think so, provided there is no money coming from vendors. That just ruins it in terms of credibility. That isn’t to say that you can’t extract value from vendor sponsored tests. You can, but you must take that with a grain of salt.
Of course, any type of company doing the testing would require about a billion pages of legal mumbo jumbo to avoid getting sued. They would also have to have some pretty precise testing methodologies to ensure valid results. There’s also the issue of not being able to test every possible scenario and pre-package the results. You could develop the most popular configurations over time and test one-offs when requested.
What do you think about something like this being a reality in the IT industry? Would companies pay for that kind of data or is this not realistic?
I think the idea is interesting but from a business perspective i doubt you’d see the return anytime soon as most customers and VARS still expect the vendor to provide guidance (and assurance) for code rev and interoperability testing so there would be a long road to convincing customers or VARs that its worth paying for this service (and what assurances would you provide against being wrong or missing a test case, etc) versus just leaning on the vendors. I think ultimately the folks who are really that risk averse are the ones with the clout to lean on the vendors for the POCs.
Practically you’d have to be extremely thorough in documenting code and hardware revs, testing new code revs as they come out, and even new hardware revisions as well – it could quickly become unmanageable unless you tightly limit the scope of your testing (for example – are you testing strict software interoperability or are you also testing the effects of different hardware configurations, module in slot 1 vs slot 11, etc)
It probably is an unrealistic idea because if the granularity each test requires doesn’t kill you, the lawyers will.
Your assertion that the manufacturers of equipment actually test extensively may be a little flawed. At least, to be explicit, the software development and product management teams tend to not be the best advocates for comprehensive protocol testing. In terms of the ability to test in a vendor-neutral way, I’d submit we already have this with a number of third party labs (look at ISOCORE for not-the-best example) and what we should do is try to convince them that this is a worthwhile business to be involved in. In a homogenous network, the vendor is uniquely positioned to sell advanced services in order to validate that testing, and in some cases they may also, for enough money, do that service in a heterogenous environment as well. See, for instance, Cisco’s NOS and NAIS service offerings and the ‘Safe Harbor’ program.
This problem is bigger than going off and building a lab. While developing a comprehensive list of features and example topologies could be done there, the real challenge for the community is building effective testing frameworks and the challenge for the vendor is building better internal instrumentation frameworks with APIs we can all use. Companies like Arista really have the right idea where this is concerned; very little of the functionality is abstracted in such a way that the eventual consumer of the product could not access it.
Perhaps more importantly, there’s a real challenge in the network engineering community in that most people who would point out problems in gear are in many cases just generating noise. There isn’t enough competency except at the top tier of engineering talent to be able to describe or theorize why a box behaves the way it does, and most people just don’t understand things like hardware design, FIB architecture, Clos and Benes topologies, SerDes, etc. Before we as a community ask the vendors for more support so that such a lab could be built and truly be useful beyond just one-off testing a topology on the same physical equipment, we should ask ourselves how much of this could be abstracted into a software or virtualized environment and push the vendors for effective licensing and functionality to make such a thing possible.
Nonetheless, good thoughts and ideas.
Thanks for your comments. I agree that there is much to do in the realm of industry standards and vendors giving others hooks into their systems via API’s and the like. I’m glad you mentioned Arista. The first time I listened to their pitch regarding their switches, I was amazed at the level of access they gave you in regards to their platform.
That’s all well and good but few will take advantage of such a service. Those that would use the service would get little value for the money IMO. In the real world – mine anyways – we don’t have the time or budget to consult a VAR before every infrastructure upgrade. Code updates, specifically, are so rarely a problem that it’d feel like throwing money away. Besides, what are the chances that the magical testing center will be able to reproduce your environment exactly? How many times have we been on support calls and heard “I don’t know why you’re having problem XYZ, that’s never happened in our labs.”?
Maybe folks like Amazon or BP can afford that sort of thing, and they probably have 1,000 painful processes covering every conceivable change/upgrade scenario already. For folks like me running much smaller/fewer data centers, it just seems kind of silly. Overkill.
Do your due diligence and then trust your staff to push the button. Like you said – we have support contracts for a reason.
The bigger operations like Amazon are doing this, so it wouldn’t really be applicable to them. The networks that are much smaller than them, which are the majority around the world, can and do have problems with code changes.
I have always maintained that bumping up code levels just to stay current is a bad idea. I’ll do it for a specific feature, or to patch a security vulnerability. In environments that require very high levels of uptime, and have SLA’s with various portions of the business, doing code changes is a risky thing. You don’t know what will possibly go wrong. Add in multiple vendors, and it gets even trickier. I’ve been on numerous support calls where one vendor blames the other and only after many exhausting hours, does the problem finally get solved.
I’d agree that if you hardly ever have an issue, that it would seem silly to spend money on some sort of service that seeks to validate whatever you are about to do. I’ve also seen companies take that same approach to DR and have it bite them when they suffer a failure they can’t recover from in a timely manner.
As for due diligence, most companies don’t have the time. Even worse, when it comes to trusting their staff, many companies don’t do that either. They would rather pay a third party to tell them what their staff already did. Something like code validation and testing is an easy sell to the C level people. The engineers tend to be a bit more skeptical, and rightly so.