Dangers of an outsourced NOC – Network Operations Center

Yesterday evening I was asked to perform an emergency debugging of an MPLS problem on a rather large international backbone. To my astonishment their NOC Manager and Technical Design Engineer did not understand what I meant with:

Is the LSP still up and running or is the signalling lost somewhere?

Apparently the term LSP was unknown to them. So after explaining the term I started to ask a few more detailed questions to all the engineers present in the NOC.

Most of the engineers only knew the very, very basics of MPLS. The fact that packets are assigned a label and switched accordingly, not looking at the Layer 3 information was basicly their limit. Aside from configuring and debugging by the book no knowledge was present, at all.

Detailed and more advanced items like Forward Equivalence Class, MPLS Traffic Engineering,  Resource Reservation Protocol, Pseudowire Redundancy, Label Distribution Protocol, QoS / DiffServ, Path – Link and Node Protection where just unexplored territory.

This is a common problem of large Internet Service Providers outsourcing their NOC  to external companies abroad. Situations and issues like this happen on a daily base and reflect poorly on the ISP itself.

Either way, the issue was resolved very quickly so I can stop ranting about poorly maintained Network Operations Centers.

I am taking this opportunity to tell you guys a bit more about MPLS in general. The next few days I will talk about MPLS technologies, implementation guides and best practices. It’s also a good way to prepare myself for my upcomming Cisco CCIE R&S and CCDE certification exams.

As more and more networks are staring to implement MPLS on a massive scale I find it extremely important that a descent background knowledge of the implemented technologies are a fundamental requirement. I hope you will enjoy and learn from the upcomming series of MPLS related blog posts!