SDN Journal Authors: Elizabeth White, Yeshim Deniz, Liz McMillan, Pat Romanski, TJ Randall

Related Topics: Containers Expo Blog, @CloudExpo, SDN Journal

Containers Expo Blog: Blog Post

The Network as a Complex Distributed System By @MartenT1999 | @CloudExpo

Yes, we have gotten better at protecting this distributed animal that is the network

Through http://blog.ipspace.net I landed on this article on acm.org discussing the complexity of distributed systems. Through some good examples, George Neville-Neil makes it clear that creating and scaling distributed systems is very complex and “any one that tells you it is easy is either drunk or lying, and possibly both”.

Networks are of course inherently distributed systems. Most everyone that has managed a good sized network before knows that like the example in the article, minor changes in traffic or connectivity can have huge implications on the overall performance of a network. In my time supporting some very large networks I have seen huge chain reactions of events based on what appear to be some minor issues.

Very few networks are extensively modeled before they are implemented. Manufacturers of machines, cars and many other things go through extensive modeling to understand the behaviors of what they created and their design choices. Using modeling they will look at all possible inputs and outputs, conditions, failure scenarios and anything else we can think of to see how their product behaves.

There are few if any true modeling tools for networks. We build networks with extensive distributed protocols to control connectivity and reachability. The protocols and their behaviors are fairly well understood and we design our networks based on our knowledge of how these protocols will react to failures. There is very little modeling of actual traffic or expected traffic patterns. We have all seen networks melt down because of forwarding loops, excessive multicast or broadcast that affect portions of the networks we had not expected it would affect. And a chain reaction of failures follows. I have seen single transceivers go through soft failures (lots of errors and intermittent link state) and bring large networks to their knees.

Of course we learn from our mistakes (mostly) and created protection mechanisms for the things we know can go wrong. We created “hello” based protocols to guard against those soft transceiver failures. We create rate limiting and overload protection mechanisms to guard against those broadcast and multicast storms. We have implemented dampening mechanisms in many of our protocols to ensure state transitions can be managed more gracefully.

But in the end, like the example in the article, we have created artificial limits and protection mechanisms. And they are based on the things we know, because they have hurt us in the past. Less protective but more pro-active, we have implemented hashing mechanisms in our networks to distribute the traffic. As explained, these are generic mechanisms that are designed for uniform best case scenarios. Provide it with less uniform traffic, and the expected distribution is not what we want it to be. The 2014 favorite example of “elephant flow” is perfect in how a single change in a network can drastically change its behavior and performance.

Yes, we have gotten better at protecting this distributed animal that is the network. We have stubbed our toes, bashed our heads and figured out how to protect against it. But we have not taken it to that next level of engineering. The network is a product. It is a highly distributed system where we know way too little about the inputs. We do not model the network and its behavior. We do not test its resiliency. I have asked this question before in this forum and to customers. Every customer has maintenance windows to do proactive maintenance on servers, OSs and the like. Every school and company has fire drills. I have yet to meet a single customer that actually schedules network failure drills to ensure that the resiliency designed into the distributed system actually performs as expected.

We have to start treating the network like the complex distributed system it is. That means that complex algorithms that manage the distribution of traffic and take failure scenarios into account need to be in charge of our networks. Local decision making based on partially complete information can never be enough to truly create a network that pushes the boundaries of what it is capable of. It is not easy, but we have to recognize what the network is. And then approach it with the right tools and methods to model and manage it.

The post The Network as a Complex Distributed System appeared first on Plexxi.

Read the original blog entry...

More Stories By Marten Terpstra

Marten Terpstra is a Product Management Director at Plexxi Inc. Marten has extensive knowledge of the architecture, design, deployment and management of enterprise and carrier networks.

CloudEXPO Stories
Your job is mostly boring. Many of the IT operations tasks you perform on a day-to-day basis are repetitive and dull. Utilizing automation can improve your work life, automating away the drudgery and embracing the passion for technology that got you started in the first place. In this presentation, I'll talk about what automation is, and how to approach implementing it in the context of IT Operations. Ned will discuss keys to success in the long term and include practical real-world examples. Get started on automating your way to a brighter future!
The challenges of aggregating data from consumer-oriented devices, such as wearable technologies and smart thermostats, are fairly well-understood. However, there are a new set of challenges for IoT devices that generate megabytes or gigabytes of data per second. Certainly, the infrastructure will have to change, as those volumes of data will likely overwhelm the available bandwidth for aggregating the data into a central repository. Ochandarena discusses a whole new way to think about your next-gen applications and how to address the challenges of building applications that harness all data types and sources.
Whenever a new technology hits the high points of hype, everyone starts talking about it like it will solve all their business problems. Blockchain is one of those technologies. According to Gartner's latest report on the hype cycle of emerging technologies, blockchain has just passed the peak of their hype cycle curve. If you read the news articles about it, one would think it has taken over the technology world. No disruptive technology is without its challenges and potential impediments that frequently get lost in the hype. The panel will discuss their perspective on what they see as they key challenges and/or impediments to adoption, and how they see those issues could be resolved or mitigated.
CloudEXPO New York 2018, colocated with DevOpsSUMMIT and DXWorldEXPO New York 2018 will be held November 12-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI and Machine Learning to one location.
DXWorldEXPO LLC announced today that Nutanix has been named "Platinum Sponsor" of CloudEXPO | DevOpsSUMMIT | DXWorldEXPO New York, which will take place November 12-13, 2018 in New York City. Nutanix makes infrastructure invisible, elevating IT to focus on the applications and services that power their business. The Nutanix Enterprise Cloud Platform blends web-scale engineering and consumer-grade design to natively converge server, storage, virtualization and networking into a resilient, software-defined solution with rich machine intelligence.