In many ways, observability is a measure of validation. At its best; network observability is the capacity to easily answer any question about your network, right now. The easier it is to answer your questions (to validate your network), the better your observability.
Furthermore, observability is not monitoring. Monitoring focuses on known unknowns, it watches for things we know to watch for (like high or low interface utilization), typically with the goal of alerting us when they happen. Observability on the other hand focuses on unknown unknowns, on being able to answer questions we didn’t know we were ever going to ask at all.
Origins of Observability
While this concept of observability is quite novel and abstract for many folks, it isn’t new. Like most of the terms already covered, observability too is borrowed. It originally comes from control systems engineering and more recently has become a hot topic in software engineering.
To understand it better, we’ll start with the Wikipedia definitions: “Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.” and “A system is said to be observable if, for every possible evolution of state and control vectors, the current state can be estimated using only the information from outputs.”
All that really says is that observability means understanding how a system works by observing it. It allows you to “determine the behavior of the entire system from the system’s outputs.”
In physical systems, these outputs are typically information gleaned from sensors. Think of the thermometer(sensor) telling us the temperature of the room (output) in our example above. In software systems, the outputs can come from almost anywhere and traditionally include logs, metrics, and traces; although more modern approaches tend to use additional and different outputs.
The Need for Observability
The reason behind the rise of observability as a hot topic in software development is the ongoing move to ever more distributed systems. Service oriented architectures, virtualization, microservices, and containers are all similar in that they have made software applications more distributed and thus more complex.
This is interesting because networks are inherently complex distributed systems. And that means that we should be really excited about observability too. It just requires that we think of our networks as systems, instead of as collections of links, devices, and functions.
Troubleshooting Your Network
When most of us remember the last time we had to troubleshoot a network problem, it’s very likely that the first thing we did was log into a network device. Step two is often running a bunch of show commands. And step three is logging into the next device, as indicated by what we learned from the first device.
While this is an example of network validation, it is also an example of low or poor observability. Using this approach makes it quite hard to answer our fundamental questions: Is there something wrong with the network? What is wrong with the network? And ultimately; what do I need to do to fix it?
Network Observability
To make a network more observable we need to regularly (automatically) pull data (outputs) into a central repository. Then, from this central location we need to be able to analyze all of the data at once and on-demand, whenever we need (or want) to. We should also store historical data so that we can ask questions about the past, compare current and historical state, or deal with change and trends more generally.
Luckily, we have an advantage in all of this over physical systems in that we can actually pull data about state directly from the devices and functions.
If you want to understand the state of a crane arm, for example, you must attach and calibrate a sensor to track and report its position. But if you want to understand the state of an interface, you can pull its status directly from the switch, router, firewall, etc. We don’t need to infer or estimate the internal state of the system, we can collect control plane state directly, making this data the most important to have when it comes to network observability.
Controllability
Finally, it is worth noting that: “In control theory, the observability and controllability of a linear system are mathematical duals.”
Which means that, at least according to mathematical theory, the more observable a system is, the more controllable it is - and the more controllable a system is, the more observable it is. We alluded to this a bit above, and will dive deeper into the concept in a future post, when we talk about intent.
For now we can just meditate on the words of a pioneer of observability in distributed systems, who said that “observability is about understanding performance from the perspective of the user.”
Beyond Observability
In the next installment of our “NetDevOps Primer” series, we’ll dive into the exciting world of network virtualization.
For those with less patience; everything in this series is available RIGHT NOW in the FullCtl whitepaper: The NetDevOps Primer.