MASTER’S DEGREE THESIS FOR THE
M
SC
. in COMPUTER SCIENCE AND
NETWORKING
Multi-instance SDN control of OpenFlow networks:
scalability and reliability
SUPERVISOR
CO-SUPERVISOR
Dr. Alessio Giorgetti
Dr. Piero Castoldi
By
Tajammul Mustafa
Academic Year 2020-2021
Pisa, Italy March 5, 2021
2 | P a g e
Abstract
Distributed systems play an increasingly important role in Software Defined Networks (SDN), where replicating network state information is essential to balance the controller load and provide resilience in case of failures. Such an example, if the traffic requests on the network exceed expected levels for a period, it is important that the SDN system continues to work properly seamlessly serving the incoming requests. For guaranteeing the required level of
scalability and reliability the open-source ONOS SDN controller adopted a logically centralized, but physically distributed architecture. Using this solution, a number of controller instances exploit the Java-based Atomix framework to share and synchronize the network state information. However, previous work demonstrated that under traffic requests overload the ONOS controller may show critical behavior resulting in the loss of requests to the network, and with the entire SDN network eventually crashing. This work first provides a literature overview on the topic of multi-instance SDN control of OpenFlow networks. Then, an accurate SDN environment is deployed with an emulated data plane based on OpenFlow virtual switches and a multi-instance ONOS controller. In this thesis some measurements have been performed to evaluate the scalability and the reliability provided by the system.
3 | P a g e
Acknowledgments
First and foremost, praises and thanks to God, the Almighty, for His blessings throughout my life and career.
I would like to thank all the people that have been involved in my thesis and study. I would express my sincere gratitude to my supervisor Dr. Alessio Giorgetti for the continuous support, patience and availability during my thesis work and writing.
I have learnt a lot during this master’s in computer science and Networking. I am delighted to have had all the support, dedication and knowledge from the administrative and teaching staff at the University of Pisa and Scuola Superiore Sant’Anna.
I would like to express my special thanks and gratitude to Professor Marco Danelutto for his guidance, assistance and availability during my master’s career.
From the bottom of my heart, I would thank my family, my parents and my brother Kashif Mustafa for their support, prayers and belief in me. They have been my motivation for the completion of this degree.
I acknowledge and extend my thanks to the Tosoni family. I was overwhelmed by their care and concern in helping me finish writing up this thesis.
4 | P a g e
Contents
Chapter 1 ... 6
Chapter 2 ... 10
2.1 Software defined networks ... 10
2.1.2 OpenFlow Architecture ... 14
2.1.3 SDN Applications ... 17
2.2 Why Multi-Instance ... 20
2.2.1 Controller Evolution ... 21
2.2.2 Multiple SDN Controller Use-cases ... 24
Chapter 3 ... 28
3.1 Research challenges ... 28
3.1.1 Multi controller Topology Designs ... 28
3.1.2 Multi controller Placement ... 31
3.1.3 Multi controller Synchronization ... 32
Chapter 4 ... 36 4.1 Northbound Applications ... 37 4.2 Southbound Applications ... 38 4.3 Distributed Core ... 38 4.3.1 Cluster Subsystem ... 40 4.3.2 Intent Subsystem ... 44 4.3.3 Application Subsystem ... 52
4.3 ONOS Multi controller Architecture ... 55
4.3.1 ONOS Pre-Owl Cluster Architecture... 56
4.3.2 ONOS Owl Architecture with Atomix 3 ... 59
Chapter 5 ... 63
5.1 Data Plane Device ... 63
5.2 Environment Configuration Setup ... 66
5.3 Create ONOS Application ... 75
5.4 Prepare Tests ... 76 Chapter 6 ... 84 Chapter 7 ... 102 7.1 Future work ... 103 Chapter 8 ... 105 8.1 List of Figures ... 105
5 | P a g e
6 | P a g e
Chapter 1
Introduction
Telecommunications today are as pervasive as they never had been in the past. The need of connectivity everywhere cannot be ignored with an ever-increasing number of devices that need internet access to share and retrieve data. The development in the internet, public and private network infrastructure have shown an escalating interest and movement towards software defined networking (SDN). SDN is a concept that promises to simplify and unify how networks are designed and controlled [1]. As seen in figure 1, SDN provides an abstraction that segregates the control from the data plane and introduces the concept of a logically centralized controller.
7 | P a g e
Figure 1 An SDN architecture contains 3 planes
This offers increased flexibility [18] in the network management, unifying the heterogeneous network technologies, and cutting down operational expenses (opex) [2-3]. Hence providing new and innovative possibilities to expand the networking solutions. For this reason, in chapter 2 we describe SDN in detail and its benefits in relation to traditional networks. We describe the SDN architecture and its applications in the real world followed by its implications and challenges [4],[5],[43]. Moreover, distributed systems and solutions are in big demand in the modern-day networking in software defined networks to guarantee the high availability of the SDN
controllers, in cloud and data center networks. For these networks to be expected to scale up while providing high availability assurances, essential data and the associated functionality need to be replicated and distributed in order to avoid a single point of failure [15] at the controller plane. Figure 2 shows multiple controllers managing the network devices, ensuring that the network remains robust even when one of the controllers fail.
8 | P a g e
Figure 2 Logically centralized & physically distributed controller
For this reason, we discuss the need of a multi controller setup to overcome the implications above. Also providing some examples where a distributed architecture can be beneficial. Chapter 3 describes the multi-controller architectures and its research challenges both open-source and commercial for controller placement [10], topology design [11] and controller synchronizations [16]. It also describes past works and proposals for the above concerns. To emphasize the importance of a distributed controller architecture in chapter 4 we describe the distributed core layer of ONOS. The Open Network Operating System is a leading open source SDN controller for building next-generation SDN networks. Designed to maintain high speed and large-scale networks with its main distinguishing characteristic to support hybrid networks. We describe the recent and past ONOS architectural designs for a multi-controller system. We talk about the ONOS subsystems. With the intent, application and cluster subsystems in detail that will be used in the analysis of this thesis. Chapter 5 describes the deployment of multi-controller architecture using the post owl version of ONOS nodes together with Atomix nodes. Moreover, we describe the development of an application for the Northbound that uses the intent subsystem to listen upon the intents submitted to the system and gather metrics like response time under different topology settings and constraints. The results will be used in chapter 6 to perform analysis for the various performance metrics of the system like scalability, reliability and availability of the system. We conclude the thesis in chapter 7 where we draw conclusions for the outcome and graphs, we obtained in the previous section.
9 | P a g e
Chapter 2
State of the Art
During these years, the features provided by traditional networks with in-band distributed control plane, rigid structure, legacy protocols are not anymore able to satisfy the continuous requests made by the new technologies and applications [7],[17]. In traditional networks [5], network devices would manage the data transfer and control functions. Keeping track of the state changes for several network devices and updating policies accordingly is much more challenging with an increase of the sophisticated policies through a constrained set of low-level configuration
commands on commodity networking hardware. This was prone to error [7] and often resulted into misconfigurations as the traffic conditions change need repeated manual interventions to reconfigure the network. New technologies needed network flexibility [18], programmability, network virtualization [19], [20], scalability [11] and reliability [78], fine grain and techniques that can respond quickly for changing requirements.
2.1 Software defined networks
Software defined networking [6]-[9] has developed as a technology trend. Generally, computer networks were managed with handling complications by the adding more protocols into the protocol suite. However, the advent of SDN created a huge hype in the network community due to the first ever introduction of the concept of modularity in networking [112]. Consequently, system components and heterogenous protocols were substituted with reusable abstractions.
10 | P a g e
Figure 3 In the SDN approach the control features are moved into a single and centralized network controller
The decoupling of the control functions from the forwarding functions as seen in figure 3, allows more automation and programmability in the network. SDN is often coupled with network function virtualization (NFV) [19], [20], which splits up network functions from hardware in the form of virtualized network functions (VNFs). SDN enables cloud-like computing in a network. It simplifies the network architecture by moving the control plane intelligence of L2 and L3 devices [43] into a centralized brain [7]. This allows for quicker updates to changes in business requirements due to the centralized control [22],[23].
11 | P a g e
As seen in figure 1 there are three main planes that can be used to describe the SDN architecture.
Application plane in SDN has a key role to manage the network applications. As SDN can be
implemented across a broad range of network environments giving the possibility of applications for routing, firewalls, policy services, and load balancing of network traffic etc. For instance, to provide load balancing [21] to a network, an application can properly route traffic uniformly across multiple channels. Deployment of a similar use-case in a traditional network is quite problematic [7] and resource consuming where the control plane of every individual device must be configured. With SDN in place the work is uplifted as the application plane offers a simple and easy solution to implement this across the network.
Control plane in SDN is decoupled from the data plane. The control plane has the entire
network view of its domain [25]. It uses a Network Operating System (NOS) which is
implemented in a logically centralized architecture to facilitate network management. The NOS can be physically centralized or distributed (to improve control plane scalability and reliability) [25],[26],[11] depending on the design aspects and the architecture. As seen in the layered SDN architecture in figure 1 the control plane traffic between data and application planes. Having access to the global network view it forwards flow rules to the data plane devices and the network state information to the applications in the application plane. The SDN controller consists of a set of modules each performing a task. Gathers network statistics and performs discovery of network devices. For more advanced feature support, extensions can be added. All SDN controllers provide general services [94] including topology manager, stats manager, routing module, device manager, etc. that have very differentiated properties.
Data plane denotes the forwarding network devices (switch, router, load balancer, etc.). Since
the routing control is decoupled from these devices, SDN uses only programmable devices. The devices have flow tables, a secure channel for communication with the control plane, and a protocol. Devices may have one or more flow tables. The devices can connect to the control plane using the secure channel and the OpenFlow protocol [28],[29],[30],[31]. Flow tables consists of flow entries that holds information about the match, actions and counters. Each header of an incoming packet is matched with the flow entries and an action is performed
12 | P a g e
updating the counter. Instructions are then sent back and installed on the devices by the control plane using the southbound interfaces [6].
The northbound interfaces are mainly composed of open-source application programming interfaces (APIs) [6]. They are usually REST APIs. Applications that use SDN need available resources and network state information and thereby depend upon the controller providing them across the northbound interfaces. Moreover, the controller receives policies from network administrators [32],[33] and ensures the routing of the traffic. The applications request the controller for information when it lacks. The controller handles how these requests are processed with the needed information. As the controller has the view of the state it finds out the optimal path for the application in terms of latency and security measures. However, the northbound APIs implementation is yet not standardized and solely depends on the controller
implementation.
The southbound interfaces are mainly composed of protocols that allow the controller to communicate and route policies with the data plane including switches and routers. The policies include which routes the application data must take as decided by the controller. Hence,
controlling how the routers and switches move the data in the network. The controller makes the informed choices that optimizes the route taken for the data rather than having the devices and routing tables to determine.
Various data programming and configuration protocols are available for the flow of traffic across networks. Examples are OpenFlow, Netconf, SNMP etc. that assist with communication of a variety of network devices. [91] lists other examples. Open Network Operating System (ONOS) which is similar to other controllers like OpenDaylight [34], uses the idea of pluggable providers for each southbound protocol having some protocol specific behavior or requirement for its controller platform. Pluggable means that we can develop our own device protocol-specific provider and register it with the controller.
We will discuss OpenFlow as it is a commonly used API and considered as the standard southbound interface [91].
13 | P a g e
2.1.2 OpenFlow Architecture
OpenFlow is “a communications protocol that provides an abstraction of the forwarding plane of a switch or router in the network” [34]. OpenFlow is thought to become the standard-de-facto in SDN. It allows access and manipulation of the data plane. Proving external applications with simple primitives to use to program the data. The SDN control software programs static and dynamic match rules for the network traffic. An OpenFlow switch has multiple flow and group tables. A flow table consists of many flow entries. Figure 4 shows the structure of a flow entry inside the device’s flow table. A flow entry can be considered as a routing table in traditional networking architecture, but it provides much more functionalities with respect to packet
forwarding (e.g., packet fields modifications, packet duplication, checking of the interface status, etc.).
Figure 4 Structure of a flow entry
OpenFlow has two operational modes a reactive and proactive mode. Figure 5 shows describes an example for the reactive mode considering an example in which h1 is trying to ping h2.
14 | P a g e
Figure 5 Reactive flow for h1 request a ping to h2
When the ping command is started at h1, it generates an ARP request (step 1). The treatment of the first packet is not known to S3 and therefore, the packet is sent to the controller encapsulated within an OpenFlow PacketIn message (step 2). Controller learns the details of the host h1 (step 3). The controller then replies with an OpenFlow PacketOut message for the flooding of the ARP request (step 4). The ARP request is flooded at port 3 and port 2 at S3 and hence avoiding port 1 (step 5). This ensures no ARP looping. S2 and S3 consequently, send in a PacketIn message to the controller (step 6). The controller replies S2 with PacketOut message for the flooding of the ARP request. Also sending a PacketOut message for a drop of the packet at S1. Hence the packet will not proceed from S1 (step 7). S2 will broadcast the ARP request to h2 and h3.
15 | P a g e
Figure 6 Reactive flow for h2 reply to the ping request from h1
Figure 6 shows the process for the reply from h2 to h1. Upon arrival of the ARP request, h2 replies with an ARP reply (step 1). It also stores the IP/Mac details for h1 in its ARP table (step 2). S2 forwards this ARP reply to the controller within a PacketIn message (step 3). The
controller learns the details of h2 (step 4), uses a PacketOut message to forward the ARP reply toward S3, and uses FlowMod message to write rules in S2 (step 5). The ARP reply reaches S3 (step 6). The switch then forwards the reply to the controller in a PacketIn to write the rules in S3 (step 8) like step 5 for S2. The ARP finally reaches h1 (step 9), that learns the IP/Mac details of h2 (step 10). Now as the path is recorded, and rules installed on the switches on the path from h1 to h2. When h1 sends the first ICMP request the switches already know the treatment for the packet (i.e., using the flow rules installed in steps 5) without having to query the controller again. The second approach is the proactive approach, the controller installs the flow entries in the flow table a priori to the packet arrivals.
16 | P a g e
Figure 7 Proactive flow to pre-install flow entries
As seen in figure 7 we pre-defining all our flows and actions ahead of time in the flow tables of the switches S1, S2 and S3 (step 1,2 & 3), the PacketIn event never occurs. The key is the selection which rules need to be installed in maximizing network performance. This is important in large-scale networks such as data centers. When a packet arrives at the device it is matched against flow entries in the flow table. The controller is only queried if a match is not found. The idea behind this approach is to reduce the overhead incurred by communication between
switches and controllers [36].
Given the power and ability of SDN to manage and control networks we will discuss some of its real applications.
17 | P a g e
Network features in traditional networks are typically implemented in a distributed way
exploiting specific network protocols (e.g., the ARP protocol is devoted to mapping IP and MAC addresses). The same features, in an SDN network, can be implemented as a network application installed at the controller. For this reason, the typical architecture of SDN controller as seen in figure 8 is strongly modular, i.e., to allow network users to develop its own application to be deployed in the network. With this vision many of the traditional network features can be realized as an SDN application that can be dynamically activated and configured at the controller. Some examples of applications normally supported by SDN controllers are: ARP proxy, DHCP server, Firewalls, Network slicing, etc.
Figure 8 Modular controller running multiple applications
A less simple example of SDN application can use bandwidth management to ensure the end users to receive online video watching and optimal browsing experiences that can also monitor
18 | P a g e
the bandwidth requirements then provision user flows to match the latency and bandwidth requirements. SDN applications will be able to provision flows in the network depending on the availability and types of the content which is built to handle content. The delivery of rich
applications like graphics design software, engineering, GIS and CAD is allowed by
virtualization. These workloads traditionally needed bare-metal architectures with their own connections. With virtualization, VDI can help in creating powerful desktop experiences and applications. The integration of SDN into the application control at the network layer offers functions like segmenting heavy traffic, securing confidential data, creating powerful QoS policies, and even creating threshold alerts around bottlenecks within SDN will help to support rich and high-performance applications. SDN Applications can allow easier movement of data between cloud locations by abstracting the critical network controls. Network as a service (NaaS) [37] can manage traffic for cloud computing services in a proficient and flexible manner.
Operators can control network metrics like bandwidth, routing, and QoS requirements of their data. When coupled with SDN, the current NaaS initiatives can be improved to allow operators to build their own SDN infrastructure [39].
The scalability and complexity of Data center networks (DCN) [38] are likely to approach the limit of traditional networking infrastructure and IT operations [17]. The SDN security policy applications [27] benefit from the network segments being centrally configured, that provides an end-to-end visibility of the network. Thereby tightening up the DCN in terms of security,
infrastructure, its power consumption, and its various metrics.
SDN also virtualizes hardware and services, resulting in benefits of a reduced capex [2] & potentially limits the need to purchase purpose-built, ASIC-based networking hardware. Instead, it provides scaling solutions and pay-as-you-grow models [40]. Today most vendors offer SDN and OpenFlow supported devices [41],[42]. Any network infrastructure that has SDN supported switches, can be activated to use the SDN capabilities by simple activation options. SDN impacts the opex [2] of an infrastructure. Changes to business requirements can easily be automated and integrated as updates to the network’s software. Hence there is no need to replace and remove parts of the whole infrastructure [21].
19 | P a g e
2.2 Why Multi-Instance
SDN is a promising solution for the IT, service and cloud providers and business enterprises [2]. However, it encounters challenges that impact its performance and implementation in various areas. Especially in the cloud and the wireless networks. Mostly related with the control plane scalability [11], reliability [78] and security [43]. We will discuss past research for these
challenges in the next part providing use-cases. The main drawback of this SDN single controller approach is the reliability and scalability of the control plane having a single controller. A single SDN controller does not scale and is often working at full capacity, accumulating processes. Hence there is a high latency because of a bottleneck at the controller. A seen in figure 9 this is an expected behavior seen in devices when the rate of incoming requests (1 / ∑Tn) is higher than
the rate of service rate (Tid controller) offered by the controller when processing the requests.
Figure 9 Bottleneck at a single controller
A centralized architecture has the risk of a single point of failure (SPOF). The controller may experience a hardware or software failure. A server failure in a production network or a cluster is referred as controller failure [45]. As the software that manages the devices runs on the
20 | P a g e
controller, a failure will cause disconnection and no control between the controller and the managed devices affecting the network resilience.
A distributed architecture for the controller [15] is a solution to counter the service load in a centralized design when there is a high number of requests from the devices. Deploying multiple controllers as a backup, peer or replication [46] resolves the SPOF problem. Each controller is expected to manage a segment of the incoming workload. Thereby the network traffic (or flows) will be distributed through several controllers according to some topological model [11].
Additionally, this design provides redundancy to the network [25]. A controller can fail anytime and, abandon the network without its control plane [25]. All controllers except the last can fail providing resilient high availability to the network [27].
However, there are other issues including latency because of state synchronization of the controller [16],[11],[47],[48], number of controllers needed [49], controllers Placement [10], security issues [20] and policy consistency between controllers [11].
We will discuss the origin of multi-controllers in the next section with 2 scenarios to demonstrate some of the issues mentioned above.
2.2.1 Controller Evolution
In figure 10, a single controller manages the entire network of four switches. This is similar to the initial SDN design. We will use the terms device and switches interchangeably. If the
aforementioned reactive behavior is applied, when the source host intends to forward a packet to the destination host, it sends a new packet to its connected switch S1. The switch lacks
information for the forwarding function due to the no prior knowledge of the flow rules for this new packet. S1 sends a Packet-in message to the controller to get the needed flow rules. If a dedicated application is running at the controller for reactive routing of traffic, the Controller 1 sends back the Packet-out message [50] and one or more flow rules to be installed on the device. S1 can now forward the packet to the next device S2. The next switch in the path performs the same operation of contacting the controller 1. The packet eventually reaches its destination host
21 | P a g e
without disruptions. The controller as seen has the main role in this process of packet
forwarding. As these new incoming network traffic increases at a high rate than being processed, as in case of several hosts sending recursive requests or new devices joining the network, the single controller cannot serve up with this great number of flow requests due to its limited controller capacity [51].
Figure 10 Single controller managing routing traffic
When, as the single controller fails to process the requests as seen in figure 11 the switches cannot perform the routing for a newly arrived packet.
22 | P a g e
Figure 11 Failure of a single SDN controller
This impacts the communication between the devices and the applications of the network. In figure 12, we use 2 controllers for the same network topology. Each controller controls a part of the network.
23 | P a g e
Figure 12 Multi-controller managing routing traffic
This new configuration solves the problem faced in the previous scenario [43]. Both controllers, controller 1 and controller 2 are logically centralized by sharing the same logic and synchronize themselves by sharing the state. When packets arrive at S1, both controllers can directly install the forwarding rules in their managed switches. Both controllers working together increases the effective service rate of the controller plane to process flows. Importantly, this solves the single point of failure in the previous scenario as controllers act as a backup for one another. As the control plane will be still available in case of a failure of one of the controllers, this increases the reliability of the controller service. A single switch can connect to a single master controller and several standby controllers [52].
Network and service providers aim to improve the network controller performance. But fulfilling the ever-increasing demands is a challenging task. As an example, Ryu [53], a legacy controller, can process approximately 6000/sec packet-in requests with an average latency of 6ms. This is not sufficient especially during peak hours due to the limited capacity of the single controller [51].
Summarizing the above, we can understand that a multi-controller design is on demand that can improve the performance of SDN networks. Hence, multi-controller designs are a popular research area in the recent years. It is obvious to design a multicontroller architecture as the control plane [12], [13] to meet huge demands of large-scale networks.
2.2.2 Multiple SDN Controller Use-cases
The SDN cluster design apart from addressing problems of single SDN also provides efficient topology management and corresponding path/route computations, consistent data stores (data that is shared across controller instances) [59], and efficient northbound APIs [7]. Here we provide some use-cases and works that use a multiple SDN controller solution.
24 | P a g e
Multitenant Transport Networks using optical network virtualization and network function virtualization (NFV) [19], [20] for on-demand OpenFlow controlled virtual optical networks (VON) [54] where each tenant is a SDN controller that runs in the cloud controlling a single VON.
A multi-controller deployment algorithm proposed in [16] guarantees reduced request delay, load balancing, and reliability in hierarchical control architecture for a SDN in a wide-area network (SDWAN).
Coordination between optical and IP/MPLS networks can be achieved by deploying multiple SDN controllers for efficient interactions to provide multilayer optimization and intelligent traffic engineering [55]. In addition to provide end-to-end provisioning and resource
management of multi-layered networks this scheme will also offer restorations and optimizations in case of failures.
A recent research demonstrates runtime provisioning of data connectivity services and advanced automatic failure recovery at the control and data plane levels. Using ONOS in a multi-controller environment to manage the optical network elements (Transponders and Reconfigurable Optical Add/Drop Multiplexers-ROADMs), as seen in figure 13 under normal and failure conditions [56]
25 | P a g e
In Transport SDN [57] is an area that can benefit from SDN multi-controllers. As it employs several technologies, such as IP/MPLS, Optical, etc. The controllers will manage switches in the data centers, aggregation switches, and the optical switches.
A similar use case is the multi-domain environments like data center, provider network, internet exchange points and mobile backhaul etc. as seen in figure 14. Some real-world implementations use domains such as metro, cloud, and core networks, where service solutions bridge these multiple domains. These networks interconnect data centers, enterprise networks, mobile entities and customer sites. These virtual or physical networks are usually distributed across separate administrative and geographical zones. Multiple SDN-controller deployment developments can target end-to-end service provisioning and optimization [62] of such multiple domains [61]. Multiple controller solutions boost performance, reliability, scalability and provide efficient end-to-end services
26 | P a g e
An SDN-based underwater acoustic sensor networks (UASN) framework is proposed [58], that uses SDN in a clustering design that speed up computations like learning automata and degree-constrained connected dominating sets.
Several commercial vendors of SDN controllers, such as NEC, Fujitsu, Cisco, BigSwitch, insist to deploy multi controllers [60] to achieve either availability or scalability or sometimes for its services.
There are several other examples that make multi-controller solutions a popular research for the recent years [15].
27 | P a g e
Chapter 3
Multi controller architecture
In this section we will describe some challenges and past works in relation with multicontroller SDN.
3.1 Research challenges
The multicontroller design partitions the network flow across controllers given a topological model [11]. When a controller fails, other controllers in the cluster continue to push flows to the devices.
However, several challenges in building distributed control platform are the scalability, reliability and performance which makes way to research about topics like consistency among controllers [63] namely controller placement, controller topology, etc.
3.1.1 Multi controller Topology Designs
Multi-controller architecture topologies are based on different characteristics as illustrated in [25]. We will discuss two basic topology designs namely hierarchical topology and the flat topology [11], [12-14],[15].
28 | P a g e
In a hierarchical topology, there are multiple management levels. The level consists of a group of controllers, where every controller is either responsible for managing its own domain or the set of controllers in the immediate underneath layer (child controllers). The controllers in every level get management instructions from the controllers that are logically placed in the immediate top level (parent controllers). The number of logically considered levels depends on the size of the network topology, the number of flow processing requests and the collected network status information. Figure 15 shows the layered or the hierarchical architecture with a centralized root controller also known as the super root controller that manages the global view of the network topology and is responsible for sending its controllers with the needed information to adaptively and dynamically respond to topology changes based on the collected network status information at the higher levels. The child controllers in the last layer are only responsible for managing the switches in their own domain and report to their connected parents in the higher layer. In
hierarchical multi-controller architecture, controllers in the same layer do not communicate with each other. The communication is made possible via the root controller if needed. The
hierarchical model has proven to improve the performance of SDN-enabled networks [64]. But as there is a single root node controller the single point of failure still persists leading to a failure.
29 | P a g e
Flat Topology
The flat topology, however, does not have the leveled architecture. In this model, refer figure 16, the network topology is structured into a number of domains or segments, each managed by a single controller. Although the topology does not need a root controller in the higher level to globally manage the domain controllers, this makes the management between the controllers difficult with extra control overloads [15] to share the same view regarding network status
information, scalability issues, topology failure and potential changes in domain formation which is considered one of the drawbacks of this topology. Controllers use peer-to-peer communication to share information. However, there is no adequate standard for this between different
controllers [63]. This topology, however, has a rapid installation and configuration of additional domains, including the connectivity links between controllers. It also provides more resilience to failures. ONOS [65], HyperFlow [12] and Onix [14] are examples of a flat controller topology.
Figure 16 shows the flat topology of multi-controller architecture
These two topology designs are typically determined with a few key business and technical constraints like scalability [11] and reliability, performance improvement [66], multiple
30 | P a g e
administrative domains and their interactions, corresponding fault management and multiple network layers [16].
3.1.2 Multi controller Placement
The number of controllers deployed and its placement in the distributed network architecture has been a quite active research topic, indeed it considerably affects the overall performance like delays, fault tolerance, resiliency, etc. of the control plane [11]. There are various proposals for distributed architectures, how they interact, how the information is shared. All these techniques for multicontroller deployments can be divided into two groups of research.
The first set of research deploy controllers in the network according to some network parameters like delay, traffic, distance etc. The authors in [67] propose some Fully Polynomial Time
Approximation Schemes (FPTAS) for the centralized controller to achieve improvements in network utilization and to reduce packet losses and delays. [106] suggest a zero-sum game model between the controllers and devices by implementing elastic control. It also implements a
decentralized model of game-based decision. [13] introduces a K-center method, that considers the load of the controllers. It implements clustering to place the controllers with the goal to reduce the latency between the controllers and devices. [68] discusses the dynamic Controller Placement Problem (CPP). It determines the number of controllers [49] to support dynamic load and locations for the controllers to guarantee some communication latency. A mathematical model is introduced in [69] for the controller placement. The model computes the optimal number, locations, types of controllers and the interconnections between all the network components. [14] describes the controller deployment problem. It considers the average and maximum delay in the network. With this it builds a corresponding mathematical model and determines the optimum deployment states of the controllers. [15] proposes a load variance-based synchronization (LVS) method. That improves the performance for load balancing in a multi-domain network. If the load of a specific server or subdomain exceeds a certain threshold the LVS conducts state synchronization between controllers. [70], however includes the hop count and delay to dynamically explore the positions and number of controllers needed
31 | P a g e
depending on the network conditions. [71] designs the ElastiCon that shifts the controller load when certain threshold values reached.
The second set of research deploys the controllers according to the reliability of the network. The authors in [10] try to maximize the reliability of the SDN control plane. For the control path it proposes a new metric, called expected percentage that characterizes reliability. Also, a NP-hard problem is demonstrated while examining several placement algorithms to solve the problem. The Pareto-based Optimal Controller placement is discussed in [72] that proposes pareto optimal placements for controller failure, isolated node, load balancing and inter-controller latency. The availability of the model is discussed in detail using numerous real-world topologies. Survivor is proposed in [73], that studies path diversity, capacity and failover mechanisms as key aspects during controller failover. [74] reduces the average response time of the control plane by introducing the Dynamic Controller Assignment (DCA). DCA is similar to a stable matching problem that incorporates key notions from coalitional games and matching theory. [75] enhances the network resiliency using an elastic mechanism for controller placement and control-traffic routing. A protected switch acts as a backup interface to reconnect the controller when the output interface fails. The Reliable and Load balance-aware Multi-controller
Deployment (RLMD) is introduced in [76] that sets up the devices and controllers according to the controller load balancing rate node attractability. [77] estimates the reaction time at the switches due to the inter-controller communication. The authors use the single data-ownership (SDO) and the multiple data-ownership (MDO) models with the controller traffic exchanged, together with the OpenFlow control traffic exchanged between controllers and devices. The works concluded that larger switch-to-controller delays with respect to the minimum ones are well compensated by much smaller controller-to-controller delays. Lastly the works of [114] present DynRaft a dynamic add-on to Raft implementations, which continues to satisfy the formally proven strong consistency properties of Raft under overload conditions.
32 | P a g e
Distributed controller architecture greatly impacts network reliability [78]. But it is important that the memory synchronization must be maintained between the multiple controllers to avoid any inconsistent network topology state. As [52] suggests some attributes, a good network topology state management solution to have in a distributed scheme.
Completeness describes that even though the controllers in the cluster have direct visibility and
control over a subset or domain of the network, they should work as a monolithic system that ensures that the network topology view of each controller is in sync with the entire network.
Accuracy suggests that each controller’s network view should always show the correct state for
it various network elements. Because devices, links and ports turn on and off. This also
concludes that the network view of the controllers should respond quickly and spontaneously to changes in the underlying network.
Low latency access proposes a low latency access to the network view. The network topology
view is a frequently consumed part of the state and a proper mechanism needs to be chosen that reduces the latency.
As an example, figure 17 shows a network managed by 2 controllers. The state information local to a controller is distributed across the cluster as network events through the connected link so to achieve synchronization of the distributed states. If the peer-to-peer communication is abrupted between the controllers 1 and 2 as seen, it results in a control network partition.
33 | P a g e
Figure 17 shows inconsistent states between domains
If the topology in the domain of controller 1 changes such as a new device joining or leaving the update cannot be propagated to controller 2. Controller 2 will take any routing decisions based on its old topology view of the domain under controller 1. Leading to unexpected behaviors like routing loops or network packet drops.
As the logical consistency problem [63] suggests, the old rules in the SDN flow table must be flushed with the new set of rules. Input packets must be processed and routed correctly to its destination using the new fresh rules on each device. Use of stale and fresh flow rules at the same time must be avoided. There is a higher probability of inconsistency in the logical state when the controllers are physical dispersed. This leads to more errors in packet handling and a higher probability of network failure. Domain information that includes controller reachability information, consistency protocols, network topology and flow setup coordination [7] must be communicated across the controllers to maintain a consistent global view.
Several consistency models have been specified [79]. Here we will describe two data consistency models deployed in ONOS SDN networks.
34 | P a g e
Strong Consistency model
The model guarantees that the controllers in the network always process the most updated data. Ensuring that any consecutive operation that updates the internal state of the controller is distributed and acknowledged by all other controllers. Providing some blocking mechanism for subsequent transactions that need data that is yet not been across all or by most of the controllers. A strong consistency model constrains a large distributed into a monolithic system [107]
allowing a more consistent view of the network at the cost of introducing more communication overhead and latency. Hence impacting the performance of the controller and eventually
reducing the control plane availability [108]. The RAFT consensus protocol, implemented by the Atomix is a distributed, persistent, in memory framework that was integrated with the ONOS framework since ONOS 1.4 supports this consistency model. We will discuss the Raft protocol in the next section.
Eventual Consistency model
This model offers a weak form of consistency in the network. Any data update local to a controller will be asynchronously updated across all the controllers. The updates flow-out asynchronously from one controller to the next making it a non-blocking process. This model avoids incurring additional delays in SDN networks [107]. However, due to the lack of
constrained state updates the model may offer possible conflicting updates. This model is useful for data structures that offer high availability. The ONOS framework since 1.4 includes eventual consistency with an anti-entropy protocol based on a gossip algorithm where each controller periodically at random chooses another controller in the cluster and compares it state replica. Synchronization messages are exchanged and the stores across the controllers are updated with a timestamp. This ensures that all controllers in the cluster achieve consensus to an eventually consistent model.
35 | P a g e
Chapter 4
The ONOS Controller
The section will discuss the distributed architectural details of ONOS and its modular
subsystems that provide a scalable and a reliable control platform [8]. That offers policy-driven network programmability and as it is operator-friendly [92], it targets service provider networks business needs. ONOS is a multi-module architecture with modules that are managed as OSGi bundles. That is designed to provide some features [86].
Code Modularity allows new functionalities to be introduced as self-contained units. Configurability makes it easy to load and unload features, at startup or at runtime.
Separation of Concern is an important concept that facilitates modularity by maintaining clear
separations between the subsystems.
Protocol agnosticism adds independency to its applications to any specific protocol libraries or
implementations.
A control plane that offers the above key features solves problems faced by distributed systems to offer high availability, scalability and performance for the service providers. ONOS allows compatibility of multiple protocols at its southbound interface to connect several network
devices. It exposes the correct REST APIs at its northbound interface to support service provider requirements, use cases and easy application development. Figure 18 shows the ONOS
36 | P a g e
Figure 18 Architecture of ONOS [B]
The ONOS distributed architecture is a three-layered structure comprising the application, core and southbound protocol layers. The components available in the layers build up the subsystem where each subsystem is responsible for a service. Some examples are the Cluster, Device, Packet, FlowRule, Intent, Link, Host and Path subsystems. We will describe the southbound northbound layer briefly and a detailed description for the distributed core.
4.1 Northbound Applications
ONOS supports applications for control, configuration and management of the network. Some applications come in stack with the ONOS package provided by the Open Networking Lab (ONF) [42]. Including applications for segment routing, multi-layer SDN control, topology viewer, path computation and SDN-IP peering [87],[88],[89].
37 | P a g e
ONOS and other controllers like OpenDaylight [34], and Beacon [90] offer application functions at the core-layer simply by sending and receiving command request and response. The core layer provides the AdminService and Service interface for each service module in the core. The
applications can extend it with the needed information the service module provides. All application names are registered with the CoreService, that sequentially assigns a unique
ApplicationId to the applications. ONOS uses the ApplicationId associated with the application
to keep track of the tasks and objectives, such as intents and flow rules.
4.2 Southbound Applications
An ONOS supports different control pluggable protocols at its southbound like Netconf, Openflow, SNMP, etc. [91] for the communication between network devices. ONOS uses the concept of providers as discussed in chapter 2 that encapsulates the protocol complexity. The providers provide the core layer with the necessary descriptions of the network elements. Like the ApplicationId at the northbound interface, ONOS uses the concept of a unique identification
ProviderID, which is assigned to each provider that we plug in the southbound interface. This
keeps track of proper mapping with the network devices.
4.3 Distributed Core
ONOS is architected with a tier of functionalities [92]. A unit of functionality is a service that is composed of multiple components also known as the subsystem [52]. Each subsystem is
responsible for an aspect of the network state like topology, host tracking, packet intercept, flow programming information, etc. Every subsystem maintains its own service abstraction, where the implementation is responsible for propagating the state around the cluster.
What is distinctive about ONOS is the set of maps it defines. Which means the semantics of the keys it stores, and the types of the values associated with those keys. It is this data model that makes up the Network. ONOS uses Atomix as its store, to provide a rich set of programming
38 | P a g e
primitives and distributed tables (maps) that make it simple for the control applications to access. The store itself is like the design of a modern cloud service. As it is distributed across the nodes it scales across.
Majority of the ONOS features are architected in a modular fashion which is mostly independent of each other’s internal implementation by exposing sufficient APIs to allow the dependent modules to work with.
Figure 19 ONOS subsystem [C]
Figure 19 shows the ONOS architecture as divided into various services and subsystems. The three most important services used in this project are the cluster, intent and the application subsystems.
39 | P a g e
The ONOS core is composed of various services that interact with each other. As an example, the Device service will use the Mastership service to check the mastership of a device service. Also, it will use the Communication service to share information and other services like the
Cluster service. Each service as seen in figure 19 has its own store to manage its contents. The
store would implement its own distributed mechanism according to a consistency model which can be strongly consistent, eventually consistent or both [79]. The mastership, host, network configuration, security stores are examples of data structures that implement the strong
consistency models using the Raft protocol. Whereas the intent, flow and the network topology stores adopt the eventual consistency models using the anti-entropy protocols like the gossip protocol.
The store as we described is distributed across the nodes in the cluster. Each store only persists and synchronizes its related states of the subsystem. As an example, the ApplicationStore has the knowledge about the state of the applications and no knowledge about how host or link state is tracked.
Each controller in a cluster has a unique NodeId [52] and manages the devices in its domain. The controller will update any changes across the distributed data stores. Figure 20 shows an ONOS cluster topology. According to the node mastership lifecycle all nodes by default start off with the NONE role. If a node learns about a directly connected device without a MASTER, it becomes the MASTER of that device and other subsequent nodes on which are directly connected become STANDBY controllers for that device increasing the availability. The mastership
subsystem in ONOS guarantees that at any instant in time, each device has exactly one MASTER that has full control read/write control on the switch forwarding tables. The mastership store manages a mastership term that closely tracks which controller is the master of a given switch at any instant of time. Each time the device comes online, the master is elected for the device the value is set to 1. Now every time the device changes mastership the term number is increased. We will use the term node and controller interchangeably.
40 | P a g e
Figure 20 steady-state for devices connected to master and backup controllers
All ONOS nodes in the cluster have the entire global network topology state cached in-memory. The global network view is constructed with the network topology and new local updates such as switch, port, link and host information that are discovered by every node.
The current master maintains an event number for each device that tracks all the topology events such as which switch is connected, port online and offline status, link down, etc. When a new topology event is detected this number is increased. The node first creates a new logical timestamp for this event using the event number as unique tuple (device_Id, mastership_term,
event _number).
As seen in figure 21 the master node receives a topology event E from the device it manages. The event is applied the local topology state machine of the master, and then in parallel, also broadcasted with a time stamp to all other controller nodes in the cluster using a specific TCP port (9876) for inter-controller interaction.
41 | P a g e
Figure 21 Topology event send and broadcast
When this topology event is received by the recipient nodes, they first check their local state if they already know of a much recent update by comparing the time stamps of the most recent event it knows with the one it has received. The local topology state machine is updated by applying the received event. If there are any stale events, they are simply dropped
The event logical timestamps ensure that the topology state machine updates in the correct direction by integrating only newer information. It allows the complete total ordering of the topology events emitted from the same device. Ordering is first done based on the mastership term. If the mastership terms are the same, we compare the event numbers.
The controllers monitor other cluster nodes by sending keep-alive messages to one another in the topology. Figure 22 demonstrates that the master node is temporarily disconnected from the
42 | P a g e
network. ONOS elects a new master for the device. The new master can continue to monitor the state of the network and program the devices under its control.
Figure 22 A failing controller in a cluster
However, there are some concerns with maintaining the global network view consistency over the time. The disconnected controller does not receive any updates from its peers in the network. Also, at any instant some controller instances can be more updated than others. Additionally, even at normal conditions there is no assurance that broadcasted messages will be received by all the controllers. These dropped messages need to be handled. There is a need to update them as there are applications running on the instances that that have access to this in-memory local network view and take decisions based on it. Such differences or state disorder across the nodes is called “entropy”. ONOS uses a lightweight gossip style mechanism [79]. A controller on randomly selects another node and sends the set of network device_Id(s) with the associated timestamps. Interestingly the entire network topology is not sent across. On the receiver side the node looks at the associated timestamps and sees if it has events that are more recent than the ones received. If this is the case, it sends these recent events to the sender controller to update its
43 | P a g e
local state. Instead, if the entries are stale or if it does not have the entries it requests the sender node for them. Thus, the discrepancy in the network views is removed. Also, if there is a new instance joining which has no information about the network looks like. So, the first anti-entropy interaction it has with other node is possibly up to date, the node updates its view of the network. So, without the need of any new bootstrapping mechanism the new nodes can join the network and in sync with other controllers’ network view.
4.3.2 Intent Subsystem
The intent subsystem is a policy-driven programmatic framework that is a black-box for users to specify what they need without knowing how it processed on the underlying network [44]. Hence providing the users with a high-level abstraction of network complexity. Figure 23 shows an example of the intent pathway on a network topology.
44 | P a g e
At a high-level an intent is an immutable object with a set of flow rules that are identified by the
ApplicationId of the application that submitted it, and a unique IntentId, generated at its creation.
At low levels, however intents can be described with the following terms [93]
Network Resource is a set of data models that are affected by an intent. The links in the network
are affected when intents are configured on the network.
Constraints such as bandwidth, optical frequency, and link type that are applied to the network
resources.
Instructions are actions applied to a portion of the network like header field modifications, or
outputting traffic through specific ports.
Criteria are header fields or patterns that also can be applied to a portion of traffic.
The ONOS intent framework, provides different types of intents that address different use-cases [94]
Host to Host Intents provides connectivity between two hosts in the network.
Point to Point Intents provides connectivity between any given points in the network.
Single-point to multi-Single-point or multi-Single-point to a single-Single-point for connectivity. They offer a fine control over the network resources compared to host to host intents.
Optical Path Intents allows connectivity across an optical domain
Protected Intents allows connectivity not only on primary path but also provides a backup path
across the network if the primary path fails.
We will describe how we can specify intent specifications that will be sent to the controller to be translated using the intent compilation service.
45 | P a g e
ONOS offers us REST APIs and CLI commands to submit intents. Moreover, we can create our own application and install it for this purpose. The intents as we know will be submitted to the running controller that will compiled and translated into specific flow rules for the network. We will discuss these methods below.
To install an intent with an application we use the IntentService that defines our intent. First, we request the CoreService for an ApplicationId in the activate method. We should use
the ApplicationId to construct the intents we need. The IntentService provides our application a reference that will automatically be available when our application is loaded.
For a host to host we will create intents between the available hosts in the network. We use the builder pattern below to add two hosts. The one() and two() will call the addHost() module. We can then submit the intent using the IntentService.
Figure 24 Builder pattern to create a host-to-host intent
In figure 24 we use line 1-7 will create and submit our intent as a part of an ONOS application. To install intents using the ONOS CLI we will issue the following
46 | P a g e
Figure 25 Host to host intent path
For the host-to-host intent as seen in figure 25 the controller performs all the heavy lifting. It finds the shortest path from the topology database. Figuring out which switches need new flow rules. Those flow rules are sent down to the switches on the shortest through the southbound interface. We can issue the command via the CLI to see the installed flows
47 | P a g e
Figure 26 Intent as flow rules
It shows a top-level listing of the intents given to the controller. All intents are displayed in tabular form. We can see the flow rules for the intent compiled as the input and the output ports. An intent can be removed by specifying
Yet another way we can use to install intents is to use the ONOS north-bound REST API to interact with the application subsystem directly. This is a standard POST request made to any endpoint with the headers, endpoint and data. The data object here contains the object with the
appId (application Id), the one (source host Id), two (destination host Id) we can submit an intent
48 | P a g e
Figure 27 Post host-to-host intents using the GUI
Note that for host-to-host intent request, we sent only a single intent for each switch as the intents are bi-directional. However, for point-to-point intents this is not true. We need to install intents in both directions. Also, the controller does not select the ports or the routes as in the latter, but it only handles the translation of the intents into flow rule and southbound and sends down to the switches.
49 | P a g e
Figure 28 Intent transition state machine [E]
An intent goes through a set of states inside the ONOS core as seen in figure 28 until being installed and converted into flow rules [93]. The orange states are the transitional states where the intents spent a short amount of time. Then there are the parking states in blue where the intents spent some time. A submitted intent is forwarded asynchronously to the compiling state,
installing state before it falls into the installed state. Intents no longer needed by the application
will be withdrawn. Common scenarios can be that an application requesting for a resource cannot be provided and hence the compiling fails. If later the resource is available, the intent can move back into the compiling state. Other cases can be the network losing connectivity and the successfully compiled or installed intent will attempted to be recompiled. To keep track of the failed intent there are registered network resources
50 | P a g e
recompilation, however, is a multi-threading process that discards stale, failed intents, and computes a new entire path from source to destination generating new installable intents.
Figure 29 Example: Host to Host intent compilation steps.
Figure 29 shows the compilation and installation of an intent between two hosts. The
IntentCompiler compiles the intents into installable intents. The IntentInstaller converts these
installable intents into flow rules called FlowRuleBatchOperations [110]. Finally, the
OpenFlowRuleProvider converts these FlowRuleBatchOperations into OFFlowMod
protocol-specific messages to be sent out to the correct.
51 | P a g e
All controller instances are software-wise identical giving ONOS its symmetric nature. The ONOS platform includes an application management subsystem [95] to manage the network control applications in a distributed context. Network applications are developed independently from the ONOS core system, similarly to desktop and server applications been developed independently from the kernel and operating system. Hence allowing its ecosystem of
applications to expand. The subsystem makes use of an eventually consistent map [96] and inter-node communication mechanism to synchronize its inventory of applications all across the ONOS cluster. So, installing an application on a single controller will move it out to other controllers in the network. ONOS application subsystem extends the Apache Karaf feature service as seen in figure 30, which inherits the standard OSGi bundle management services [95]. ONOS applications can be written as a simple component that offers no API and is
self-contained. They generally interact with the network environment. Examples are the reactive forwarding and proxy ARP applications. Then we have an application with a Service Interface that offers an API for other Apps, CLI, REST or GUI. Such applications interact with network environment, but also other software entities. Next up we can have applications as a singleton “service component” with the activate, deactivate and modify methods. They have dependencies on other services that are auto wired by OSGi SCR. Finally, we can have applications that can have their own state using the store pattern that delegates responsibility for tracking the state to a separate component.
52 | P a g e
Figure 30 Application subsystem layers
ONOS comes with built-in sample and test applications pre-installed and bundled, although they need to be activated. All optional software components can be installed or withdrawn from the system without rebuilding and reconfiguring ONOS.
We will discuss how northbound applications can be written and installed into the ONOS core. A process that we will use when creating our own application for intents.
4.3.3.1 Application Writing
We can generate an ONOS template application using the onos-create-app utility.
Creating an app within a folder named name-app. We need to update the properties file (pom.xml) [105] with the name and title as seen in Figure 31
Figure 31 Properties for a POM file
Applications at bundle loading and unloading make use of annotations like @Activate or
53 | P a g e
the application logic. Karaf loads all the services the application is expected to use with these hooks in the entry and exit functions. For example, an application may need the packets received by ONOS. The PacketProcessor interface is registered with the
CoreService.registerApplication() method in the org.onosproject.core.CoreService package that
will provide the packets at application startup.
We write our core logic or business logic for the application in Java files that will be compiled to the appropriate class files. We can then build our ONOS app. That is usually done at the root directory by calling the following Maven commands
The Apache Maven compiler takes care of compiling all the application java files, linking all the dependency bundles, downloading those bundles which are not locally available. Once the process is complete, the Java archive (jar) is be placed at a location within the ONOS folder for Karaf to access. Now our application is ready to be activated
We can edit our code, rebuild, reinstall and reactivate the application.
We can also interact with the REST API [97] GUI to upload our application packages as seen in figure 32 in the highlighted box.
54 | P a g e
Figure 32 Upload .oar packages
We now discuss the ONOS multi-controller architecture.
4.3 ONOS Multi controller Architecture
A high-level of the pre-Owl multicontroller cluster architecture is presented with its drawbacks and finally the ONOS 1.4 cluster architecture with its improvements that provides scalability and availability in the network [79] is discussed.
4.3.1 ONOS Pre-Owl Cluster Architecture
ONOS is installed on the target machines. Setting up the cluster was easy that could be configured either using a configuration file cluster.json or with the onos-form-cluster script executed from the CLI.
55 | P a g e
The $TARGET_MACHINE_X_IP is the IP address of the target machines that should form the cluster [103]. This would spawn a topology as seen in figure 33
Figure 33 Distributed SDN architecture with a single cluster of five controllers
The architecture layer had embedded Atomix nodes to form raft clusters that communicated with one another to elect the leaders in the network. On the nodes themselves the masters were
responsible to decide which controller node is regulating which switch. The Raft protocol running would replicate the state and coordinate the state changes across the nodes on the target machines. The Netty [104] event-driven framework was used for the peer-to-peer communication that support flows, including communications between nodes. Typically, the network would consist of an odd number of nodes (3,5 & 7). This was because as a consensus protocol, raft
56 | P a g e
required strict cluster membership information to preserve consistency while forming the cluster by counting the votes and hence avoiding the brain-spilt problem [56].
However, there were some complications with the architecture. Embedded raft imposed strict cluster configuration requirements with a little advantage. Fixed cluster configurations need to be defined in advance like nodes and the distribution of the raft partitions. Running the consensus on the controllers themselves reduced availability. As seen in figure 34 raft requires most of the controller to be available to make progress.
Figure 34 State consistency through Raft consensus
The throughput would reduce as the number of nodes increase in the cluster. So, in general embedding an expensive protocol like raft or many instances of an expensive protocol affected other portions of the system like the southbound protocols. Also, it has been proven to be difficult, expensive & troublesome to re-configure the scale of clusters [80]. Since cluster configuration is built inside raft, it is a complex process to add & remove nodes to a cluster. In general, as raft is a leader-based protocol, every operation must go through a leader. As an example, in a 7-node cluster, 6 of them must send operations to some other node, so there is always flow of traffic between the nodes. 1/𝑛 primitive operations still require 2 𝑅𝑇𝑇. Hence the cost is too much without any substantial added benefit. All the distributed premises in ONOS
