The objective of the described protocol is to support the flexible incorporation of 5G experimentation infrastructures into a multi-site NFV ecosystem, through a VPN-based overlay network architecture. Moreover, the protocol defines how to validate the effectiveness of the integration, including a multi-site vertical service deployment with NFV-capable small aerial vehicles.
Network Function Virtualization (NFV) has been regarded as one of the key enablers for the 5th Generation of mobile networks, or 5G. This paradigm allows to reduce the dependence on specialized hardware to deploy telecommunications and vertical services. To this purpose, it relies on virtualization techniques to softwarize network functions, simplifying their development and reducing deployment time and costs. In this context, Universidad Carlos III de Madrid, Telefónica, and IMDEA Networks Institute have developed an NFV ecosystem inside 5TONIC, an open network innovation center focused on 5G technologies, enabling the creation of complex, close to reality experimentation scenarios across a distributed set of NFV infrastructures, which can be made available by stakeholders at different geographic locations. This article presents the protocol that has been defined to incorporate new remote NFV sites into the multi-site NFV ecosystem based on 5TONIC, describing the requirements for both the existing and the newly incorporated infrastructures, their connectivity through an overlay network architecture, and the steps necessary for the inclusion of new sites. The protocol is exemplified through the incorporation of an external site to the 5TONIC NFV ecosystem. Afterwards, the protocol details the verification steps required to validate a successful site integration. These include the deployment of a multi-site vertical service using a remote NFV infrastructure with Small Unmanned Aerial Vehicles (SUAVs). This serves to showcase the potential of the protocol to enable distributed experimentation scenarios.
The introduction of the fifth generation of mobile networks (5G) has implied revolutionizing the telecommunications industry since the beginning of the decade, requiring telecommunication operators to address the much more demanding specifications of the new networking services and applications developed under the 5G umbrella1,2. These new specifications include, but are not limited to, data rate increases, wireless transmission latency improvements, and operational costs reduction. Among the technologies that constitute the foundations of the improvements for this new generation, Network Functions Virtualization3 (NFV) has become one of its key enablers. NFV provides the capacity to softwarize network functions, traditionally relaying on specialized hardware, by using generic-purpose physical equipment instead, such as server computers in a datacenter. With this new paradigm, telecommunication operators and vertical industries can deploy network functions and services as a set of software components, and save costs in both service deployment and maintenance, as well as facilitating a much higher network infrastructure elasticity. This approach alleviates or eliminates the necessity to use dedicated (and usually more complex and less reusable) devices for most network and vertical-specific functions, and supports a much higher and denser degree of operational automation, hence reducing deployment and maintenance costs.
Taking into consideration all the advantages that an NFV environment is able to provide, it is natural that a great number of relevant stakeholders from the telecommunications sector have increasingly been involved in testing new service ideas on NFV environments. In this context, Telefónica and IMDEA Networks Institute have created 5TONIC4, an open research and innovation laboratory focused on 5G technologies. Based in Madrid (Spain), this laboratory has a wide range of technologies available for researches and partners to boost the development and validation of 5G services. In particular, this laboratory has an experimental NFV platform where developers are able to deploy and test their new NFV-based applications and services over on an ETSI-compliant NFV ecosystem5. Thus, experimental conclusions about design choices and technology proposals can be derived in a realistic much more flexible environment than production networks. This platform has been designed to support experimentation activities across multiple external sites, which may be flexibly interconnected to 5TONIC using a well-defined protocol.
The technical solution adopted for the 5TONIC NFV ecosystem considers the utilization of a single NFV orchestrator, implemented using the ETSI-hosted Open Source MANO (OSM) software6. This is the element in charge of managing and coordinating the lifecycle of Network Services (NS). These services may be built as a composition of Virtualized Network/Vertical Functions (VNF), which can be deployed at any of the sites integrated on the NFV platform. The design of the 5TONIC NFV ecosystem has been done in the context of the H2020 5GINFIRE project7,8, where the platform was used to support the execution of more than 25 experiments, selected through a competitive open-call process, across eight vertical-specific experimental infrastructures located in Europe and one in Brazil, the latter connected through a transoceanic link. In addition, the platform was leveraged to build a distributed NFV testbed at a national scale, in Spain, supporting experimentation activities within the Spanish 5GCity project9,10. More recently, an additional Brazilian site has been integrated into the platform, to support joint demonstration activities in the context of a research and innovation cooperation established between Brazil and Europe (i.e., the 5GRANGE project11,12). Last but not least, the infrastructure has been used to support third-party experiments in the scope of the 5G-VINNI project13,14. The geographic distribution of the NFV platform can be seen in Figure 1.
Interested organizations hosting their own NFV infrastructure can flexibly connect to the 5TONIC NFV ecosystem, subject to approval by the 5TONIC Steering Board, become testbed providers within the distributed ecosystem, and be involved in joint experimentation and demonstration activities. To this end, they must feature a VIM (Virtual Infrastructure Manager) compliant with the OSM software stack. The 5TONIC NFV orchestrator is able to interact with the VIMs at the sites involved in a given service deployment, coordinating the allocation and setup of the computing, storage and network resources needed for the instantiation and interconnection of the VNFs that compose a network service, and controlling its lifecycle, from its on-boarding to its final decommissioning.
In order to manage the exchange of control and data traffic within all the interconnected sites, the 5TONIC NFV ecosystem makes use of an overlay network architecture based on Virtual Private Networks (VPN). This approach provides secure PKI-based access to the external sites that are integrated into the 5TONIC ecosystem, allowing the exchange of NFV control information between the OSM software stack and the different VIMs distributed across the testbeds, as well as the exchange of information that is required to manage and configure all the VNFs. Moreover, this overlay network supports the dissemination of data traffic among VNFs that are deployed at different sites.
In this context, this paper details the protocol designed to incorporate an external site to an NFV ecosystem. The protocol assumes that the ecosystem is governed by a single NFV orchestrator, installed at a central site, and external sites feature a VIM solution compliant with the orchestrator software stack. The proposed protocol allows to increment the portfolio of resources of the experimental ecosystem, with the flexible incorporation of NFV sites and vertical-specific infrastructures. This enables the creation of a distributed MANO platform capable of testing and validating novel network and vertical services across multiple sites, under the control of a single NFV orchestrator. In order to illustrate the inner operation of the protocol, the process will be exemplified by adding an external NFV site to the current 5TONIC NFV ecosystem, describing the needed components at the external site and 5TONIC, as well as all the steps to be taken during the integration process. Figure 2 provides an overview of the objective of the integration, with the new NFV-based testbed attached to the 5TONIC platform from where network services can be deployed, by means of VPN connections between the central site and the rest of the external infrastructures.
In addition, to showcase the effectiveness of the protocol, the deployment of a simple vertical service will be shown, using the 5TONIC ecosystem and an external site with NFV-capable small unmanned aerial vehicles (SUAVs). The design of the vertical service has been inspired by an experiment presented in Vidal et al.9, which has been simplified for the illustration purposes of this paper. Figure 3 outlines the service, which aims at aiding smart farming activities on a remote area. The service considers a smart farming service provider who uses SUAVs to collect and disseminate the data produced by meteorological sensors scattered over a crop field. For simplicity, the experiment presented in the paper considers a single SUAV and a sensor, capable of providing temperature, humidity, and pressure measurements. In the experiment, the external NFV site hosts a Wi-Fi access point that is deployed as VNF over the SUAV. This VNF offers network access connectivity to the sensor, forwarding the sensed data towards a gateway function. The latter is deployed as a VNF on a ground equipment (a mini-ITX computer). The dissemination of data from the sensor to the gateway function follows a Publish/Subscribe approach based on the Message Queuing Telemetry Transport (MQTT) protocol15. The gateway function processes and then disseminates the data towards an Internet-of-things (IoT) server, which is made available as a VNF at the central site of the NFV ecosystem, based on the Mainflux16 open-source platform. Finally, the scenario assumes a remote area where Internet connectivity is provided by a cellular non-3GPP access network. Hence, the service includes two additional VNFs: 1) an access router VNF, which implements the user-plane protocol stack of a 3GPP user equipment connected to a non-3GPP access network17; and 2) a baseline implementation of a 5G core network, supporting the forwarding of information between the access router and the IoT server VNFs. To this purpose, the 5G core VNF provides a simplified implementation of the user-plane of a non-3GPP interworking function and a user plane function, as defined by 3GPP17.
Finally, Figure 4 represents the most relevant processes involved during the development of the protocol, highlighting their logical interconnections and the entities in charge of their execution.
1. Provision of the central site of the NFV ecosystem (prior requisites of the experiment)
2. Configuration of the virtual private network service
3. Integration of an external NFV site
4. Validation of the NFV multi-site platform with a realistic vertical service
After carefully following the protocol to incorporate a new site to the central platform and run one network service to validate its proper functionality, Figure 6 depicts a screenshot of the open-vpn-monitor tool. It can be observed how the new site is using the VPN for all its communications, showing how its communications follow the VPN to allow this data exchange and, in consequence, the correct addition of the new site to the VPN service.
As depicted in Figure 3, the network service is delivering information from a sensor located in a remote infrastructure to the server located in the central site. In addition, Figure 7 displays the successful deployment of the network service from the OSM web GUI, showing how the experiment can be properly instantiated in the new remote infrastructure from the MANO stack located within the central site. Moreover, the time required in the experiment to complete the deployment of the service is around eight minutes. This value, along with the time needed to on-board the service descriptors into the orchestration platform (about 9 seconds, with 1.3 seconds per descriptor, considering both the NS and each VNF descriptors), enable to satisfy the Key Performance Indicator (KPI) of 90 minutes for the service creation time, as indicated by the 5G Infrastructure Public Private Partnership34. In this context, the work presented in Vidal et al.9 includes an in-depth analysis of the service creation time with multiple sites using the presented protocol.
Figure 8 displays the data collected from the sensor, including the values of humidity, temperature and pressure respectively. These samples correspond to all data sent from the sensor to a remote server located in 5TONIC, where these values are stored in a database. All these data demonstrate that the platform is able to deploy practical network services after the inclusion of a new infrastructure, as well as to correctly enable communications between sites.
Figure 1: VPN service site distribution. Distribution of the VPN service through the platform and their link connectivity (all passing through 5TONIC). Please click here to view a larger version of this figure.
Figure 2. Overview of the platform and VPN service. This figure shows all elements of the platform: the central location, along with its NFV Infrastructure, the VPN service and a new infrastructure aggregated to the system. It also includes the connections between its elements. Please click here to view a larger version of this figure.
Figure 3: Overview of the network service. It depicts the elements involved in the network service, its distribution and its logical, and networking, connectivity. Please click here to view a larger version of this figure.
Figure 4: Protocol Workflows. Each column represents one section of the protocol, where every action performed is described, its logical connection between them and the component in charge of its execution. Please click here to view a larger version of this figure.
Figure 5: Pin configuration scheme. Diagram representing how to make the physical connections between the board pins of the sensors and the GPIO pins of the SBC that incorporates that sensor. Please click here to view a larger version of this figure.
Figure 6: OpenVPN-monitor snapshot. The picture shows that the aggregated infrastructure is connected to the VPN service, including some of its details regarding its connection. Moreover, the figure also depicts additional connections belonging to other remote infrastructures. Please click here to view a larger version of this figure.
Figure 7: OSM NS deployment status. OSM graphical interface, showing the successful deployment of the test network service in the remote infrastructure. Please click here to view a larger version of this figure.
Figure 8: Representative analysis of the data collected by the sensor. (A) Illustration of the temperature data periodically collected by the sensor every 5 seconds. (B) Graphical representation of the humidity data collected by the sensor every 5 seconds. (C) Visual depict of the pressure data collected by the sensor every 5 seconds. Please click here to view a larger version of this figure.
One of the most important aspects of the previously described protocol is its outstanding flexibility to incorporate new computational infrastructures to an NFV ecosystem, regardless of their distribution in terms of geographical location (as long as bandwidth and latency of the network communications with remote sites supports it). This is possible through a VPN-based overlay network architecture, which enables the establishment of a virtual link to connect remote sites to the central premises of the NFV ecosystem. This approach enables the provision of an effective and secure channel to support the NFV and data communications among sites of an NFV ecosystem, reducing the probability of external parties accessing and/or modifying sensitive information regarding NFV orchestration processes and data from deployed services. In this context, the protocol also describes a specific methodology to share securely the VPN credentials with the external sites that will enable the integration of new infrastructures. The protocol has been exemplified using the NFV ecosystem made available at 5TONIC by Universidad Carlos III de Madrid, Telefónica, and IMDEA Networks Institute, although it is generic to be utilized in other NFV environments satisfying the prior requisites mentioned in step 1 of this protocol.
In addition, it is worth emphasizing the exclusive utilization of open-source tools and software for the protocol implementation. Notwithstanding the potentially beneficial functionalities that could be offered by different proprietary solutions (e.g., Fortinet35), the use of open-source developments has facilitated the integration of all elements encompassed by the protocol due to their inherent characteristics such as cost effectiveness, an extensive software support provided by the open-source community, and a high level of reliability, just to name a few of them. Moreover, the utilization of open-source technologies can also promote synergies between components of similar nature. For instance, in order to monitor the VPN connection status for the clients using the platform, the VPN service implemented throughout the protocol could rely on the open-vpn monitor tool36 (a python-based monitoring tool capable of interoperating with OpenVPN servers).
On the other hand, the protocol specification considers the instantiation of networking services across different sites for validation purposes. In this regard, it is important to highlight that the deployment of services on a given site is subject to the availability of compute, storage and network resources at the site, as well as of specialized equipment that might be needed to perform the deployment (e.g., NFV-enabled SUAVs). This is not a limitation of the protocol, and should be taken into account by stakeholders interested in reproducing the experiment described in this paper.
Moreover, it should be noted that the time required to carry out the deployment of network services highly depends on several factors such as the network path between the orchestrator and the different VIMs, the performance of data communications between the VIM and its managed computational nodes, and also in the intrinsic nature of these computational nodes (not only because of their available computing resources, but also the technologies incorporated to conduct the virtualization of network functions).
Finally, and given the outstanding performance that this platform and its VPN service had on the European projects and collaborative works where it has been used so far (e.g., 5GINFIRE, 5GRANGE or 5GCity, mentioned in the introduction of this document), it will be regarded as an important element in emerging European projects where Universidad Carlos III de Madrid, Telefónica, and IMDEA Networks Institute participate, such as the Horizon 2020 LABYRINTH, or national projects, like TRUE-5G.
The authors have nothing to disclose.
This work was partially supported by the European H2020 LABYRINTH project (grant agreement H2020-MG-2019-TwoStages-861696), and by the TRUE5G project (PID2019-108713RB-C52PID2019-108713RB-C52 / AEI / 10.13039/501100011033) funded by the Spanish National Research Agency. In addition, the work of Borja Nogales, Ivan Vidal and Diego R. Lopez has partially been supported by the European H2020 5G-VINNI project (grant agreement number 815279). Finally, the authors thank Alejandro Rodríguez García for his support during the realization of this work.
Bebop 2 | Parrot | UAV used in the experiment to transport the RPis and thus, provide mobility to the compute units of external site. | |
BME280 Sensor | Bosch | Sensor capable of providing readings of the environmental conditions regarding temperature, barometric pressure, and humidity. | |
Commercial Intel Core Mini-ITX Computer | Logic Suppy | Computer server which hosts the OpenStack controller node (being executed as a VM) of the experiment's extternal aite. In addition, another unit of this equipment (along with the RPis) conforms the computational resources of the NFV insfrastrucure included in that site. | |
Iptables | Netfilter – Open source tool | (Software) An open source command line utility for configuring Linux kernel firewall rulset. Source-code available online: https://www.netfilter.org/projects/iptables/ | |
Lithium Battery Pack Expansion Board. Model KY68C-UK | Kuman | Battery-power supply HAT (Hardware Attached on Top) for the UAV computation units composing the NFV infrastructure of the external site. | |
MacBook Pro | Apple | Commodity laptop utilized during the experiment to obtain and gather the results as described in the manuscript. | |
Mainflux | Mainflux Labs – Open source platform | (Software) Open source Internet of Things (IoT) platform used in the experiment for implementing the virtual network function called as IoT Server VNF. In addition, this platform includes an open-source software based on Grafana which allows the visualization and formatting of the metric data. Source code available online: https://www.mainflux.com/ | |
Open Source MANO (OSM) – Release FOUR | ETSI OSM – Open source community | (Software) Management and Orchestration (MANO) software stack of the NFV system configured in the experiment. Source-code available online: https://osm.etsi.org/docs/user-guide/ | |
OpenStack – Release Ocata | OpenStack – Open source community | (Software) Open source software used for setting up both the NFV infrastrucure of the central site and the NFV infrastructure of external site within the experiment. Source-code available online: https://docs.openstack.org/ocata/install-guide-ubuntu | |
OpenVPN – Version 2.3.10 | OpenVPN – Open source community | Open source software implementing the VPN service presented in the experiment for the creation of the overlay network that will enable the operations of the NFV ecosystem (providing connectivity among all the sites comprising the ecosystem). Source-code available online: https://openvpn.net/ | |
Openvpn-monitor | Python – Open source software | (Software) Open source program based on Python code that allows the visualization of the state of the VPN service, as well as the representation of the sites that are connected at every instant. For this purpose, the program check priodically the information provided by the VPN server implemented with OpenVPN. Source-code available online: https://github.com/furlongm/openvpn-monitor | |
Paho-mqtt 1.5.0 | Python – Open source library | (Software) Open source library developed in Python code that enables the trasmission of the data read by the sensor through the use of MQTT standard Source-code available online: https://pypi.org/project/paho-mqtt/ | |
Ping | Debian – Open source tool | (Software) An open source test tool, which verifies the connectivity between two devices connected through a communications network. In addition, this tool allows to assess the network performance since it calculates the Round Trip Time (i.e., the time taken to send and received a data packet from the network). Source-code available online: https://packages.debian.org/es/sid/iputils-ping | |
Power Edge R430 | Dell | High-profile computer server which provides the computational capacity within the central site presented in the experiment. | |
Power Edge R430 | Dell | High-profile computer server in charge of hosting the virtual private network (VPN) service. Note that the computing requirements for provisioning this service are high due to the resource consumption of the encryption operations present in the service. | |
Power Edge R630 | Dell | Equipment used for hosting the virtual machine (VM) on charge of executing the MANO stack. In addition, the OpenStack controller node of the central site is also executed as a VM in this device. Note that the use of this device is not strictly needed. The operations carried out by this device could be done by a lower performance equipment due to the non-high resource specifications of the before mentioned VMs. | |
Raspberry PI. Model 3b | Raspberry Pi Foundation | Selected model of Single Board Computer (SBC ) used for providing the computational capacity to the experiment's external site. In addition, this SBC model is used during the deployment of the included realistic service for interpreting and sending the data collected by a sensor. | |
RPi.bme280 0.2.3 | Python – Open source library | (Software) Open source library developed in Python code that allows to interface the sensor Bosch BME280, and interpret the readings offered by that sensor. Source-code available online: https://pypi.org/project/RPi.bme280/ |