I have created this post or notes while searching to enable a Layer-3-only setup for OpenStack. I need to wrap up my knowledge and a fundamental explanation of some of the terms or problems for my colleagues.
I explained the cloud network from a fundamental approach, which may not even be true for some aspects like this can not be done, etc. What I mean by that kind of information is that it may lead a very complex technics or even more complex operations, which could be more usable from the technical point of view or operational point of view.
I intend to explore dynamic routing for OpenStack to build Layer-3-only underlay, which I will cover in more detail in the following post.

Why Overlay Networks

There can be two different operational working models for virtual environment customers, similar to any seller-buyer product. The first model involves administrators creating resources for customers, while the second model allows customers to develop resources either free or from a pre-defined list. As network resources are part of virtualization, customers can freely create their own networks or choose from pre-defined networks already made by the administrator.
Network virtualization is frequently used interchangeably with overlay networks, but they differ. Network virtualization involves building a logical network over physical resources, as with all types of virtualization. Let’s now understand overlay networks.
Overlay networking involves building your own network over another without the underlying network being aware. A typical example of this is the VPN services used by companies. When you connect to your company network over the Internet, the VPN service builds a logical tunnel between your computer and the company network.
This is almost true cloud networking. Cloud operator builds a platform and gives flexibility to the customers for creating their own networks.
In data center networks, a single server is insufficient, so a large group of network devices is used to create the underlying network, called the underlay. The cloud platform builds an overlay network for each customer over this underlay. However, sometimes there needs to be clarity between overlay networks and network segmentation, which separates customer network traffic from each other. Let’s explain this with a couple of examples.
Overlay networks are generally needed for IPv4 networks. The problem of IPv4 resource exhaustion is introduced by using private IPv4 addresses, which are not routable on the Internet as they are Private. They are used nearly everywhere, including home networks, campus networks, company networks, etc.
Since those private networks do not belong to anyone, the same prefix can be used in different companies or home networks as long as they do not need to be interconnected. This is the case when different companies use the same IP address space and want to connect their networks, and using NAT solves these problems.

From a data center point of view, cloud platforms face challenges in providing dedicated public IPv4 addresses due to their depletion. As a result, they often use private address spaces. However, it is welcome if a customer can afford to allocate public IP addresses to every VM.

Here comes the problem of self-service networks. Let’s look at the picture assuming all customers can create networks and assign any prefix from a private IP address space. Also, we have a network that allows servers to communicate within our data center network, which may also use prefixes from a Private IP address space.

In addition to addressing overlapping prefixes, cloud platforms need to segregate customer traffic to prevent different customer networks from communicating with each other. This can be solved using IP rules, access control lists, etc. However, overlay networks can also solve this problem, eliminating the need for complex access lists between customers.
In addition to addressing overlapping prefixes, cloud platforms need to segregate customer traffic to prevent different customer networks from communicating with each other. This can be solved using IP rules, access control lists, etc. However, overlay networks can also solve this problem, eliminating the need for complex access lists between customers.
By building an overlay network for each customer, cloud platforms can allow customers to use the same IPv4 private address space, prevent customer networks from communicating with each other unless authorized by the cloud platform, and enable customer networks to be built on any data center network, even through servers connected over the Internet. This leads to a very technical term for the cloud network, which is an external network.

How to create networks for cloud users

The creation of virtual networks depends on the mechanism used on the cloud servers. For example, OpenStack cloud uses an Open vSwitch and OpenFlow-based solution. Additionally, it’s essential to understand the concept of using pre-defined networks and the path that involves using Layer-2 between data center switches and cloud servers. Let’s explain these concepts with some examples.
The first type of network that can be used is a VLAN network, defined on the data center network equipment. The data center network is responsible for building those VLAN networks, and the default gateway of these VLANs may be a router, firewall, etc. Different customers can use the same VLAN, or different VLANs can be used for other customers. However, traffic isolation between customers can be challenging. They can be put in different zones on a firewall, or different routers can be used for each VLAN. The main point is that these networks must be pre-defined by the data center operator before they are used by customers. All traffic filtering can be done by the cloud platform. The cloud platform can isolate customer traffic with Layer 3/4 filters applied to the customer’s VM port. In that case, we do not have a firewall for this purpose.

OpenStack refers to these as provider networks. Those networks are provided to the OpenStack by the data center administrators, and they configure VLAN-ID, prefix, default gateway, etc., for those networks and give this information to the cloud administrators.

The first problem comes into play from scaling for Data Center Network.

• Large Layer-2 domains cause high BUM traffic.
• Centralized routing for east-west traffic.
• Switch hardware has limited resources. They have a limited number of Layer-2 records. Using more advanced technologies like EVPN, etc., Evey needs unique physical resources that affect the scaling.

This will be covered later, and let’s skip those issues and continue.

The other one, as mentioned, is to use overlay networks built by the cloud platform.

These networks can be VLANs, VXLANs, or any other type of network. Customers can use the same VLAN ID, prefix, etc., and they can create multiple networks for their projects. These types of networks are called self-service networks.

So, what happens when a customer wants to access the Internet or needs to connect to their different networks?
For provider-type networks, this is simple. They can be routed through a firewall, router, load balancer, etc., and then NAT’d to the Internet. There can be a more complex setup using provider networks where all VLANs can use the same prefix, which requires VRFs on the data center network.
For self-service networks, the cloud platform has to route this traffic to an external network.
The solution to this problem is introducing router functionality to the customers.

Customers can create networks and connect via routers; different customer networks can connect to the same router. If they use the same router, which means sharing a pre-defined router, they cannot use the same prefix, and they have to choose from a pool of prefixes for their project. Traffic isolation of different customers at the router generates another problem that needs to be solved.

How do we connect those routers to the external world, the Internet, etc.? Undoublty using another network, a provider network:

Again, with the same principle for self-service network routers, different routers can use the same external network. One of the main problems of using overlay networks is that the data center network needs to learn about the customer network, making it difficult to route traffic. By the way, I just remembered to mention distributed routing. Let’s change the picture into a centralized one.

• Data center networks do not know the customer networks.
• Data center networks need to understand using the same prefix for different customers. They can only route a prefix to a single destination.

Using Source NAT towards the external network can solve the second problem but won’t solve the first one.
Using overlay networks in data center networks can solve both problems, as it allows customers to use any private IP address space and still isolate their traffic from other customers. However, this requires more advanced networking technologies and may add complexity to the network infrastructure.
In summary, there are different approaches to solving the challenges of routing traffic between customer networks and external networks in a cloud environment, and each method has its advantages and disadvantages. It’s essential to consider the specific requirements and constraints of the environment when choosing a solution. Putting those aside, moving forward.

If we want to avoid using NAT and teach the data center network about the customer networks, we can use static routes or dynamic routing protocols between the server responsible for cloud routing and the data center network devices. However, in this case, customers cannot use the same prefix, and we need to ensure that the routing information is updated whenever a new customer network is added or an existing one is modified or deleted. This requires additional management and configuration effort, which can be challenging to scale as the number of customer networks and complexity grows.

Using a centralized router can become a bottleneck for traffic and cause scaling issues as the traffic from multiple customers will be routed through a single point. This can lead to increased latency and decreased performance, especially as the number of customers and virtual servers grows. Additionally, it can be challenging to provide optimal routing for each customer’s virtual servers when they all go through the same central router.

Distributed Routers are an approach to overcome the scaling issues and non-optimal traffic flows caused by using a central router in a cloud platform. Instead of having a main router, the routing functionality is distributed across multiple servers, with each server responsible for a portion of the routing workload.

By duplicating the routers on each server, distributed routers ensure that traffic between virtual servers in the same cloud platform stays within the same server, thus avoiding bottlenecks and improving performance. Additionally, distributed routers can use advanced techniques like proxy-ARP to optimize traffic flows between virtual servers, reducing latency and increasing throughput.

Overall, distributed routers provide a more scalable and efficient approach to routing in cloud platforms, allowing for better resource utilization and improved virtual server performance.

• •  NAT (floating IP) with DVR. The router, defined on the server, owns the outside NAT ip address via arp.

• • Traffic between customer networks connected to the same router is routed locally.

As we can not use different nat ip addresses for SNAT on every server, SNAT traffic of customers must be centrally routed.

 

If we can not use Source NAT or want to use routing between the data center network and cloud overlay networks, DVR causes some problems. Using static routing on each switch towards the distributed router on every server will not be practical. Again we have two options, dynamic routing neighborship between the central routing agent.

The central node instructs the data center network about floating IP addresses and the overlay networks it has. The data center network installs a route toward the central gateway node for those networks. DVR is only used for routing between customer overlay networks.

This can be solved by using some routing technics and telling the data center network to go specific router for a particular VM. This requires every distributed router to own an ip address on the external network.

A more scalable approach to establish dynamic routing between every DVR and its connected network node.

Do we need Layer 2 between the cloud platform and data center network the underlay?

This question is about something other than the historical reasons behind it. We can avoid using Layer-2 between the data center network and the cloud platform. However, the problem with the large Layer-2 domain is there. Logically it is like sitting in a conference room where everyone tries to speak simultaneously.
Many cloud platforms use Layer-2 networks for simplicity, and technologies such as DVR work well with Layer-2 networks. However, as cloud platforms become more advanced and customers’ needs become more complex, there may be a greater need for advanced routing technologies like MPLS VPN and EVPN. These technologies can help improve network scalability, performance, and security by providing advanced routing and forwarding mechanisms.

Using those technologies, it is possible to build a network with Layer-2 free in its backbone, which has better ECMP, convergence, and resiliency capacities than the traditional Layer-2 networks. They also carry DVR-like features and distributed gateways for VLANs for edge networks.
Currently supported OpenStack mechanisms for overlay networks are OVS and OVN, which are nearly identical to OVS. OVN is an OpenFlow controller.
OVS is a logical switch installed on Linux servers and a virtual switch controlled by OpenFlow rules. There is at least one OVS bridge, which is used for connecting the external networks, and commonly it is called br-ex.
Some may think different customer networks can communicate over a data center network as they are exposed outside the cloud platform. But as we leverage the routing to the edge for the cloud platform, data centers seem like external networks. So each customer sees other customer networks on the Internet, which is the same case with using Layer-2 networks and NAT. So it’s the cloud platform’s responsibility to be able to control customer traffic toward the Internet, even between other customer networks.

The real problem arises when you want to make different routing for each customer. This requires integration between the cloud platform and data center network, like interconnection for MPLS VPN between service providers. Either way, nearly all solutions need to integrate the control plane of the data center network with the cloud platform. When you create a self-service network, its parameters need to be the same as with the data center network to be consistent with the data center network.

Generally, the problem is about using private space usage and the solutions that cloud platforms integrate. They heavily use NAT, ARP, and encapsulation technics to build customer networks. Some of those technics can not be used with IPv6. They do not overlay networks anymore. The expectation should be;

• Isolation of traffic for customer network without using overlays. Which is already implemented with ip rules etc.
• Unhappily, Layer-2 networks for the customer. This is needed for the most legacy application.
• Distributed routing of customer networks using a Layer-3 connection with a data center network.

Oneway or another, this transforms every server into a provider edge device for a data center which should be fine unless we only use a Layer-3 network.

More detail about OpenStack Networking

Openstack uses Neutron for networking services. The service sends requests to create networks to the network driver and is not responsible for making those networks on the servers. Different types of mechanism drivers exist for creating those networks with different technics like Linux bridge driver, open virtual switch driver, and open virtual network driver.
I will focus on the basics of OVS or OVN, which are the same as from a server perspective. OVN is an OpenFlow controller that works well with OVS, and it simply creates flow for each OVS on the servers while seeing all the servers as a single entity.
If you create a self-service network, OVS or OVN driver creates a VLAN network. Thus every self-service network is a different VLAN ID. And the VLAN on the remote servers is connected through tunneling techniques like VXLAN, etc. As those tunnels are over ip, the OVS bridge needs at least one ip address routable to the remote servers.

This is also the same with using provider networks; the only difference is that the extension of the VLAN network is done via underlay, not the OVS.

There is a common difference between those two types of networks. If you use security rules deployed on openstack, you can only restrict routing between VLAN 10 and VLAN 20 if you have a tenancy in your underlay(using VPN/VRF for VLAN 10 and 20).

You can combine those two types using openstack routers (Assuming distributed routing is used).

Openstack Routers allows us to;

• Besides creating a network, we can create multiple networks and route between them within openstack using data center devices.
• We can also SNAT to the router’s external gateway interface and 1-1 NAT from a routable pool. Generally, the IP Pool of 1-1 NAT, or floating ip, is generated from the external network that the router is connected to. This network is a VLAN network that is used to connect the router to an external network.
• Distributed routing of E-W traffic and centralized routing N-S only for SNAT. Floating ip can also be distributed.

Dirtubted routing for floating ip is required special technics. Without a dynamic routing protocol, we can only use a Layer-2 network for the external network. With this, we can send traffic directly to the server with the floating IP. Only that server will respond to the arp request sent by the data center network device it is connected to.

There is also some limitation from the openstack perspective. There can only be one external gateway, and you can only define up to one. Multiple feature req have seen openstack, but I need to know if it’s implemented.
One of the main differences by default is that you can not share a router between different projects. To overcome this issue, you need to play with RBAC rules.
As a result of using ARP for influencing return traffic with DVR, it’s only possible to connect servers to the data center with a Layer-2 external network. To overcome this issue, there is a BGP feature in which Neutron announces the receiving compute note ip addresses as a next-hop for floating ip.