Elastic IP without hardware

[Kukunin]

Hello. I've been messing with my first cloudstack installation, and getting more into networks and administration. I'm trying to build a self-hosted multi-tenant AWS-like cloud, where different clients (accounts) have different VMs. Ideally, I'd give them ability to manage their own VPC, but it's optional.

While it took some time for me to understand different types of networks: management, public, guests and their isolations: VLAN, VXLAN, GRE, one question remains unanswered - how to assign public IPs for specific guest VMs? Especially if they are on different servers/nodes.

I don't have a corporate network, my servers are just dedicated servers that have public IPs assigned to them. These IPs might be from different subnets, and they belong to specific nodes/servers, and can't be shared between. I have no network hardware, just dedicated servers itself. Maybe just vRack (if it's helpful).

Let's say I have server A with a.a.a.a IP address, and server B with b.b.b.b IP address. I have RFC1819 public network 172.16.1.0/24.
I have a client's guest VM running on Server B, and I want to assign a.a.a.a from server A to it.

Is it something that cloudstack provides/supports? Is using OVS plugin would help in this case? If cloudstack doesn't support, what direction would you advise implementing it myself? Can pfSense or VYOS help me with this?

I understand, that I can configure iptables to forward public IP to specific VM gateway. But also, I need to route any outgoing traffic from the VM (guest network) via that public IP versus default gateway. Also, it seems there is internode communication is involved (VM running on Server B, but public IP is assigned to Server A).

This should be a common task, but I've found no information on that regard.

Thank you so much.

[DaanHoogland]

If I understand correctly this is what you use static nat for. Not the server would have public IPs but you have a pool of IPs configured for the public net. The routers (if you use those for your VPCs/networks) would get ip addresses and you can assign extra addresses to them and configure one to be staticly tight to a VM.

Does this sound like your scenario?

[Kukunin]

Yes, it sounds correct. So I did an extra research, and here is what I found. I'm going to leave it here, so if it serves for other people that will find this question, and to validate my thoughts

TL;DR: it's outside cloudstack's scope and should be implemented on a side.

A longer answer, that took some time for me to click is the following:

Basics

CloudStack has a notion of 4 networks:

• management
• guest
• public
• storage (if defined. by default uses management)

Each type of network should be assigned to a physical network (bridge interface), and have some kind of subnet. The physical networks are configured outside cloudstack, there is just an assumption that all nodes and management servers are reachable within the network.

Guest network is used to host accounts' VMs. There are three types of guest networks:

• shared
• isolated
• L2

In my case I need isolated, because I want each client to be totally isolated from each other. CloudStack handles guest to public networks routing via Virtual Router (VR). It supports features like SNAT, Port Forwarding, Load Balancing, IP assignments etc.

Public network

Now it's becoming interesting. VMs are hosted in guests networks, and they use CloudStack's VR to get access to the Public network.
Public network should be a common for the whole zone, all nodes should be reachable from any hypervisor to make Live Migrations and other features work. Public network can be both publicly routable (more corporate world with dedicated IPv4 blocks assigned) or RFC1819 (private LAN, e.g. 192.168.x.x). In my case it has to be RFC1819.

The answer is that this is the place where cloudstack's responsibility ends. If we have a VM in a guest network with IP 10.1.1.15/24, routed to the public network and has a Public IP 192.168.3.5/24 assigned, it's our job to:

• Make sure there is another gateway in our Public network that would give Internet access to VMs
• Control NAT rules to assign public IP (e.g. 15.204.56.58) to 192.168.3.5.

Question 1: is there any notion for true public IPs in cloudstack for RFC1819 public networks? It was confusing that public IPs are not public, and there are true routable public IPs.
Question 2: are there any hooks to allow users to configure this outside router in CloudStack management UI? Literally add Elastic IP functionality, with custom backend. I guess I'd need to write a plugin for this.

Alternatives

The most of the network configurations are outside of CloudStack's responsibility, which makes sense for flexibility. CloudStack just gives some functionality and provides different types of network. With Isolated Networks there are two gateways: guest <> public and public <> internet. You can eliminate the first cloudstack VR by using L2 networks.

Potential solutions for this foreign router

I'm not a big network expert, so everything is pretty new for me.

The main challenge is that each hypervisor has its own internet gateway. Public IPs are assigned to hypervisors. Ideally, VMs running on a hypervisor should use its local gateway. But there are cases, where Public IP from Hypervisor A is assigned to a VM running on Hypervisor B. Then the traffic should be routed via Hypervisor A. Also, there are Live migrations, where VM transparently moves between hypervisors. Ideally, we can also load balance the ISP gateways, so if Hypervisor A's public traffic bandwidth is almost full, the traffic goes via Hypervisor B.

So far, I found the following viable free solutions for this external gateway:

• frrouting on hypervisors
• VYOS in a VM
• tungsten fabric, but I'm afraid about that it's abandoned project

I see some support for 3rd-party proprietary vendors like Netscaler HPX or Nicira NVP, but I'm not sure how actual they are.

For now, I'm going to start easy, with hard-coded NAT rules (iptables) and static routes (ip route), but on a scale, there has to be something more sophisticated. Overall, I'm leaning towards frrouting: less VMs, configuration on the hypervisors, easy to provision (vtysh and config files)

So, the bottom line is that with RFC1819 Public networks, it's completely outside CloudStack to build the proper functionality. The only my question remained is that how to integrate my custom solution into management so users can assign public IPs themselves.

[kiranchavala]

@Kukunin

I was wondering if you had tried vnf and dynamic routing feature of cloudstack and see if it solves ur use case

Cloudstack has support for vnf

https://www.shapeblue.com/vnf-appliance-integration-deep-dive/
https://www.youtube.com/watch?v=JZTT9OlqtDs

Dynamic routing

https://docs.cloudstack.apache.org/en/4.20.0.0/adminguide/networking/dynamic_static_routing.html

https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.youtube.com/watch%3Fv%3DeiK44dNliDE&ved=2ahUKEwiJuLq1steMAxWER2wGHdv3D5AQwqsBegQIDBAG&usg=AOvVaw1VejpxyGNKzbTpFJ_hPwS7

[Kukunin]

Haven't tried but sounds exactly what I was looking for. Thank you for the reference!