CYC-002 · 2026-08 · cloud / infrastructure

Building an OpenStack private cloud, and everything that broke

By Mohsin Rasul · source & reports

Over three deployments I took OpenStack from a single-node DevStack sandbox to a two-node Kolla-Ansible cloud with Heat orchestration and Octavia load balancing. The happy path is in the docs. This is mostly about the other path: the disk that filled up, the scheduler that found no valid host, and a load balancer that sat in PENDING_CREATE forever.

Phase 1: an isolated tenant on DevStack

The goal was a tenant that can only be reached the way I say it can: its own network and router, a firewall that denies by default, and fixed resource limits so one VM can't starve the others.

openstack network create tenant-net
openstack subnet create --network tenant-net \
  --subnet-range 192.168.100.0/24 --dns-nameserver 8.8.8.8 tenant-subnet
openstack router create tenant-router
openstack router set tenant-router --external-gateway public
openstack router add subnet tenant-router tenant-subnet

# default-deny: a new security group allows nothing inbound until you add rules
openstack security group create tenant-secgroup
openstack security group rule create --protocol tcp --dst-port 22 tenant-secgroup
openstack security group rule create --protocol tcp --dst-port 80 tenant-secgroup

# 1 vCPU / 1 GB / 5 GB so instances can't outgrow their share
openstack flavor create --ram 1024 --disk 5 --vcpus 1 project-flavor
openstack server create --flavor project-flavor --image <ubuntu-image> \
  --network tenant-net --security-group tenant-secgroup instance-1

Each instance got a floating IP, a one-to-one NAT from the public pool, so the VMs themselves stay on the private subnet. From outside, only the two ports I opened answer:

nmap -Pn -p 22,80 172.24.4.35   # 22 and 80 open, everything else filtered

What broke

• The disk filled up mid-install. Ubuntu's installer only gives the root logical volume part of the disk by default, and DevStack plus a 3 GB ISO in Glance doesn't fit. Growing the LV into the free space fixed it:

  sudo lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
  sudo resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv

• Networking hung after a reboot. DevStack isn't built to survive restarts, and Open vSwitch/OVN came back in a bad state. Restarting both and rebuilding the stack got it healthy again:

  ./unstack.sh
  sudo systemctl restart openvswitch-switch ovn-central
  ./stack.sh

That second problem is the real lesson of phase 1: DevStack is for experiments, not for anything you want to keep. It's what pushed me to Kolla-Ansible.

Phase 2: two nodes with Kolla-Ansible and Heat

Kolla-Ansible runs every OpenStack service as a Docker container and deploys them from an Ansible inventory. I split the cloud across two machines: ubuntu-node1 for control and networking, ubuntu-node2 for compute.

python3 -m venv ~/kolla-venv && source ~/kolla-venv/bin/activate
pip install 'ansible>=8,<9' kolla-ansible

ansible -i ~/multinode all -m ping          # check both nodes answer first
kolla-ansible bootstrap-servers -i ~/multinode
kolla-ansible deploy -i ~/multinode
kolla-ansible post-deploy -i ~/multinode    # writes /etc/kolla/admin-openrc.sh

Instead of creating resources by hand like in phase 1, I described them in a Heat template so the whole tenant comes up with one command. An excerpt:

resources:
  project_network:
    type: OS::Neutron::Net
    properties:
      name: heat-private-net

  project_subnet:
    type: OS::Neutron::Subnet
    properties:
      name: heat-private-subnet
      network: { get_resource: project_network }
      cidr: 192.168.100.0/24
      dns_nameservers: [ "8.8.8.8" ]

  project_security_group:
    type: OS::Neutron::SecurityGroup
    properties:
      name: heat-sec-group
      description: "Allow ICMP and SSH traffic"
      rules:
        - protocol: icmp
  # … SSH rule and the server resource follow

openstack stack create -t heatStack.yaml Cirros

What broke

• openstack stack create didn't exist. Heat isn't enabled in Kolla by default, and the CLI plugin is a separate package. Fix: set enable_heat: "yes" in /etc/kolla/globals.yml, deploy just that service, and install the client:

  kolla-ansible deploy -i ~/multinode --tags heat
  pip install python-heatclient

• The stack failed with NoValidHost. Nova couldn't find anywhere to put the VM, because my nodes were themselves VMs on ESXi and KVM hardware acceleration wasn't available inside them. Telling Nova to use software emulation fixed it:

  # /etc/kolla/globals.yml
  nova_compute_virt_type: "qemu"
  
  kolla-ansible reconfigure -i ~/multinode --tags nova

  QEMU emulation is slow, but for a lab that's the trade-off for running a cloud inside a hypervisor.

After that the stack reached CREATE_COMPLETE and heat-test-vm came up on the compute node without any manual steps.

Phase 3: Octavia load balancing

Octavia provides load balancers as a service. For each one it boots a small VM, an amphora, running HAProxy. The plan was three CirrOS web servers behind one load balancer. My first load balancers never left PENDING_CREATE, and trying to delete them only moved them to PENDING_DELETE, where they stayed. Getting unstuck took three separate fixes.

Fix 1: delete load balancers stuck in PENDING_*

Octavia refuses to delete anything in a PENDING_* state, because it assumes a worker is still busy with it. Nothing was. The way out is to mark them as failed in Octavia's database, after which a cascade delete works:

DB_PASS=$(awk -F': ' '/database_password/ {print $2}' /etc/kolla/passwords.yml | tr -d '\r')
sudo docker exec -it mariadb mysql -u root -p"$DB_PASS" -h 127.0.0.1 -e \
  "UPDATE octavia.load_balancer SET provisioning_status='ERROR' WHERE id IN ('<lb-id>');"
openstack loadbalancer delete --cascade <lb-id>

Editing a service's database by hand is a last resort, so I only touched the status column of the stuck rows.

Fix 2: turn off the jobboard

That cleared the symptom, not the cause. New load balancers got stuck the same way. The cause was enable_octavia_jobboard, which defaulted to on. The jobboard stores Octavia's task state in Redis, and I hadn't deployed Redis, so workers queued tasks that nothing could pick up. Nothing logged an obvious error. Disabling it puts Octavia back on plain RabbitMQ task processing:

# /etc/kolla/globals.yml
enable_octavia_jobboard: "no"

kolla-ansible reconfigure -i ~/multinode --tags octavia

Fix 3: give the amphora image to the service project

Load balancers now progressed, then failed with ComputeBuildException. The amphora image had the right tag but was owned by the admin project, and Octavia only looks for amphora images owned by the service project. Moving it over fixed the build:

SERVICE_PROJECT_ID=$(openstack project show service -c id -f value)
openstack image set --project "$SERVICE_PROJECT_ID" <amphora-image-id>

The backends

CirrOS is tiny and has no real web server, so each backend runs BusyBox's. I first tried a Netcat one-liner, which handles one connection at a time: the load balancer's health checks collided with real requests and produced intermittent 503s. BusyBox httpd handles concurrent connections and the 503s went away.

mkdir -p /home/cirros/html
echo "Hello from web-server-1" > /home/cirros/html/index.html
sudo busybox httpd -p 80 -h /home/cirros/html

The security group also needed HTTP opened explicitly before the load balancer could reach them; until then requests just timed out.

openstack loadbalancer create --name prod-web-lb --vip-subnet-id <subnet-id>
openstack loadbalancer listener create --name web-listener --protocol HTTP --protocol-port 80 prod-web-lb
openstack loadbalancer pool create --name web-pool --lb-algorithm ROUND_ROBIN \
  --listener web-listener --protocol HTTP
openstack loadbalancer member create --subnet-id <subnet-id> \
  --address 192.168.100.231 --protocol-port 80 web-pool   # repeated for each backend

Verification

The VIP is on a private tenant network, so I tested from inside Neutron's DHCP namespace for that network, which sits on the same subnet:

for i in {1..6}; do
  sudo ip netns exec qdhcp-<network-id> curl -s http://192.168.100.71; echo
done

Hello from web-server-3
Hello from web-server-1
Hello from web-server-2
Hello from web-server-3
Hello from web-server-1
Hello from web-server-2

Six requests, each backend hit twice, in strict rotation.

Troubleshooting cheat sheet

What I'd do differently

• Restrict who can reach SSH. My rules opened ports 22 and 80 to any source address, which is OpenStack's default when you don't pass --remote-ip. HTTP can stay open, but SSH should be limited to an admin network or a bastion host. Default-deny only helps if the allow rules are narrow too.
• Read the defaults before deploying. Two of the worst problems, the jobboard and the image ownership, were documented behaviour I hadn't checked. Neither produced an obvious error message.
• Put everything in code. Heat took phase 2 from a list of commands to one file. The phase 3 load balancer should be in the template too, and Terraform would make the whole thing portable beyond OpenStack.
• Add TLS at the listener. Everything here is plain HTTP. Octavia can terminate TLS with certificates stored in Barbican, OpenStack's secrets service. That's the next step.

full reports and command history: github.com/Mohsin-Rasul/openstack-private-cloud-infrastructure