How to Manually Deploy oVirt Hosted Engine and Fix Some Common Issues
oVirt is a great product, and it’s hosted engine is a great concept. It is a mostly automated deploy of a VM containing the oVirt Hosted Engine and the initial setup of your first host. This is similar to how VMware and XCP-ng/XenServer does it, however the default install method of the hosted engine tries to do too much. With VMware and XCP-ng/XenServer, you are expected to have a functional hypervisor first with all of its networking, storage, and permissions all setup and working, then you deploy the virtual appliance to manage it more completely. oVirt on the other-hand tries to do everything in one huge Ansible script that if one thing breaks, it removes everything (or tries too) and requires you to redeploy, instead of trying to fix the issue.
If oVirt wants to fix this over complication, they could do a few things. Firstly, have the script fail gracefully, and allow the user to attempt to fix it before purging everything. Secondly, just have the hosted engine deploy to an existing KVM/QEMU setup initially, instead of dumping everything to /tmp/ then purging it if something goes wrong. Lastly, create a basic plugin for cockpit for extremely basic oVirt management for vm’s, like turn on, see XML, delete, etc. You could even fork or just use the standard cockpit virtualization plugin, that would be great! This would go a long way in diagnosing issues when ovirt-engine is down, and it is how all of the other big boys do it in the industry.
Setup and Dependencies
First, we need to install all of oVirts dependencies. Follow oVirt’s guide on the cockpit wizard here up until the engine deploy. https://www.ovirt.org/download/
Next, we need to install tmux
dnf install tmux -y
Next, we need to enable scrolling on tmux.
echo "set -g mouse on" > .tmux.conf
Now enter tmux with the command tmux then run
tmux source-file .tmux.conf
Getting the SEED.iso
The oVirt hosted engine setup has a seed.iso on it that has your answer file in it, as well as a cloud-init script to set the password, hostname, and a few other options on the Hosted Engine. We need to get far enough in the setup script to launch the hosted-engine VM where that seed.iso is created so we can copy it. This ISO is not strictly needed, but it is easier than entering single user mode and changing the default creds on the machine. Running the setup also has a bonus of setting up libvirt and KVM for us, so we don’t have to do that manually.
To grab it, open 2 ssh sessions, And also open a cockpit session, install the virtual machine plugin, and then navigate to it. On the first ssh session, we want to run the ovirt-hosted-engine-setup. Run though the setup, answering all of the questions, and wait to see when it deploys the hosted-engine vm. When it does, we want to navigate to Disks on cockpit, and find the Seed.iso. in the second ssh terminal, copy that to another directory, like /opt/ or /root/
Getting the Hosted-Engine VM
There are 2 ways to go about getting the Hosted-Egnine VM. The way I’ll be showing you is by getting the latest OVA from the oVirt repository. Another way this could be done is by copying or cloning the VM that the setup-script deploys before it has a chance to delete it. That way is much easier, but not everyone will have that option, so I’ll go though the RPM way.
Firstly, we need to get the VM OVA. You can download it from the oVirt repository. At the time of writing this, it is here: https://resources.ovirt.org/pub/ovirt-4.4/rpm/el8/x86_64/ovirt-engine-appliance-4.4-20201110154142.1.el8.x86_64.rpm
Next, we need to extract it. In my case to commands were
wget https://resources.ovirt.org/pub/ovirt-4.4/rpm/el8/x86_64/ovirt-engine-appliance-4.4-20201110154142.1.el8.x86_64.rpm rpm2cpio ovirt-engine-appliance-4.4-20201110154142.1.el8.x86_64.rpm | cpio -idmv cd usr/share/ovirt-engine-appliance # note this is not /usr/ tar -xvf ovirt-engine-appliance-4.4-20201110154142.1.el8.ova
Finally, we need to get the qcow from it. It should just be some filename UUID with no extension. That is the qcow2. It is in images/SOME-UUID/. In my case it is images/6d1ea50f-eecf-4077-bcd2-2e14035c39b5/ with the qcow2 being named e412657a-5448-49e4-9b6a-d1b35f5d9892. I renamed it to Ovirt-Hosted-Engine.qcow2
cd images/6d1ea50f-eecf-4077-bcd2-2e14035c39b5 mv e412657a-5448-49e4-9b6a-d1b35f5d9892 Ovirt-Hosted-Engine.qcow2
You will need to move the qcow2 into the place where you create your storage pool, wherever that is.
Creating the VM
Now we need to setup QEMU\KVM. Make a storage pool somewhere on the system. Cockpit will make this process easy. You can also use Virsh or Virt-manager. I created mine in /home/libvirt/. You are also going to need to see if you have a network bridge already setup. Ovirt should have setup one named ovirtmgmt using the nic you selected in the setup script.
There are a few ways we can create the VM. commandline via visrh, Via Cockpit, and via Virtual Machine Manager. Im choosing the latter. Im connecting via ssh from another machine, so you need to have PKI setup to do so. ssh-keygen, then ssh-copy-host-id root@ip. On virt-manager the libvirt-url is qemu+ssh://user@ipaddress/system. See the “Virsh Credentials” heading on this article for the credentials for virsh.
You want to create a VM with the RHEL 8.2 Template. Q35 chipset with Bios Boot, VirtIO disk and network adaper, SATA CD with the SEED.iso if you have it, and Video QXL. Set it to have 4 CPU’s and at minimum 6GB of RAM. You will want to use the ovirtmgmt bridge if you can. If it is not there, go and add your main NIC to a bridge named ovirtmgmt. The hosted engine script should leave it intact. Go and start the VM, and the login to root with the pasword you set earlier. If you dont have the SEED.iso, you will need to enter single user mode to change the root password to something you know.
Configuring the Engine
Next, we are going to want to ssh into the virtal machine. From there, login to root and run engine-setup. Answer the questions, and then hopefully all will workout fine! If you cant ssh into the box, you need to make sure the firewall is allowing the port though on the VM, and check how your networking is setup. It should just be using a bridge not macvtap or nat.
Once the ovirt-egnine is setup, you should be able to add the host that the VM is running on. You may have to manually run it via a QEMU one liner for the host to get added, but I’m not sure on that one, as it may restart libvirt and kill active virtual machines. Another option is to copy the VM to anther computer and have it run there, then copy it back after the host is done setting up.
Fix VDSM Certificate Error “ovirt Unable to access credentials /etc/pki/vdsm/libvirt-vnc/ca-cert.pem: Permission denied\nDomain installation does not appear to have been successful.”
run this script:
#!/bin/bash rm -rf /etc/pki/vdsm/certs/* rm -rf /etc/pki/vdsm/keys/vdsmkey.pem rm -rf /etc/pki/vdsm/libvirt-migrate/* rm -rf /etc/pki/vdsm/libvirt-spice/* rm -rf /etc/pki/vdsm/libvirt-vnc/* vdsm-tool configure --force cp /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem cp /etc/pki/vdsm/keys/vdsmkey.pem /etc/pki/vdsm/libvirt-spice/server-key.pem cp /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem cp /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem cp /etc/pki/vdsm/keys/vdsmkey.pem /etc/pki/vdsm/libvirt-vnc/server-key.pem cp /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem cp /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/libvirt/clientcert.pem cp /etc/pki/vdsm/keys/vdsmkey.pem /etc/pki/libvirt/private/clientkey.pem cp /etc/pki/vdsm/certs/cacert.pem /etc/pki/CA/cacert.pem chown root:kvm /etc/pki/vdsm/libvirt-spice/ca-cert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem /etc/pki/libvirt/clientcert.pem chown vdsm:kvm /etc/pki/vdsm/libvirt-spice/server-key.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem /etc/pki/vdsm/libvirt-vnc/server-key.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem /etc/pki/libvirt/private/clientkey.pem /etc/pki/CA/cacert.pem chmod 644 /etc/pki/vdsm/libvirt-spice/ca-cert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem /etc/pki/libvirt/clientcert.pem chmod 440 /etc/pki/vdsm/libvirt-spice/server-key.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem /etc/pki/vdsm/libvirt-vnc/server-key.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem /etc/pki/libvirt/private/clientkey.pem /etc/pki/CA/cacert.pem systemctl restart vdsmd libvirtd
Virsh Credentials
To connect via virsh, you will have to run this localy to get into virsh. Alternitively, you can cat /etc/ovirt-hosted-engine/virsh_auth.conf and see the username and password
virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf
