On the face of it, three day work weeks are pretty nice. But I stay busy, and cramming a week’s worth of productivity into three days is less fun than it sounds like. That said, I got done what needed doing, and the coming week is prepped and ready.
* * *
Last night, we went back for a second round of Annapolis Shakespeare‘s production of Much Ado About Nothing. With 17 actors and a two story set, there’s always more going on than one can take in at one sitting. Since opening night, the actors have really settled into their roles, and we enjoyed it even more, if that were possible. They’ve been getting stellar reviews all over the place and I can only say this: If you’re in the area, there are nine more productions of this show: today’s matinee and four shows each of the next two weekends. Get tickets and go!!!
* * *
The daylight hours yesterday were full, too. Much of the day, I puttered with virtualization on my main home server, a FreeBSD 11.1 box that does internal SMB, internal IMAP, backups, and virtual machine hosting. When I started with virtualization on the system, I was using Oracle’s VirtualBox product, because the price is right (free, as in beer), and easy, easy to setup and use. But easy isn’t always my primary goal. So I’ve been experimenting with the native virtualization tool on FreeBSD: bhyve.
“bhyve, the “BSD hypervisor”, pronounced “beehive” is a hypervisor/virtual machine manager developed on FreeBSD.”
I make use of the appropriate section of the FreeBSD Handbook to provide guidance. As such things go, it’s relatively simple to stand up FreeBSD virtual guests, and a bit trickier for Linux guests. I’ll document some of the fun I had with that here, because there are gotcha’s that aren’t covered in the Handbook.
The Setup
I’m going to build an Ubuntu 17.04 virtual machine (VM), using a ZFS volume as a datastore. The use of ZFS is recommended for performance reasons. There are other advantages, too, like the ability to make quick clones of a VM. More on that later. So, my configuration is this:
root@serenity:// > ls /data/bhyve
images iso
root@serenity:~/ > zfs list zroot/data/vmimages
NAME USED AVAIL REFER MOUNTPOINT
zroot/data/vmimages 52.9G 1.07T 96K /data/vmimages
/data/bhyve/images is actually where I keep the runtime configuration and startup scripts for virtual machines.
/data/bhyve/iso is the repository for CD images for installation of virtual machines.
The ZFS path zroot/data/vmimages is the parent for all of my virtual machine disks.
I’ve also already done the initial networking setup with bridge and tap0 interfaces, per the Handbook sub-section, “Preparing the Host.”
Preparation
Create and check the VM disk:
root@serenity:/data/bhyve/images/ > zfs create -V16G -o volmode=dev zroot/data/vmimages/ub1704new
root@serenity:/data/bhyve/images/ > ls -al /dev/zvol/zroot/data/vmimages/ub1704new
crw-r----- 1 root operator 0x9b Oct 15 13:59 /dev/zvol/zroot/data/vmimages/ub1704new
With the disk volume in place, I can create the device map file, which sets (hd0) to the path to the new disk volume I created, and (cd0) to the path to the ISO file (vim is the text editor I use):
root@serenity:/data/bhyve/images/ > vim ub1704new-device.map
root@serenity:/data/bhyve/images/ > cat ub1704new-device.map
(hd0) /dev/zvol/zroot/data/vmimages/ub1704new
(cd0) /data/bhyve/iso/ubuntu-17.04-server-amd64.iso
Note that when a VM is or has been running, it creates an entry in the device tree, at /dev/vmm. Normally, one must always “destroy” that file before one can start/restart the VM (seems clunky, but there it is). But because this is the first time this VM will have been run (on creation), there should be no corresponding device file at /dev/vmm/ub1704new. I’ll check that, then create the VM using the grub-bhyve tool, which prepares the boot environment for the VM:
root@serenity:/data/bhyve/images/ > ls /dev/vmm/ub1704new
ls: /dev/vmm/ub1704new: No such file or directory
root@serenity:/data/bhyve/images/ > grub-bhyve -m ub1704new-device.map -r cd0 -M 1024M ub1704new
...
GNU GRUB version 2.00
+--------------------------------------------------------------------------+
|Install Ubuntu Server |
|OEM install (for manufacturers) |
|Install MAAS Region Controller |
|Install MAAS Rack Controller |
|Check disc for defects |
|Rescue a broken system |
| |
| |
+--------------------------------------------------------------------------+
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, `e' to edit the commands
before booting or `c' for a command-line.
root@serenity:/data/bhyve/images/ > ls /dev/vmm/ub1704new
/dev/vmm/ub1704new
The “Install Ubuntu Server” line was highlighted, so I simple pressed the Enter key to accept that option. Disconcertingly, one is then dropped right back onto the command line. This is expected, however. And as you can see, we now have a VM entry for the new guest under /dev/vmm.
The next gotcha is this: There has to be a free tapN interface for the VM to attach to. The documentation wasn’t really clear on that, I think I assumed that multiple VMs could attach to a single tap interface. But in reality, think of the bridge interface as the virtual switch, and each tap interface as a port on that switch. So, let’s check if tap0 is in use:
root@serenity:/data/bhyve/images/ > ifconfig | egrep "^tap[0-9]+:"
tap0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
tap2: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
As you can see, I’ve run into this problem already, and have a couple of spare taps available. This output shows that of the three tap interfaces, tap0 and tap2 are available, while tap1 is in use (see the word UP in the flags). For the purposes of this exercise I’ll just use tap0. But it’s trivial to add more tap devices on the fly, and to add them to the /etc/rc.conf file so that they are present for future runs. In a super-happy world, my VM automation script will look for any available tap device, and use one if found, otherwise dynamically add yet another one and use it. But that’s another post.
Install Time
So, it’s time to start the VM for the first time. Important note: One should set the amount of memory for the bhyve run to match the amount one set with grub-bhyve, or errors ensue. Observe that the memory setting with grub-bhyve above uses the -M flag, and a trailing M. The bhyve command uses a -m flag, and Megabytes are assumed.
I’m going to give the VM two processors (it can certainly take advantage of two, even during the installation)
root@serenity:/data/bhyve/images/ > bhyve -c 2 -m 1024 -H -P -A -s 0:0,hostbridge -s 1:0,lpc \
-s 2:0,virtio-net,tap0 -l com1,stdio -s 3,ahci-cd,/data/bhyve/iso/ubuntu-17.04-server-amd64.iso \
-s 4,virtio-blk,/dev/zvol/zroot/data/vmimages/ub1704new ub1704new
...
┌───────────────────────┤ [!!] Select a language ├────────────────────────┐
│ │
│ Choose the language to be used for the installation process. The │
│ selected language will also be the default language for the installed │
│ system. │
│ │
│ Language: │
│ │
│ C │
│ English │
│ │
│ <Go Back> │
│ │
└─────────────────────────────────────────────────────────────────────────┘
<Tab> moves; <Space> selects; <Enter> activates buttons
And so starts the text-mode Ubuntu installer. I’m going to assume you can find your way to figuring that out or find useful directions on the interwebs. A couple of installation tips:
- The installer configures networking using DHCP by default. It’s easy to change to a static IP later, if desired.
- Hostname entry – I generally use the name of the virtual machine I created. It’s just easier to keep straight in my head that way.
- Partitioning – I’ve gone with “Guided – use entire disk and set up LVM”, but there are repercussions down the line. Manual isn’t hard, but can be confusing if you’ve not done much manual partitioning. LVM is a good choice because you can later add more diskspace to the volume(s) without even rebooting the system.
- Automatic updates – These can be a good idea, some of the time. But with servers, I tend to have process around patching, booting, and testing, so I selected No Automatic Updates.
- Software Selection – The only important choice for me at system installation is OpenSSH server: I need this to remotely administer any system: local or remote, physical or virtual.
Once the installer is done, there’s at least one more trick up my sleeve…
But first, we have to “destroy” the remnants of the prior run, then re-run grub-bhyve to figure out what our root and boot devices are:
root@serenity:/data/bhyve/images/ > bhyvectl --destroy --vm=ub1704new
root@serenity:/data/bhyve/images/ > grub-bhyve -m ub1704new-device.map -r hd0 -M 1024M ub1704new
...
grub> ls
(hd0) (hd0,msdos1) (cd0) (cd0,apple2) (cd0,apple1) (cd0,msdos2) (host)
(lvm/ub1704new--vg-swap_1) (lvm/ub1704new--vg-root)
grub> ls (hd0)/
error: unknown filesystem.
grub> ls (hd0,msdos1)/
error: unknown filesystem.
grub> ls (lvm/ub1704new--vg-root)/
lost+found/ etc/ media/ bin/ boot/ dev/ home/ lib/ lib64/ mnt/ opt/ proc/ root/ run/
sbin/ srv/ sys/ tmp/ usr/ var/ initrd.img vmlinuz snap/
grub> cat (lvm/ub1704new--vg-root)/etc/fstab
...
/dev/mapper/ub1704new--vg-root / ext4 errors=remount-ro 0 1
/dev/mapper/ub1704new--vg-swap_1 none swap sw 0 0
And there’s the information we need to configure a file to prime grub automatically, but first, let’s get this system running for the first time after installation:
grub> linux (lvm/ub1704new--vg-root)/vmlinuz root=/dev/mapper/ub1704new--vg-root
grub> initrd (lvm/ub1704new--vg-root)/initrd.img
grub> boot
root@serenity:/data/bhyve/images/ >
There’s our prep done, now to run the machine:
root@serenity:/data/bhyve/images/ > bhyve -c 2 -m 1024 -H -P -A -s 0:0,hostbridge -s 1:0,lpc \
> -s 2:0,virtio-net,tap0 -l com1,stdio -s 4,virtio-blk,/dev/zvol/zroot/data/vmimages/ub1704new ub1704new
...
Ubuntu 17.04 ub1704new ttyS0
ub1704new login:bilbrey
Password:
Welcome to Ubuntu 17.04 (GNU/Linux 4.10.0-19-generic x86_64)
...
bilbrey@ub1704new:~$
The next step is to update the freshly built system to with current packages and security updates, because the CD and DVD images are not respun every time there’s a changed package:
bilbrey@ub1704new:~$ sudo su -
[sudo] password for bilbrey:
root@ub1704new:~# apt update && apt upgrade -y
...
root@ub1704new:~# sync
sroot@ub1704new:~# sync
root@ub1704new:~# shutdown -h now
With that done, now I’ll create a couple of files to make the startup much easier – we’ll create a file to feed grub-bhyve what it needs, and a quick and dirty shell script to automate all the startup options and run the VM:
root@serenity:/data/bhyve/images/ > vim ub1704new-grub.in # pull together our grub info from the first startup...
root@serenity:/data/bhyve/images/ > cat ub1704new-grub.in
set root=(lvm/ub1704new--vg-root)
linux /vmlinuz root=/dev/mapper/ub1704new--vg-root
initrd /initrd.img
boot
root@serenity:/data/bhyve/images/ > vim start_ub1704new.sh # shell script to config and run
root@serenity:/data/bhyve/images/ > cat start_ub1704new.sh
#!/usr/local/bin/bash
#defaults
imgname="ub1704new"
imgpath="/dev/zvol/zroot/data/vmimages/${imgname}"
cpus=2
mem=2048
tap="tap0"
stkargs="-H -P -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,${tap} -l com1,stdio"
cd /data/bhyve/images
bhyvectl --destroy --vm=${imgname} # Clean up prior run
grub-bhyve -r hd0 -m ${imgname}-device.map -M ${mem}M ${imgname} < ${imgname}-grub.in # prep grub boot
bhyve -c ${cpus} -m ${mem} ${stkargs} -s 4,virtio-blk,${imgpath} ${imgname} # Run the VM
root@serenity:/data/bhyve/images/ > chmod 700 start_ub1704new.sh # Make the script runnable (by root)
All done, now I can just start the VM:
root@serenity:/data/bhyve/images/ > ./start_ub1704new.sh
...
Ubuntu 17.04 ub1704new ttyS0
ub1704new login: bilbrey
Password:
...
bilbrey@ub1704new:~$ sudo su -
[sudo] password for bilbrey:
root@ub1704new:~# sync
root@ub1704new:~# sync
root@ub1704new:~# shutdown -h now
Making Copies and Clones
Okay, a simple script run to start up the VM. That’s good. But we’ve put in a fair bit of work on this VM, what if I want some more of exactly that? I can use ZFS utilities to clone the VM image, do a couple of edits in copies of the files we just created, and we can have one or more copies without all the installation effort and pain. Here goes:
root@serenity:/data/bhyve/images/ > zfs list -rt all zroot/data/vmimages/ub1704new
NAME USED AVAIL REFER MOUNTPOINT
zroot/data/vmimages/ub1704new 16.5G 1.07T 3.32G -
root@serenity:/data/bhyve/images/ > zfs snapshot zroot/data/vmimages/ub1704new@copy1
root@serenity:/data/bhyve/images/ > zfs clone zroot/data/vmimages/ub1704new@copy1 zroot/data/vmimages/ub1704copy1
root@serenity:/data/bhyve/images/ > zfs list -rt all zroot/data/vmimages
NAME USED AVAIL REFER MOUNTPOINT
zroot/data/vmimages 72.9G 1.05T 96K /data/vmimages
...
zroot/data/vmimages/ub1704copy1 8K 1.05T 3.32G -
zroot/data/vmimages/ub1704new 19.8G 1.07T 3.32G -
zroot/data/vmimages/ub1704new@copy1 0 - 3.32G -
root@serenity:/data/bhyve/images/ > zfs get origin zroot/data/vmimages/ub1704copy1
NAME PROPERTY VALUE SOURCE
zroot/data/vmimages/ub1704copy1 origin zroot/data/vmimages/ub1704new@copy1 -
[* Editors note – Updated above to add the zfs snapshot command, which did not survive the original cut and paste]
This read/write clone, ub1704copy1, takes about as long as it takes to run the snapshot and clone commands – no time at all, really. But it will be dependent on the snapshot (see the output of the zfs get origin command), and not an independent copy of the VM. So for quick-and-dirty testing, this is a great tool. If, on the other hand, you want to make use of that snapshot to make a long-lived copy of the VM, use the ZFS send/receive functionality:
root@serenity:/data/bhyve/images/ > zfs send zroot/data/vmimages/ub1704new@copy1 \
| zfs receive zroot/data/vmimages/ub1704copy2
root@serenity:/data/bhyve/images/ > zfs list -rt all zroot/data/vmimages
NAME USED AVAIL REFER MOUNTPOINT
zroot/data/vmimages 76.2G 1.05T 96K /data/vmimages
zroot/data/vmimages/ub1704copy1 8K 1.05T 3.32G -
zroot/data/vmimages/ub1704copy2 3.32G 1.05T 3.32G -
zroot/data/vmimages/ub1704copy2@copy1 0 - 3.32G -
zroot/data/vmimages/ub1704new 19.8G 1.06T 3.32G -
zroot/data/vmimages/ub1704new@copy1 0 - 3.32G -
root@serenity:/data/bhyve/images/ > zfs get origin zroot/data/vmimages/ub1704copy2
NAME PROPERTY VALUE SOURCE
zroot/data/vmimages/ub1704copy2 origin - -
root@serenity:/data/bhyve/images/ > zfs destroy zroot/data/vmimages/ub1704copy2@copy1
Note that the send/receive ALSO copied the snapshot, so I disposed of the copied snapshot… The send/receive took a couple of minutes for this small VM. A much larger VM would take a correspondingly longer time. Let’s create the scripts to run ub1704copy2:
root@serenity:/data/bhyve/images/ > cp ub1704new-grub.in ub1704copy2-grub.in
root@serenity:/data/bhyve/images/ > cp ub1704new-device.map ub1704copy2-device.map
root@serenity:/data/bhyve/images/ > cp start_ub1704new.sh start_ub1704copy2.sh
root@serenity:/data/bhyve/images/ > vim *ub1704copy2*
...
root@serenity:/data/bhyve/images/ > diff start_ub1704new.sh start_ub1704copy2.sh
5c5
< imgname="ub1704new"
---
> imgname="ub1704copy2"
8,9c8,9
< mem=2048
< tap="tap0"
---
> mem=4096
> tap="tap2"
root@serenity:/data/bhyve/images/ > diff ub1704new-device.map ub1704copy2-device.map
1c1
< (hd0) /dev/zvol/zroot/data/vmimages/ub1704new
---
> (hd0) /dev/zvol/zroot/data/vmimages/ub1704copy2
root@serenity:/data/bhyve/images/ > diff ub1704new-grub.in ub1704copy2-grub.in
So, no changes to the grub.in file, as all things are the same, including the name of the LVM filesystem that is root. Remember, even though the VM is now ub1704copy2, it’s a copy of ub1704new, and will be until we run it, change the hostname, and make it different.
The device.map file has to change to point to the new ZFS volume, but that’s all.
And for the start_ub1704copy2.sh file, I really only had to change the imgname variable to make everything work. But I also bumped the memory up to 4G, and changed the network device to tap2, so that new and copy2 could be running simultaneously. Now let’s boot copy2, change the hostname, and boot it again:
root@serenity:/data/bhyve/images/ > ./start_ub1704copy2.sh
...
ub1704new login: bilbrey
Password:
...
bilbrey@ub1704new:~$ sudo su -
[sudo] password for bilbrey:
root@ub1704new:~# vim /etc/hostname
root@ub1704new:~# cat /etc/hostname
ub1704copy1
root@ub1704new:~# sync
root@ub1704new:~# sync
root@ub1704new:~# shutdown -h now
root@serenity:/data/bhyve/images/ > ./start_ub1704copy2.sh
...
Ubuntu 17.04 ub1704copy1 ttyS0
ub1704copy1 login: bilbrey
Password:
...
bilbrey@ub1704copy1:~$ ip addr show dev enp0s2
2: enp0s2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:a0:98:27:32:75 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.108/24 brd 192.168.1.255 scope global enp0s2
valid_lft forever preferred_lft forever
inet6 fe80::2a0:98ff:fe27:3275/64 scope link
valid_lft forever preferred_lft forever
Okay, we’re running in copy2, renamed the host, and we have an IP address. Let’s start up ub1704new, and ping the copy:
root@serenity:/data/bhyve/images/ > ./start_ub1704new.sh
...
Ubuntu 17.04 ub1704new ttyS0
ub1704new login: bilbrey
Password:
...
bilbrey@ub1704new:~$ ip addr show dev enp0s2
2: enp0s2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:a0:98:d4:48:eb brd ff:ff:ff:ff:ff:ff
inet 192.168.1.136/24 brd 192.168.1.255 scope global enp0s2
valid_lft forever preferred_lft forever
inet6 fe80::2a0:98ff:fed4:48eb/64 scope link
valid_lft forever preferred_lft forever
bilbrey@ub1704new:~$ ping 192.168.1.108
PING 192.168.1.108 (192.168.1.108) 56(84) bytes of data.
64 bytes from 192.168.1.108: icmp_seq=1 ttl=64 time=0.889 ms
64 bytes from 192.168.1.108: icmp_seq=2 ttl=64 time=0.652 ms
...
bilbrey@ub1704new:~$ ssh 192.168.1.108
The authenticity of host '192.168.1.108 (192.168.1.108)' can't be established.
ECDSA key fingerprint is SHA256:yARJTbiR8K2S1pTrYZ8xdDZawGMVqtukB3th2cf1Zjw.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.108' (ECDSA) to the list of known hosts.
[email protected]'s password:
...
Last login: Sun Oct 15 20:26:57 2017
bilbrey@ub1704copy1:~$
There we go. The clones and copies are super fast and easy. I’m told by the interwebs that there are tools called vm-bhyve and iohyve that might be useful, but those are for another day.
* * *
DoD announced no new casualties in the last week. Ciao!
