One of the first things that stands out to people when they first start playing around with unikernels is that the concept of an interactive server that you can pop into and start running all sorts of arbitrary programs (also known as 'commands') is supplanted by just one program - yours. So even though it's running 'as a server' it's really just your program running much the same way you might start it normally on your laptop. It's through the magic of virtualization that we can even do this.
Engineers have developed different patterns to deal with configuration differences and environment changes. Some will create a package which is essentially just a tarball of a root filesystem layout and some configuration dressing on top. Some create multiple packages per deploy target while others provide an additional config to be used for each deploy in their CI system. Done this way images are created on-demand at deploy time, which for most public clouds is how things work. This sounds like it would take a long time but a deploy done this way to Amazon only takes tens of seconds to build a brand new AMI and spin it up as an instance.
Ops run is pretty fast and by default re-generates images on the fly which is great for dev/test but no one uses 'ops run' in production. Even if you are running your own infrastructure you'd still use 'ops image create' and 'ops instance create' - just with the onprem target instead.
Other options exist too such as the cloud_init klib which allows you to inject env vars at boot time. This is a common practice to slurp down secrets such as TLS certs. We've even added vsock support in the past.
However there are still some cases where you might want to dynamically configure various options even before booting without having to rework the image and without having to hook up a secret store.
Enter Firecracker
Nanos has had firecracker support for quite a long time now. I originally wrote about it a few years ago and we of course have documentation on booting unikernels with firecracker. Back then a lot of people weren't necessarily using firecracker as most of our existing users were on the cloud and you need virtualization for firecracker but now we are starting to see more and more firecracker based platform-as-a-services pop up and we wanted the ability to make that experience even better.
One of our users noticed that it was increasingly common for people to pass in configuration via the kernel init line with firecracker and was wondering why we didn't support it. The answer is relatively simple. 99% of our current user base deploys to GCP, AWS, Azure or some other public cloud. When you deploy to these clouds you typically don't have the capability to inject various kernel parameters into the image at boot time so we've just done things differently.
To cut to the chase - we added support for changing arbitrary settings in the root tuple via the kernel command line, which is retrieved by the kernel when booting under AWS Firecracker.
Example Time
This capability is a lot more extensible than one might think at first blush because of Nanos unique TFS (tuple file system).
Even this simple hello world go app which we dumped shows the mix of fs and metadata:
eyberg@venus:~$ ~/.ops/0.1.48/dump ~/.ops/images/g
detected filesystem at 0xc10600
Label:
UUID: 73f8c8d6-df99-c3a0-0942-a0bf8dc25874
metadata
(..:
arguments:[g]
booted:
children:(etc:(..:
children:(passwd:(..:
extents:(0:(allocated:1
length:1
offset:5973))
filelength:33)
resolv.conf:(..:
extents:(0:(allocated:1
length:1
offset:5972))
filelength:20)
ssl:(..:
children:(certs:(..:
children:(ca-certificates.crt:(..:
extents:(0:(allocated:406
length:406
offset:5566))
filelength:207436)))))))
g:(..:
extents:(0:(allocated:13458
length:13458
offset:5974))
filelength:6890290)
lib:(..:
children:(x86_64-linux-gnu:(..:
children:(libc.so.6:(..:
extents:(0:(allocated:4049
length:4049
offset:1464))
filelength:2072888)
libnss_dns.so.2:(..:
extents:(0:(allocated:53
length:53
offset:5513))
filelength:26936)))))
lib64:(..:
children:(ld-linux-x86-64.so.2:(..:
extents:(0:(allocated:439
length:439
offset:1025))
filelength:224376)))
proc:(..:
children:(self:(..:
atime:7313690904789804366
children:(exe:(..:
atime:7313690904791658778
linktarget:/g
mtime:7313690904791658778))
mtime:7313690904789804366)
sys:(..:
children:(kernel:(..:
children:(hostname:(..:
extents:(0:(allocated:1
length:1
offset:19432))
filelength:7))))))))
environment:(IMAGE_NAME:g
NANOS_ARCH:amd64
NANOS_VERSION:0.1.48
OPS_PORT:8080
PWD:/
USER:root)
program:/g)
However, we can theoretically change any configuration you might find in our extremely extensive configuration documentation. Typically, this would constitute an image rebuild, even if an 'ops run' is more or less instant. Now, an operator can take the same image and dynamically change this at boot time, which becomes very handy if you are an operator of a firecracker based paas.
Let's take a look at few examples using the firecracker vm_config.json following along from our documentation:
Changing an IPv4 Address:
"boot-source": {
"kernel_image_path": "/Users/bob/.ops/nightly/kernel.img",
"boot_args": "en1.ipaddr=10.3.3.6 en1.netmask=255.255.0.0 en1.gateway=10.3.0.1"
}We can put a lot of different things in that 'boot_args' config.
Change an IPv6 Address:
en1.ip6addr=20::A8FC:FF:7600:AAOverwrite an Existing Env Var:
environment.PROD=1Adjust a New Argument:
arguments.0=/python3 arguments.1=my_new_entrypoint.pyYou Can Even Turn on Debug Support Dynamically:
"boot_args": "trace=t debugsyscalls=t"Stop saying that! YoU CaNT DeBUG unIkernelSZ!!!!
Getting Started
Ops was built to make building and deploying unikernels easily and takes care of networking locally if you're just using ops. However, in this example we want to use firecracker so there's a different flow. If you want to try it out - let's create a new network (bridge) first:
ops network createThis will create new bridge and a dhcp listener along with it. Then we'll add a tap interface to it.
sudo ip tuntap add dev tap0 mode tap
sudo ip link set tap0 up
sudo brctl addif br0 tap0From there we can start firecracker normally and read any of our logs.
./read_fifo.sh log.fifoAs with most changes in Nanos we typically end up making several other types of changes to get the changes we actually want done.
For example we had to add a new region to hold the command line itself:
#define REGION_INITIAL_PAGES 10 /* for page table allocations in stage2 and early stage3 */
#define REGION_CMDLINE 11 /* kernel command line */
#define REGION_FILESYSTEM 12 /* offset on disk for the
filesystem, see if we can get disk info from the bios */
We had to make a change in the id heap which in return made us have to make a change in the bitmap allocator to ensure alignment. These changes allowed us to remove padding which was wasteful.
Another change we made was to modify the tuple_notifer to operate on a composite (tuple or vector) value without modifying nested values. This was necessary to add the ability to modify the root config tuple.
This is all to say that a seemingly innocuous, yet reasonable, feature request can both touch quite a lot, yet at the same time deliver immense benefits.
Are you using firecracker? Are you building a firecracker Paas? If so reach out.
