Xen Event Handlers

One of our users reached out stating that he was seeing a t2.medium crash rather quickly on AWS. By default the instance flavor we spin up if you don't specify is a t2.micro which works well under the free tier on Amazon but he was doing some benchmarking and so was probably wanting to exercise more than 1 vcpu.

After pulling out wrk I was able to instantly reproduce what he was seeing.

Basically what was going on was that events for event handlers were being shipped to different vcpus other than the one that it was bound to. You might know that AWS has been creating a lot of their newer instance types basd on Nitro but they still have many instance types based on Xen such as the t2.medium. Different instance types can have different drivers and in this case even different hypervisors so some bugs might only manifest on certain instance types.

What are Events?

Events in this sense are kind of like interrupts that xen uses. They store one bit and when you flag it you are notifying the guest that an event has arrived. You won't get other notifications until your clear it.

Events are slightly different than interrupts though as they deal with virtual machines. They can be sent both ways and have the concept of a connection. From a permission point of view when an event channel is created it must include the domain that it is allowed to bind to. The other domain must also request binding to the other end.

Nanos doesn't support 32 bit so all xen guests will have 1023 event channels by default but could go all the way up to 131,071 if necessary.

Events are passed along event channels which are shared memory fifo queues shared between the host and guest.

You can have a few different types of events such as VIRQs (interrupts that are for virtual devices), PIRQs (interrupts from a physical device that get virtualized), IPIs (events between multiple vcpus on the same guest), unbounded ones and interdomain ones.

This change also allowed us to spread the interrupt load amongst the various vcpus by allowing drivers to request a handler be bound to a certain cpu. Before it was assumed it was only going to one.

To enable this it was necessary for us to do some musical chairs by decreasing memory in some other places, such as making vcpu_evtchn_enable dynamically allocated. This enabled us to spread this out to up to 32 vcpus (on xen).

AWS alone has over 850 instance types with varying flavors of drivers, hypervisors, etc. but we support it all. If you're ever running a nanos payload and you see a crash like this reach out and know that our customers have private issue reporting available as well.