Xen: a view from the trenches

From PrgmrWiki


Virtualization and Consolidation

what are the tradeoffs between the different virtualization technologies when using them for consolidation? and what about consolidation without virtualization?

We started out using OS level Virtualization (FreeBSD Jails) but we had a difficult time keeping users in check. Our heavy users would eat the entire box, so that performance was abysmal for our light users. Also, many users want things like firewalls and kernel modules.

We've all heard the hype about virtualization and how it saves you money and power. How can this be, considering that even the most efficient virtualization technology has overhead?

The thing is, Virtualization is nothing more than a shortcut to consolidation. It should be pretty clear to all of you when consolidation will save you money and when it won't.

Virtualization is an easy way to preserve administrative and OS-level partitioning that you previously had running your app on a bunch of smaller servers. If you wanted to figure out how to make all the apps you are consolidating work on the same OS and the same instance and not have the admins/libs step on oneanother, consolidating without virtualization is almost always more efficient, from a hardware perspective.

However, shortcuts are important. Hardware is cheap. Hell, I'll rent you a server with 8 cores, 32GB ram and 2x1TB sata disks for $512/month. And you can buy such an animal for quite a lot less than $3000 in parts. So from my perspective, wasting a couple gigs of ram is a bargain. Registered ECC ddr2 is about $25/gigabyte. SysAdmin time is usually between $50 and $100 an hour. You can see how it doesn't take many saved hours to make even horribly inefficient virtualization systems make sense.

full virtualization

  • Full virtualization provides 'better' (that is, closer to a real machine) virtualization,
  • slower
  • more complex to implement (thus is less secure.)
  • qemu, vmware, xen, and kvm


  • This is what we use
  • lower overhead than full virtualization
  • higher overhead than os level virtualization
  • each user can have his/her own kernel and modules
  • kernels must be modified to work with the paravirtualization scheme
  • kernel.org kernels support running under Xen (as a DomU) since 2.6.23

OS level virtualization (jails)

  • linux vserver, openvz, FreeBSD jails.
  • low overhead
  • weaker partitioning
  • shared kernels

overview of Xen device architecture

Networking and Xen

  • network-bridge
  • vifs

Storage and Xen

  • LVM backing devices
  • vbds in domU

partitioning: protecting users from one another

"Tragedy of the commons"

scheduling: giving each user a fair share of CPU

Xen uses the credit scheduler to give each domain a CPU weight and cap.

The credit scheduler


  • work-conserving
  • lightweight
  • accounting thread passes out credits periodically
  • weight=memory
  • dom0 weight

ram partitioning

  • fixed dom0 allocation
    • boot parameter for max
    • xend-config.sxp entry for min
  • Xen does most of the work

hypervisor's memory usage

  • Trivial
  • Maps itself in 64MB at the top of all domains.
  • For most of our purposes, the hypervisor is kind of like a pipe between the domU and dom0.


Our issues with disk mostly have to do with activity, not allocation, since all allocation is handled in advance.



dom0 disk processes

  • normal process in dom0
  • tapdisk
  • xvd


Most users are pretty easy to deal with. Some users will take as much bandwidth as possible.

monitoring tools

  • SNMP-based
  • bandwidthd
  • snort



  • setting automatic limits

Any questions?