Taking server virtualization down from the clouds to real life experience …
A myth busted and an FAQ: ESXi is *not* based on Linux! But what is it?
I have often heard or read the statement ESXi is based on Linux in this or some other variation when technical people discuss about how certain things work (or are supposed to work) in ESXi. This statement is not only plain wrong, but just doesnt make any sense! In fact it is like saying that
horses developed from cows, because both have a head, a tail and four legs
In this post I will explain why, and I also answer some other questions that are closely related to this discussion. The first one – surprisingly – is …
Linux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution.(Source:Wikipedia). It is important to understand that there is notoneLinux (like there is e.g. one Microsoft Windows), but a multitude of it, each assembled and distributed by different parties: There are community supported free distributions like Debian, Ubuntu and Fedora and commercially licensed and supported distributions by e.g. Redhat and Novell/SuSE.
All Linux distributions have one thing in common: The Linux kernel. The Wikipedia article quoted above calls itthe defining component of Linux. Almost every operating system is built upon a kernel which is a software layer that manages all hardware of a computer – it provides e.g. CPU scheduling, memory management and device drivers on the one hand, and defined interfaces to the application processes that run on top of it on the other hand.
The first version of the Linux kernel was authored byLinus Torvaldsin 1991. He is commonly referred to as the creator of Linux.
There is also a wide range of free and Open Source software that you will usually find in any Linux distribution: The X window system for graphical applications, desktop environments like Gnome and KDE, the web browser Firefox, and server services like the Apache http server, etc.
The answer to this question is a clearNo, because ESXi isnotbuilt upon the Linux kernel, but uses an own VMware proprietary kernel (the VMkernel) and software, and it misses most of the applications and components that are commonly found in all Linux distributions.
What do ESXi and Linux have in common!?
So ESXi is not based on Linux, but why do many people believe this? I think the main reason for this is that you can remotely open a terminal session on an ESXi host via thesshprotocol (if it has been enabled). And if you are in such a session you can use lots of commands (likecd,ls,df,grep, psetc.) that are also available in a Linux (or other Unix-) shell.
This is possible, because a software calledbusybox- a lightweight shell that is able to interpret and execute these commands – runs as a userworld of the VMkernel. But busybox and all other Unix-like tool that are available in an ESXi shell arenot Linux things. They are available in any Unix-like operating system and can even run on Windows (e.g. in aCygwinruntime environment)!
If you have ever taken a closer look at the ESXi internals that the shell exposes then you might have noticed more similarities to Linux (and other Unix-like operating systems), e.g.:
The typical directory structure with e.g.
A SysV-styleinitprocess that uses e.g. the configuration file
Shared Libraries of the GNU C compiler in
In ESXi 5.1 there is even theXorgX server installed! I believe that it is used to implement the new
feature in vSphere 5.1 (see number 5 in myvSphere 5.1 top new features post)
ESXi even uses the same binary format for executables (ELF) than Linux does, so it is really not a big surprise anymore that you can run some Linux binaries in an ESXi shell – provided that they are statically linked or only use libraries that are also available in ESXi! (I exploited this feature when describinghow to run HPs hpacucli tool in ESXiand when buildingthe ProFTPD package for ESXi).
After reading all this you might wonder How can I tell that ESXi isnotbased on Linux then?. It is because all these concepts and software exists independently from Linux (and the Linux kernel!) and are used inmanyoperating systems, resp. are implemented, maintained and distributed as OS-independent, autonomous packages. VMware just re-used all these mature concepts and software and this was in fact way smarter than re-inventing all these wheels! That means: ESXi and Linux are not based on each other, but share a common fundament.
But you can even use Linux device drivers in ESXi!
Not directly … you cannot use binary Linux driver modules in ESXi. Lots of Linux device drivers can be adapted to ESXi though by modifying their source code and compiling them specifically for ESXi. That means that the VMkernel of ESXi implements a sub-set of the Linux kernels driver interfaces, but also extends and adapts them to its own hypervisor-specific needs.
In my opinion this was another very clever move of the VMware ESXi architects and developers, because it makes it relatively easy to port an already existing Linux driver of a hardware device to ESXi. So the partners that produce such devices do not need to develop ESXi drivers from scratch. And it also enables non-commercial community developers to write device drivers for devices that are not supported by ESXi out-of-the-box!
However, you should be aware that the VMkernel only implements those driver interfaces that are necessary for acting as a hypervisor: that is mainly the network and storage stacks. You will e.g. not be able to easily port a driver for a scanner device or non-standard serial hardware to ESXi. The same applies to drivers for wireless LAN adapters, because they depend on kernel subsystems, other low-level drivers or management tools that are not available in ESXi and the VMkernel.
Another clearNohere! An operating system may only be called a UNIX if it fully implements theSingle UNIX specificationof theOpen Groupand has passed the associated certification process. ESXi does not (and was probably never meant to) implement all the necessary functionality and APIs that make up the UNIX standard.
You might call VMware ESXiUnix-like, but even this is disputed by official sources: Mike Foley – a Senior Technical Marketing Manager at VMware – recently explained in an interesting blog post titledIts a Unix system, I know thisthat A shell does not an OS make. His main point is that you do (or at least should) not manage an ESXi host through the shell, but via the management tools and scriptable APIs that are available for it, e.g. through PowerCLI. Or in other words: ESXi is not designed as a Unix, but as an embedded system that is managed through defined interfaces (of which most are not really Unix-like).
I used many words here to describe what ESXi isnot. But how do explain what it actuallyis? Here is my try: ESXi is
a bare metal (or type-1) hypervisor
incorporates free Open Source software components, but also uses
a unique proprietary OS kernel – the VMkernel –
which implements unique featuresthat are not available in any other operating system(e.g. the VMFS andVisorFSfile systems.
Wait … classic ESX is different – Thatisbased on Linux, right?
Those of you who have ever touched or even installed a classic ESX installation might know that it uses a Redhat Linux installation to boot itself and to provide management functionality. This Linux is the so-calledService Console OS(or shortCOS) of ESX. Its Linux kernel loads the VMkernel as a module and then passes complete control over the system to it. The VMkernel hypervisor in turn will then run the COS as a privileged VM.
So classic ESX really incorporates a Linux and depends on it for getting started and managed. According to this particular meaning you can really say that classic ESX is based on Linux, but that does not mean that the VMkernel and the VMware specific OS components developed from Linux. It just uses it as a piggyback.
The classic ESX architecture has probably very much boosted the perception that ESX(i) is based on Linux. But with the rise of theESXiarchitectureand the associated elimination of the COS the classic ESX architecture is obsolete now. The VMkernel of ESXi boots directly on the hardware and does not need a Linux OS anymore.
However, the classic ESX architecture specifically caused a lot of concerns and very controversial discussions about licensing topics and legal issues, and that brings us to the last question:
As explained above ESX and ESXi incorporate quite a few Open Source software that is licensed under theGNU Public License (GPL). The most important terms and conditions of the GPL are the following:
The GPL grants the right (or guarantees the freedom) to use, study, share (copy) and modify the software (source: Wikipedia).
Binary distributions of the software must be accompanied by its source code (to enable 1.).
In contrast to general believe GPL licensed software does not need to be provided for free (i.e. at no cost), butit can be sold.
Software that was derived from GPL licensed software must be itself licensed under the GPL.
VMware fulfills condition (2) by providing the source code of all GPL licensed software that is used in ESX(i) (and other VMware products). The Open Source code of the products are usually made available together with their binary downloads, see e.g. thedownload page of vSphere 5.1.
Back in 2008 there were voices claiming that the VMkernel is a derived work of the Linux kernel (and thus violates condition 4.), because of the fact that the VMkernel is loaded and initialized as a Linux kernel module in classic ESX. A post atsuggested this and got a lot of media coverage and lead to very lively discussions involving prominent people like the Linux kernel hackerAlan Coxand VMwares Zachary Amsden.
The accusation that VMware violates the GPL became obsolete with the architecture change towards ESXi, and – up to now – it was never decided in front of a court, because no one ever sued VMware for this. That means there is no proof as of today that VMware ever violated the GPL.
This post first appeared on theVMware Front Experience Blogand was written byAndreas Peetz
Follow him on Twitterto keep up to date with what he posts.Labels:ESXiGPLLinuxmyth debunkingVisorFSVMkernel
The answer to this question is a clear No, because ESXi is not built upon the Linux kernel, —
Ahhh, but leaked source code about a year ago does indicate it was built on the Linux kernel.
Can you point us to a source for this claim?
The leaked source code indicates no such thing. Andreass description of the relationship between Linux and ESX is very accurate.
There have been various reports of this ESX source code leakage back then. That caused security concerns, but there were no reports of Linux source code findings in there.
Well, I tend to believe that VMkernel is anyway a customized Linux kernel, probably heavily patched. But I have no proof of this, just a personal opinion =)
I had ESXi v5.0 crash on me (hardware issue) and have it display a line with CENTOS in it!
I dont believe for a minute its not *based* on some ported form of Linux, that been heavily customized, and stripped down. Ok, maybe it shouldnt be considered Linux then…
And this could not have been caused by a CentOS running on ESX?
Funny, as SSHing into one of my 5.1u1 boxes and running uname -a reports:
VMkernel **** 5.1.0 1 SMP Release build-1312873 Sep 6 2013 21:52:18 x86_64 GNU/Linux
Not only that, the HP ESXi ISO image dump shows: [ 0.006999] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.11.0-14-generic 21-Ubuntu
(logged from a failed attempt at installing onto a HP Supermicro Gen8)
You see this, because the development environment of the VMkernel and ESXi software is Linux – that means a Linux system and the GNU C compiler is used to compile the VMkernel. It also makes use of some of the Linux kernel include files to implement the Linux compatible driver interface.
Thanks for this post.. hit the nail right on the head as I also thought VMware was using a Linux based kernel
They say that it was initially based on Linux but then modified.
Could you enlight us regarding this topic.
No, you do not find this sentence (or a similar statement) there!
This Wikipedia article describes the classic ESX architecture in pretty much the same way than I do in my post. It also clearly distinguishes between the Linux kernel of the service console (that is also used to boot classic ESX) and the VMkernel that is then loaded and takes over control.
Ok, I probably made a shortcut in my reading 😉
Rereading your article I think wikipedias article may need some clarifications !!
As I do not have the knowledge, I suggest you help them out a bit 😉
Running on ESXi 5.5 I ran uname -a form ssh
VMkernel x.x.x.x 5.5.0 1 SMP Release build-2068190 Aug 22 2014 19:00:51 x86_64 GNU/Linux
uname is a link to the busybox shell that ESXi uses. The term GNU/Linux that it outputs here is hard coded in the busybox source code. VMware should have changed the busybox source code so that it displays something else, but they didnt.
However, this is in no way a proof that the VMkernel is based on the Linux kernel.
Sorry. I am still not convinced, that the ESX / ESXi / VMkernel is not based on Linux.
I still somehow got the impression, that VMware took a Linux Kernel and stripped it down to the bare minimum they needed.
And Andreas Peetz, you never clearly stated, that VMware has indeed written the Kernel from scratch. ANd your wording sounds like just dancing around a clear statement:
…not built upon the Linux kernel
…because it is not using the full kernel.
An operating system may only be called a UNIX if it fully implements the Single UNIX specification of the Open Group and has passed the associated certification process.
…about the same! It does NOT implement ALL UNIX requirements and therefore may not be called UNIX.
So all your explanations just did the opposite for me. They just proved my impression, that VMware just took Linux and stripped it down. So not using the complete Kernel, ESX(i) is not built UPON Linux and not a CERTIFIABLE UNIX neither.
That is the impression Ive got!
Unfortunately you do not add anything new to the discussion, but just state your personal impressions.
The fact that you are doing this anonymously doesnt make it better.
I welcome you back if you have any reliable source or proof of your claims to share.
I really wanted to argue with you when I first read your post… but the more I thought about it, the more I couldnt. You are correct in stating that what defines an OS as being linux is the kernel. Everything else (at least for the linux OS – definitely doesnt hold true for windows or OS X) is additional programs that do not define the OS. This is how android can still be linux even though the user interface is totally different – it uses the linux kernel.
As I thought about what defined the linux kernel, I came to one conclusion that it is the GPL opensource code within it. VMWare has not released or acknowledged using any of the linux code. Unless VMWare is doing something illegal, the VMKernel must have been re-written from the ground up. Just because it uses the same specs as the linux kernel does not make it linux. That said, it wouldnt be the first time a company used GPL linux code within a proprietary system (remember linksys/cisco?).
Finally we will probably know more about whether VMKernel is based on Linux source code or not!
Software Freedom Conservacy is funding a lawsuit in this matter against VMware. See:
It is a fact (and VMware does not deny) that the VMkernel uses GPLed driver code from the Linux kernel.
What is argued in front of a court now is whether this makes the VMkernel itself (and ESXi as a whole) subject to the GPL.
This is the Open Source Code used in the VMkernel and published by VMware. So what?
Well, I think this is more about philosophy, or even logic.
If you call operating system a means of transportation, then you have linux (truck), windows (motorbike), OSX (bike), UNIX (passenger car)…etc…
Then in trucks you have brands like Mercendes – Ubuntu, CentOS – Volvo, etc…
So if someone will build a truck using same philosophy and rules to transport other trucks then what would you call it ? I say truck.
Especially when it uses some common parts (GPL etc) as other trucks to make it work. And those parts are not used in other means of transportation like motorbikes (Windows).
So simply put for me linux is a category and vmware is in it.
If you build OS that is communicating with firmware/hardware in a new way that make common parts used in other linux systems unusable in it. That is totally different you can call it a new category – and not linux.
I like your vehicle analogy and would say that the ESXi truck does not share the engine (= kernel) with the Linux truck, but only some of the engines interfaces / control mechanisms. Indeed it does share some parts that almost every truck has, but it also has some very unique parts that only the ESXi truck has (like this gadget to transfer goods from one ESXi truck to another while both are driving – VMotion :-))
While you might say that both Linux and ESXi are trucks Id still stay that the ESXi truck is not derived from (or based on) the Linux truck.
You can directly program in c code and access most lower level functions, then compile and run the code directly in ESXi. From my point of view, if it has a duck beak and hair, it is a platypus.
Solarisallows running native Linux binaries. So doesFreeBSD. Heck, evenWindows 10allows this now!
Or do you – as a zoologist – still have a lot to learn about Operating Systems?
The author seems to make the case that VMware didnt base this on Linux, and thus VMware built something from the ground up.
Due to lack of info, youre able to semantically argue your point; however, ESXi is very obviously and so heavily based on a Unix-style OS, that youd be hard-pressed to prove that there isnt direct bit-for-bit code all over the VMkernel.
This code originated somewhere *outside* of VMwares developers minds, and therefore should be considered borrowed.
VMware does not keep it secret that it has borrowed (Id say re-used) a lot of GPLed Open Source Code, and – to comply with the GPL terms – they have also published all this code.
You seem to mistakenly believe that all Open Source software is somehow Linux based or belongs to Linux, only because it is *also* used in Linux. This is plain false. Almost all of the Open Source applications used in ESXi exist independently of Linux and are also used in other Unix-like derivates (like FreeBSD and even commercial Unix OSs).
A well documented exception to this is ESXis compatibility layer to Linux kernel device drivers, but I also explained that already in my post.
So, honestly, I just dont get your point.
I wasnt at all insinuating that all open source software was based on Linux. I was simply stripping down the argument to borrowed code (open source) vs. non-borrowed code.
Im not saying its illegal, or that you shouldnt trust them. I was simply stating they didnt build anything from the ground up.
That being said, Im not real sure why I was making that point, and apologize if I came off as combative.
Few points here: Unless we can see the full source code of the VMKernel then we cannot say for sure if it is based on the Linux kernal or not. Dogmatically stating it isnt without having seen the source code and comparing it not proof that it isnt.
Personally I do suspect that it is a rebadged stripped down Linux or BSD kernel for one reason: Both BSD and Linux are very stable efficient kernels with open (do one degree or another) code. Why go to the effort of reinventing the wheel when the world and his wife embeds particularly the Linux kernel into many devices. It just seems logistically easier to do so. Of course, I havent seen the code either so I cannot state it IS, but my suspicion still lies in it being derived from the Linux kernel or the BSD kernel.
The only issue of course is the closed nature of the VMKernel… if it IS based on Linux then the code should be released too, I am not so sure about BSD however because that is not released under the GPL, but the BSD License.
Personally, it doesnt matter one way or another to me but it would be interesting to KNOW with proof one way or another.
the Linux kernel is a pretty universal multi-purpose Operating System kernel, whereas the VMkernel was developed as a very lightweight specialized Hypervisor-only kernel that has unique features like the VMM (Virtual Machine Monitor), the VMFS file system, VMotion (Live Migration) code etc.
This is why it would have made more sense to develop the VMkernel from scratch instead of stripping down a Linux kernel.
But maybe you are right. In the end its about trust or distrust. I personally do not have any reason to distrust VMware. And although you and me are not able to inspect the full VMkernel source code Im sure that other third parties have been able to do this, e.g. in the ongoing GPL lawsuit against VMware.
What a marvellous, well put together article, thank you! (The comments section is interesting to read as well.) Aside from debating any finer nuances, this overall understanding of some of the core architecture and even how it differs from ESX/COS has been extremely helpful in approaching ESXi and VMWare for the first time.
***** All comments will be moderated! *****
– Please post only comments or questions that are related to this posts contents!
– Advertising and link spamming will not be tolerated!
How to make your unsupported NIC work with ESXi 5.x or 6.0
[Update] ESXi-Customizer-PS 2.5 – The ESXi image customization script
Active Directory issues with ESXi 6.0 Update 2 and an automated fix
How to deploy Windows Nano Server (TP5) on vSphere
[Unsupported] Defeating the VCSA shell timeout
Re-pointing vCenter Server 5.5: A Survival Guide to KB2033620
ESXi 6.5 Release Notes for free license and white box users
VMware raises the bar again: The must-know new fea…
A myth busted and an FAQ: ESXi is *not* based on L…
[Release] Veeam Backup & Replication v7
Cool free tool (not only) for the lab: MobaXTerm
[Release] vCenter Server 5.1 Update 1b