zir_blazer
Golden Member
From several years ago, I dreamed of using computers in a fully virtualized environment, where I can be the overseer of an Hypervisor, with the OSes themselves being something secondary and easily swapeable, or even being able to be used simultaneously. While virtualization has been around for years, the fact that there were a lot of Hardware pieces that still needed to be emulated, mainly the GPU, didn't allowed for a lot of flexibility, as you would still need to have a native Windows installation to do things like to play games, because you couldn't do that inside a VM. Advancements in the IOMMU virtualization area has made passthrough possible, that somewhat solves this by allowing you to give a Windows VM control of the GPU via VGA passthrough, which makes it pretty much fully functional even while inside a VM.
Yes, at first it seems to add complexity and overhead for seemingly no reason. What I want to accomplish with this? Eliminating Dual Boot. You see, one of the reason why I always through that I (And many other users) didn't even bothered to spend time in Linux, is because Dual Boot ain't for the lazy. If I can do all my everyday task on Windows, I wouldn't even bother to reboot to do in Linux what I can do in the current Windows session, because if I were to do that, I would need to reboot again if I wanted to play a game (You have Wine in Linux, but support and compatibility is not universal). Basically, while I always heared about Linux and know about some of its strengths, I never bothered with it because it couldn't fully replace Windows, and there were areas where Linux may have had an edge but it wasn't enough to even justify installing and learning to use it.
From December of last year, I managed to get quite close to what I want thanks to my new computer with VT-d support, having manage to get a fully functional Windows XP SP3 VM where I can play games, using Xen on top of an Arch Linux installation. After 3 months of using my Frankenstein setup I can say that stability and compatibility are near perfect (Not without some quirks, but almost there), and I would say that the whole thing is pretty much production-ready for power users that look at this idea the same way I do. This way, you can get the most of each OS strength by using them simultaneously. While making it work was quite an accomplishment itself, fine tuning such a complex setup seems to be even harder, from both configuration and usage sides.
I was intending to throw an extremely, extremely long wall of text bomb. You will get it anyway on the long run - this is just the tip of the iceberg. But I decided that dividing it in parts focusing on one aspect of the setup each time, would make it more palatable. Overally, the points I am interesed in, are:
1 - Set in stone the storage subsystem, that due to the fact that is the hardest thing to change after you get it working, should be also the most important to take care of
2 - Proper setup of a slim, lightweight Dom0 (Which currently is Arch Linux + Xen, however, I should re-check XenServer because there are some free versions with nearly all features), that should be secure, stable, and possibly even isolated from Internet, yet has administrative tools to do plenty of management and Hardware monitoring
3 - Tuning the performance and usability, making sure that all this increases my productivity. This step should be where scripts and cheatsheets with commands will come in to learn how to manage everything
The storage part is the first and possibily most important step. It is a thing which is better to get right the very first time you do it, because is a royal pain in the ass to re-do it later, as at the bare minimum you will require another HD with enough free space to copy all the stuff temporarily there while you start partitioning from scratch again if you have a new idea at a later time. Partitioning itself is a ridiculous mess, as every guide you read will usually have its view on what is the most efficient method to do it (Some based on technical merit, some on personal views of how to organize data), namely, amount and recommended size of partitions, File Systems, etc.
As I'm intending to do something very different to the estabilished on all the guides to meet my use case, I have to somehow define my own style - but I don't have the required knowledge to do so, which is why I need people to fill my holes and gaps. I have written a lot about both my use case and the ideas I had to define how to do it, but there are several things which still are lacking.
STORAGE
For everything I know about this matter, doing modifications like resizing partitions in a later stage is a pain in the butt, so I take that everything I do here is pretty much final, reason why I spend a lot of time thinking about it to not make mistakes. As this should also have an noticeable impact on I/O performance, it is even more important to get it right on first try.
BOOT LOADER / BOOT MANAGER
In my case, I have a single 4 TB Hard Disk. Using GPT instead of MBR is pretty much mandatory. My original idea was to use Gummiboot as Boot Manager to launch the xen.efi executable directly, however, Xen in UEFI mode never worked for me, and due to lack of tools to debug it I wasn't able to push any further. I didn't had issues using Syslinux, which can work as a BIOS Boot Loader but has GPT support, so I suppose than I will keep using it to boot the main OS until I can get everything working under UEFI.
PARTITIONING
The first thing to have in consideration is that the HD performance is variable depending on what part of the platter the data is physically at, being faster at the outer edge and slower near the motor. This means that the data that should be used often (Which should include the main Arch Linux + Xen Hypervisor installation, and maybe some of the most important VHDs) should be in the outer edge. As far that I know, LBA addresses starts from the outer edge and ends on the inner tracks, so if you make partitions following order of importance on a brand new HD, you will get it right.
The actual question should be how many partitions are actually needed, and what should be either an appropiate or confortable size for them. It should be a point where I'm not wasting tons of space that will never be used, yet will never have the need to resize them because they're too small and causes them to run out of space for critical stuff that HAS to be there.
At the very least, I would need 3 partitions: The first one will be the EFI System Partition (Which for me, will be unused, so is mostly a placeholder for when things works as I intend them to). According to some Microsoft info on the ESP, it had a recommended size of around 3xx MB and has to be FAT32 formatted. I decided to settle on 512 MB for the ESP. The second one will be the Hypervisor installation, for which 10 GB seems to be enough according to my tests (Is what I'm using currently), through I don't know how much it could grow if, say, I had anything that did intensive logging of the Hypervisor activities. It could also need to be bigger assuming I were to store multiple installation ISOs there instead of somewhere else. Finally, the third partition could be a single, big, storage partition which all the remaining space (3+ TB).
If I were to use other GPT-capable OSes, I would need a partition for each in case I'm intending to run them native, but as the idea is to run everything virtualized and not even bother with a native option, I don't see a need for those. Also, there could be more than one storage partition, as if I were to want to guarantee that the data is physically on the outer tracks boundary, so instead of a single, big, data partition, I could have two or three like if they were priority layers.
Examples of how my HD could end partitioned would look like this (And also, in this LBA order):
1- ESP (512 MB, FAT32)
2- Hypervisor (10 GB)
3- Storage fast (1 TB)
4- Storage slow (Remaining 2.9 TB or so)
1- ESP (512 MB, FAT32)
2- Hypervisor (10 GB)
3- Native OS 1 (120 GB or so)
4- Native OS 2 (120 GB or so)
5- Storage fast (1 TB)
6- Storage slow (Remaining 2.6 TB or so)
LOGICAL VOLUMES
This part overlaps a bit with partitioning, as deciding to use or not LVM may influence how I should partition on the first place. I suppose than the FAT32 ESP is untouchable as a physical partition, and that makes than the absolute bare minimum in a booteable HD is two partitions, with ESP being one for UEFI boot, and another partition using LVM comprising the entire HD, where I just set up virtual partitions following the previously proposed hierarchy without the drawbacks of physical partitions.
An important fact about LVM partitions is that they are a possible storage choice for Xen. Xen can use either standalone files as VHDs, or LVM partitions, which supposedly are faster in I/O. However, when I tried LVM partitions, it seemed that managing them was harder than just creating VHD files on the fly according to my needs, which is why I'm currently using files as VM storage. Maybe with more tools or knowledge, LVM makes more sense, and I can get more performance from the storage subsystem with these. I also don't know how much overhead LVM adds over standard physical partitions, in case I decide to futureproof the storage subsystem by using LVM, but then, instead of using LVM partitions for Xen storage, I keep using VHD files.
FILE SYSTEMS
Regarding File Systems, for ESP, FAT32 is a fixed choice. The partition where the Hypervisor will sit at, will possibly be EXT4. However, in case I take the LVM route, I don't know how much using LVM may influence in File Systems choices for logical partitions instead of physical ones.
Yet another thing that caught my attention was how much next-generation File Systems are hailed...
http://arstechnica.com/information-...-and-atomic-cows-inside-next-gen-filesystems/
http://arstechnica.com/information-...h-using-the-zfs-next-gen-filesystem-on-linux/
Most seems to hail these File Systems for system using RAID arrays with tons of HDs. I don't know how useful these File Systems will actually be for a partition in a single HD (For example, ZFS is useful against bitrot in a RAID array with redundancy, but if I want to have protection against it on a single HD, it literally halves the usable space because it needs to have two copies of everything). I also don't know about performance, how much I win/lose due to the possible added overhead for all the new features. ZFS additionally supposedly loved tons of RAM.
Basically, I'm interested in input if ZFS or BTRFS are worth the extra complications of learning all the quirks and usage forms of something more complex, compared to using just EXT4. Having looked for benchmarks, what I found on sites like Phoronix seems to be on RAID arrays and not as Single Disk setups. Also, ZFS supposedly takes cares of Logical Volumes, which is also LVM job, in what case they overlap. But I didn't researched a lot on that.
GENERAL STORAGE
Another thing that I was pondering about, was regarding where and how to store general data. While the VM's VHD with the Windows installations and all that is self explanatory, being local data exclusive for each VM, I will have also data that will be shared or needs to be easily accessible between many VMs, even if only temporarily. Examples included ISO collections of applications and games I have, movies or videos, etc. While I could store them in a LVM partition or VHD file that I could assign to a given VM at boot via the Xen Configuration File (I suppose storage devices are hotplug capable, but didn't looked into that), if I were to use ZFS for such purpose, Windows will not able to see them directly. I suppose that in order to take advantage of ZFS or BTRFS, I will have a dedicated Linux VM dedicated for storage and allowing Windows to access it via Shared Folders as in a network. Otherwise, I will have to store stuff in NTFS formatted VHDs.
One of the critical things about this, is the tools of the trade to manage files inside the LVM partitions or VHD files if I have to take down a VM for mainteinance. Also, tools that can make a LVM partition out of a VHD file or viceversa could be quite useful, too.
RAMDISK
Due to the fact that I have 32 GB of RAM, I was thinking on the possibility of making use of the excess of RAM as a RAMDisk, which provides beyond SSD I/O performance (Which I didn't had remaining budget for). Many games should be able to fit on a 20 GB or so RAMDisk, and as the computer is 24/7 on, RAMDisk volatility is a non issue for as long as the important stuff like Save Games are on the HD. I had experience working with one on WXP SP3 (Yep, using all the 32 GB of RAM on 32 Bits via PAE, long story) and had some success using symlinks (With NTFS does supports), through to get the most out of it you requiere batch files to copy, rename, and make the symlinks. I see it workable.
That is all what I have thinked about the storage part of my virtualized system. I expect that there will be people that already experimented with these things and have decided on a way or style for managing this stuff that may want to share, to help me take a choice on what to do and how. It has been more than 2 months since the last time I toyed with the configuration of this system, as after getting it to a usable state I decided to enjoy it instead of further optimizing (I was out of gaming a whole 3 weeks until getting VGA Passthrough working, it was pretty much equivalent to the Dark Ages). However, as Xen 4.4 was released recently, I was intending on starting from scratch, applying any ideas I had in the meantime for a final setup.
Yes, at first it seems to add complexity and overhead for seemingly no reason. What I want to accomplish with this? Eliminating Dual Boot. You see, one of the reason why I always through that I (And many other users) didn't even bothered to spend time in Linux, is because Dual Boot ain't for the lazy. If I can do all my everyday task on Windows, I wouldn't even bother to reboot to do in Linux what I can do in the current Windows session, because if I were to do that, I would need to reboot again if I wanted to play a game (You have Wine in Linux, but support and compatibility is not universal). Basically, while I always heared about Linux and know about some of its strengths, I never bothered with it because it couldn't fully replace Windows, and there were areas where Linux may have had an edge but it wasn't enough to even justify installing and learning to use it.
From December of last year, I managed to get quite close to what I want thanks to my new computer with VT-d support, having manage to get a fully functional Windows XP SP3 VM where I can play games, using Xen on top of an Arch Linux installation. After 3 months of using my Frankenstein setup I can say that stability and compatibility are near perfect (Not without some quirks, but almost there), and I would say that the whole thing is pretty much production-ready for power users that look at this idea the same way I do. This way, you can get the most of each OS strength by using them simultaneously. While making it work was quite an accomplishment itself, fine tuning such a complex setup seems to be even harder, from both configuration and usage sides.
I was intending to throw an extremely, extremely long wall of text bomb. You will get it anyway on the long run - this is just the tip of the iceberg. But I decided that dividing it in parts focusing on one aspect of the setup each time, would make it more palatable. Overally, the points I am interesed in, are:
1 - Set in stone the storage subsystem, that due to the fact that is the hardest thing to change after you get it working, should be also the most important to take care of
2 - Proper setup of a slim, lightweight Dom0 (Which currently is Arch Linux + Xen, however, I should re-check XenServer because there are some free versions with nearly all features), that should be secure, stable, and possibly even isolated from Internet, yet has administrative tools to do plenty of management and Hardware monitoring
3 - Tuning the performance and usability, making sure that all this increases my productivity. This step should be where scripts and cheatsheets with commands will come in to learn how to manage everything
The storage part is the first and possibily most important step. It is a thing which is better to get right the very first time you do it, because is a royal pain in the ass to re-do it later, as at the bare minimum you will require another HD with enough free space to copy all the stuff temporarily there while you start partitioning from scratch again if you have a new idea at a later time. Partitioning itself is a ridiculous mess, as every guide you read will usually have its view on what is the most efficient method to do it (Some based on technical merit, some on personal views of how to organize data), namely, amount and recommended size of partitions, File Systems, etc.
As I'm intending to do something very different to the estabilished on all the guides to meet my use case, I have to somehow define my own style - but I don't have the required knowledge to do so, which is why I need people to fill my holes and gaps. I have written a lot about both my use case and the ideas I had to define how to do it, but there are several things which still are lacking.
STORAGE
For everything I know about this matter, doing modifications like resizing partitions in a later stage is a pain in the butt, so I take that everything I do here is pretty much final, reason why I spend a lot of time thinking about it to not make mistakes. As this should also have an noticeable impact on I/O performance, it is even more important to get it right on first try.
BOOT LOADER / BOOT MANAGER
In my case, I have a single 4 TB Hard Disk. Using GPT instead of MBR is pretty much mandatory. My original idea was to use Gummiboot as Boot Manager to launch the xen.efi executable directly, however, Xen in UEFI mode never worked for me, and due to lack of tools to debug it I wasn't able to push any further. I didn't had issues using Syslinux, which can work as a BIOS Boot Loader but has GPT support, so I suppose than I will keep using it to boot the main OS until I can get everything working under UEFI.
PARTITIONING
The first thing to have in consideration is that the HD performance is variable depending on what part of the platter the data is physically at, being faster at the outer edge and slower near the motor. This means that the data that should be used often (Which should include the main Arch Linux + Xen Hypervisor installation, and maybe some of the most important VHDs) should be in the outer edge. As far that I know, LBA addresses starts from the outer edge and ends on the inner tracks, so if you make partitions following order of importance on a brand new HD, you will get it right.
The actual question should be how many partitions are actually needed, and what should be either an appropiate or confortable size for them. It should be a point where I'm not wasting tons of space that will never be used, yet will never have the need to resize them because they're too small and causes them to run out of space for critical stuff that HAS to be there.
At the very least, I would need 3 partitions: The first one will be the EFI System Partition (Which for me, will be unused, so is mostly a placeholder for when things works as I intend them to). According to some Microsoft info on the ESP, it had a recommended size of around 3xx MB and has to be FAT32 formatted. I decided to settle on 512 MB for the ESP. The second one will be the Hypervisor installation, for which 10 GB seems to be enough according to my tests (Is what I'm using currently), through I don't know how much it could grow if, say, I had anything that did intensive logging of the Hypervisor activities. It could also need to be bigger assuming I were to store multiple installation ISOs there instead of somewhere else. Finally, the third partition could be a single, big, storage partition which all the remaining space (3+ TB).
If I were to use other GPT-capable OSes, I would need a partition for each in case I'm intending to run them native, but as the idea is to run everything virtualized and not even bother with a native option, I don't see a need for those. Also, there could be more than one storage partition, as if I were to want to guarantee that the data is physically on the outer tracks boundary, so instead of a single, big, data partition, I could have two or three like if they were priority layers.
Examples of how my HD could end partitioned would look like this (And also, in this LBA order):
1- ESP (512 MB, FAT32)
2- Hypervisor (10 GB)
3- Storage fast (1 TB)
4- Storage slow (Remaining 2.9 TB or so)
1- ESP (512 MB, FAT32)
2- Hypervisor (10 GB)
3- Native OS 1 (120 GB or so)
4- Native OS 2 (120 GB or so)
5- Storage fast (1 TB)
6- Storage slow (Remaining 2.6 TB or so)
LOGICAL VOLUMES
This part overlaps a bit with partitioning, as deciding to use or not LVM may influence how I should partition on the first place. I suppose than the FAT32 ESP is untouchable as a physical partition, and that makes than the absolute bare minimum in a booteable HD is two partitions, with ESP being one for UEFI boot, and another partition using LVM comprising the entire HD, where I just set up virtual partitions following the previously proposed hierarchy without the drawbacks of physical partitions.
An important fact about LVM partitions is that they are a possible storage choice for Xen. Xen can use either standalone files as VHDs, or LVM partitions, which supposedly are faster in I/O. However, when I tried LVM partitions, it seemed that managing them was harder than just creating VHD files on the fly according to my needs, which is why I'm currently using files as VM storage. Maybe with more tools or knowledge, LVM makes more sense, and I can get more performance from the storage subsystem with these. I also don't know how much overhead LVM adds over standard physical partitions, in case I decide to futureproof the storage subsystem by using LVM, but then, instead of using LVM partitions for Xen storage, I keep using VHD files.
FILE SYSTEMS
Regarding File Systems, for ESP, FAT32 is a fixed choice. The partition where the Hypervisor will sit at, will possibly be EXT4. However, in case I take the LVM route, I don't know how much using LVM may influence in File Systems choices for logical partitions instead of physical ones.
Yet another thing that caught my attention was how much next-generation File Systems are hailed...
http://arstechnica.com/information-...-and-atomic-cows-inside-next-gen-filesystems/
http://arstechnica.com/information-...h-using-the-zfs-next-gen-filesystem-on-linux/
Most seems to hail these File Systems for system using RAID arrays with tons of HDs. I don't know how useful these File Systems will actually be for a partition in a single HD (For example, ZFS is useful against bitrot in a RAID array with redundancy, but if I want to have protection against it on a single HD, it literally halves the usable space because it needs to have two copies of everything). I also don't know about performance, how much I win/lose due to the possible added overhead for all the new features. ZFS additionally supposedly loved tons of RAM.
Basically, I'm interested in input if ZFS or BTRFS are worth the extra complications of learning all the quirks and usage forms of something more complex, compared to using just EXT4. Having looked for benchmarks, what I found on sites like Phoronix seems to be on RAID arrays and not as Single Disk setups. Also, ZFS supposedly takes cares of Logical Volumes, which is also LVM job, in what case they overlap. But I didn't researched a lot on that.
GENERAL STORAGE
Another thing that I was pondering about, was regarding where and how to store general data. While the VM's VHD with the Windows installations and all that is self explanatory, being local data exclusive for each VM, I will have also data that will be shared or needs to be easily accessible between many VMs, even if only temporarily. Examples included ISO collections of applications and games I have, movies or videos, etc. While I could store them in a LVM partition or VHD file that I could assign to a given VM at boot via the Xen Configuration File (I suppose storage devices are hotplug capable, but didn't looked into that), if I were to use ZFS for such purpose, Windows will not able to see them directly. I suppose that in order to take advantage of ZFS or BTRFS, I will have a dedicated Linux VM dedicated for storage and allowing Windows to access it via Shared Folders as in a network. Otherwise, I will have to store stuff in NTFS formatted VHDs.
One of the critical things about this, is the tools of the trade to manage files inside the LVM partitions or VHD files if I have to take down a VM for mainteinance. Also, tools that can make a LVM partition out of a VHD file or viceversa could be quite useful, too.
RAMDISK
Due to the fact that I have 32 GB of RAM, I was thinking on the possibility of making use of the excess of RAM as a RAMDisk, which provides beyond SSD I/O performance (Which I didn't had remaining budget for). Many games should be able to fit on a 20 GB or so RAMDisk, and as the computer is 24/7 on, RAMDisk volatility is a non issue for as long as the important stuff like Save Games are on the HD. I had experience working with one on WXP SP3 (Yep, using all the 32 GB of RAM on 32 Bits via PAE, long story) and had some success using symlinks (With NTFS does supports), through to get the most out of it you requiere batch files to copy, rename, and make the symlinks. I see it workable.
That is all what I have thinked about the storage part of my virtualized system. I expect that there will be people that already experimented with these things and have decided on a way or style for managing this stuff that may want to share, to help me take a choice on what to do and how. It has been more than 2 months since the last time I toyed with the configuration of this system, as after getting it to a usable state I decided to enjoy it instead of further optimizing (I was out of gaming a whole 3 weeks until getting VGA Passthrough working, it was pretty much equivalent to the Dark Ages). However, as Xen 4.4 was released recently, I was intending on starting from scratch, applying any ideas I had in the meantime for a final setup.
