Trivore Corp. logo

Welcome to Trivore VMware ESX Server v2.0.x hints 'n' tips
@ http://trivore.com/vmware/

This information was last updated on 7-Jul-2004.

This page is designed to aid in designing, setting up, and operating the VMware ESX Server v2.0.x platform. Most of this information has been gathered both during working with the product on different production environments, and during the several training sessions delivered on the product.

As this information has been gathered mostly for Trivore's VMware ESX Server class students, the representation might not always be the best possible. Also, some information is pretty purely GNU/Linux oriented. This is intentional, as many technical people are still pretty new to the GNU/Linux stuff running on the Service Console.

These hints and tips do not replace neither the excellent VMware ESX Server manual, nor a decent hands on GNU/Linux knowledge, nor a decent, hands-on-oriented VMware ESX Server classroom education. Any Unix knowledge makes your life a lot easier.

This document is a work in progress as long as v2.0.x is the latest version. Soon after that this document is frozen and a new one should be created. If there is something you might want to see here, or you would otherwise send me feedback, please email us. The maintainer of this information is Kari Mattsson of Trivore Corp.

Planning and documenting the system

This chapter focuses on the planning phase of system setup. It emphasises and reasons the importance of the documentation. More planning oriented material is also in the Storage chapter.

The files making up a VM (virtual machine)

Generally about 4 files make a VM: "xxx.dsk", "xxx.vmx", "xxx*.log", and "nvram". There could be more than one .dsk file associated to a VM (virtual machine). All files, but the .dsk file(s) are always in the same (sub)directory. It is most convenient to name that subdirectory "xxx".

The older .log files are always deleteable at will. The latest .log file can be deleted when the VM is powered off. As the log files do not take much disk space, most administrators let them be.

The nvram file might occasionally corrupt for some reason. It is known to happen, at least it did with v1.5.x. If that happens, power off the VM and just delete the file. It will be recreated next time the VM powers up. If any changes were made to the nvram via VM's BIOS setup, do not forget to remake those changes.

The most common change to the BIOS settings of a VM is to change the boot device order. The default is floppy, hard drive, CD/DVD. You might want to raise the CD/DVD to higher in the list to be able to boot from there.

In addition to the .dsk file, a .dsk.REDO file will exist in the same directory, if non-persistent, undoable, or append disk mode is active for that particular virtual SCSI disk. The .dsk.REDO file can grow up to several gigabytes. How large it will grow, depends only on what file/disk operations are done on the disk.

It is a good idea to place a brief descriptive "xxx.txt" file next to the "xxx.vmx" file to document what this VM is, what is its business purpose, what is the OS version and level, what are the main installed applications, and who is/are responsible for the VM. If you absolutely do not want the .txt file, you could place that same information to the comment lines of the .vmx file. Is the information safe there, is untested.

An alternative to the per VM .txt file approach is a single .txt file which contains information on all the VMs on the server. Okay, you can maintain this document anywhere with any tools, but remember, pure text files are most easily accessed and maintained on the ESX Server by the administrator(s).

Naming a VM, and its files is important. It will be crucial once there are tens (dozens for those of you still using legacy, non-metric systems :-) of VMs. It is a recommended practice to:

A powered on VM can be suspended, much like a laptop or any of the later Windows'. VMware uses the term "Suspend to disk". "Hibernate to disk" is actually a more accurate term here. When entering the suspend (hibernate) state, a suspend file will be created. It has a filename extension .vmss, and by default it is created to the .vmx file directory. It is usually (due to disk space restrictions) necessary to direct it to the same directory where the VM's operating system .dsk file is.

The .vmss file will be a few megabytes larger than the maximum RAM memory allocated to the VM.

Example list of files associated with vm001

This is an example list of files, which could be a real list.

/data/vm001/vm001.vmx
The configuration file defining all aspects of the VM.
/data/vm001/nvram
BIOS/CMOS settings of this VM. This file is (re)created if it is missing.
/data/vm001/vmware.log /data/vm001/vmware-0.log /data/vm001/vmware-1.log
Server created log file(s). Can be deleted to save disk space.
/data/vm001/vm001.txt
Small text file documenting this VM. It is strongly recommended to create this file.
/vmfs/public1/vm001sys.dsk
Boot disk (first physical disk) of the VM. It could have more than one partition.
/vmfs/public1/vm001data.dsk
Data disk (second physical disk) of the VM. It could have more than one partition.
Planning and documenting

Planning is important with ESX. It is especially important with the storage allocated to VMs. Some preliminary questions to answer are: What are the VMs to be installed on ESX Server? What operating systems will be installed? Are they file/print/database/application servers, or what? What kind of storage allocation will be needed? How much storage is initially reserved to a each VM? Are VMs executed on more than one ESX Server at the same time? What are the CPU requirements per VM? What are the network I/O requirements per VM? What are the disk I/O requirements per VM? What are the memory requirements per VM? How much storage is required for suspended VMs? Will non-persistent, undoable or append disk mode be used?

Documenting is another extremely important issue with ESX Server, as these systems easily become extremely complex. Mainframe and Unix style management attitude is really required. Document everything carefully. It will save your day often. The few line .txt file next to .vmx file is especially nice. An example skeleton .txt file follows: 

Name of the VM (virtual machine).:  [vm###]
Creation date for this file......:  [dd-Mmm-yyyy / creator's name]
Last update of this file.........:  [dd-Mmm-yyyy / updater's name]
Business purpose of this VM......:  [short description]
Responsible person(s) for this VM:  [name, contact (email and/or telephone)]
OS name and version in this VM...:  [OS name, update level]
Installed core application(s)....:  [app1, app2]

Storage: partitioning, filesystems, directories, files, etc.

This chapter covers the issues on storage. Some of this material is very planning oriented.

General information

Maximum number of partitions (which can contain data) per SCSI disk (=logical drive on RAID systems) in Linux kernel 2.4.x based systems is 14. The Service Console is based on kernel version 2.4.9. Of these max 15 partitions 3 are primary (1-3), 1 is extended (4), and 11 are logical (5-15). Extended partitions never contain any actual data. It merely acts as a container for the logical partitions.

This 14 data partition (partitions 1-3, and 5-15) limit could force you to create smaller logical drives than you wanted.

It is a nice practice to use primary partitions for non-vmfs filesystems, and logical partitions for vmfs filesystems. Nothing forces you to do that, it is just a clear choice.

SCSI disks are named as /dev/sda (first), /dev/sdb (second), /dev/sdc (third). The first disk is always the boot disk and (practically always) internal to the server. The other disks are usually on an external (usually fibre-connected) disk enclosure.

It is recommended to use CD/DVD ISO images, and/or floppy images. Images are very handy for OS and application installations. They are much more reliable and convenient than the actual CDs. Example commands on how to create these images are given later in this document.

On IBM, and other brand servers, the 10-100 MB System/Service/Diagnostic Partition accessible with [Alt-F1], [F10], or [F11] is partition /dev/sda4. VMware ESX Server recognises this partition as "Old VMware Core Dump Partition". Usually you do not want to touch this partiton, so leave it alone.

VMFS (vmfs, virtual machine file system)

Partitions (dubbed as volumes in VMware ESX Server) formatted as vmfs do not support subdirectories. It is one thing given up for the fast speed of vmfs.

VMware ESX Server v2.0 uses a new version 2 of vmfs dubbed VMFS-2. These volumes (partitions) have 2 possible accessibility settings:

VMFS-2 also allows spanning upto 32 extents. Use spanning only if you have to. VMFS-1 volumes can be converted to VMFS-2.

A note on .dsk files (seen as physical SCSI disks in the VM)

The virtual SCSI disks (.dsk files) have four different modes:

Last two modes are invaluable when upgrading or testing the software in a VM, as you can easily revert to the point you enabled that mode. Effectively you can undo any changes to the file system.

Strategy for placing all the files - VMware ESX Server Service Console files, and the VM files

It is wise to install VMware ESX Server to the local/internal disks (single RAID-1 SCSI logical drive). Dual 36 GB drives make plenty of space fir mid-size systems. Dual 72 GB is actually today more common and scales up-to high-end systems. You might want to go for the higher rotating speed drives to get faster ESX Swap.

Place all the VM-related files (guest OSs configs and disks) to external disks (logical drives) on fibre channel (or similar).

Standard: Partitioning the VMware ESX Server local/internal disk (/dev/sda)

This is where the VMware ESX Server and its Service Console will reside. Partitioning examples are for system with mirrored 36 GB local/internal drives.

Server with a System/Service partition (IBM, HP, etc.) 
DeviceName  PartType  Size     MntPoint    FStype  Remarks
/dev/sda4   primary   ~50      IBM xSeries System partition or equivalent.
/dev/sda1   primary   2000     /        ext2/ext3  Service Console root.
/dev/sda2   extended  33000+                       Logical partition container.
/dev/sda5   logical   1000     swap                Service Console swap.
/dev/sda6   logical   ~23000   /local        ext3  See notes below.
/dev/sda7   logical   100      vmcore              VMware Core Dump.
/dev/sda8   logical   ~8000    /vmfs/esxswap vmfs  See notes below.
Server without a System/Service partition 
DeviceName  PartType  Size     MntPoint    FStype  Remarks
/dev/sda1   primary   2000     /        ext2/ext3  Service Console root.
/dev/sda2   extended  33000+                       Logical partition container.
/dev/sda5   logical   1000     swap                Service Console swap.
/dev/sda6   logical   ~23000   /local        ext3  See notes below.
/dev/sda7   logical   100      vmcore              VMware Core Dump.
/dev/sda8   logical   ~8000    /vmfs/esxswap vmfs  See notes below.
Tuned: Partitioning the VMware ESX Server local/internal disk (/dev/sda)

This is really for experienced GNU/Linux administrators only. Use this partitioning scheme instead of the above one. For simpliciy, the possible System/Service partition is not shown.

This fine-tuning makes it possible to make the Service Console even more secure. You do not want to apply this plan unless you are very familiar with Linux and the Service Console. 

DeviceName  PartType  Size     MntPoint    FStype  Remarks
/dev/sda1   primary   100      /boot         ext3  Service Console kernel is here.
/dev/sda2   primary   500      /             ext3  Service Console root.
/dev/sda3   extended  34000+                       Logical partition container.
/dev/sda5   logical   500      /tmp          ext3  This is really not much used.
/dev/sda6   logical   1000     /var          ext3  All the local log files will be here.
/dev/sda7   logical   1500     /usr          ext3  All standard programs are here.
/dev/sda8   logical   50       /usr/local    ext3  All server local scripts/programs.
/dev/sda9   logical   350      /opt          ext3  Optional 3rd party programs.
/dev/sda10  logical   1000     swap                Service Console swap. See note below.
/dev/sda11  logical   19000    /local        ext3  See notes below.
/dev/sda12  logical   100      vmcore              VMware Core Dump.
/dev/sda13  logical   8000     /vmfs/esxswap vmfs  See notes below.
Partitioning and otherwise handling the external disk space for VMs

There are two grand philosophies (schemes) with partitioning the external disk space where VMs reside. You can combine the schemes, if required. Both schemes are presented below:

Scheme A is absolutely the most common, and used 99+ % of the time. It is however possible to mix both schemes and even do a hybrid one by following scheme A with the small files, and scheme B with the .dsk files.

Partitioning scheme A

This partitioning scheme allows easier growing of .dsk files and generally is more efficient on disk space usage. It is also a bit riskier. Something could happen to the vmfs partition where all the .dsk files are. The filesystem could corrupt or something.

Partitioning example for system with 200 GB fibre channel logical drive; all but the first formatted filesystems are of type vmfs and accesibility is set to either public (only single VMware ESX Server exists in the SAN) or public (also another VMware ESX Server exists in the SAN): 

/dev/sdb1   primary       200   ext2 fs       mounted as /data for VM configs and logs
/dev/sdb2   extended  ~200000   volume name   example name: vmhba2:1:0:2
/dev/sdb5   logical   ~160000   public1       /vmfs/public1; accessibility: public
/dev/sdb6   logical     39000   shared1       /vmfs/shared1; accessibility: shared
Partitioning scheme B

This partitioning scheme needs more preliminary planning, but is more convenient for the administrators operating the server.

It is important to note, that VMware does not endorse this scheme. They recommend placing just .dsk files to VMFS volumes.

Partitioning example for system with 200 GB fibre channel logical drive; all formatted filesystems are of type vmfs and accesibility is set to either public (only single VMware ESX Server exists in the SAN) or shared (also another VMware ESX Server exists in the SAN): 

/dev/sdb1   extended  ~200000	volume name   example name: vmhba2:1:0:1
/dev/sdb5   logical     10000	vm001         vm001-os.dsk, 6000
/dev/sdb6   logical     30000	vm002         vm002-os.dsk, 4000 & vm002-data.dsk, 20000
/dev/sdb7   logical     10000	vm003         vm003-os.dsk, 7000
/dev/sdb8   logical     95000	vm004         vm004root.dsk,10000 & vm004home.dsk, 75000
Multiple ESX Servers connected to the same fibre channel infrastructure

On multiple VMware ESX Server configurations with shared fibre channel storage, the following (seemingly a little conflictiong) issues and restrictions apply:

Network: setup and configuration-related issues

Please consider the following issues:

User structure on VMware ESX Server Service Console

There are again two schemes you can choose. The first one (A) is the one which is the default. The second one (B) has a quite different approach.

No matter which one you choose, you can always convert to the other one rather easily. Change-over propably takes quite a long time, as it requires powering off all the VMs one-by-one for a short period of time.

Service Console user structure scheme A

Using this scheme, you create one userid for each administrator. The are all placed to one primary group "users". When VMs are created, the creator-administrator gets "rwx" rights to the VM, and the group gets "rx" rights. This means only the creator-administrator can make changes to the VM. Depending on your environment, this could, or could not be what you want.

Service Console user structure scheme B

Using this scheme, you create one userid for each administrator. The are all placed to one primary group "vmadmins". (If multiple groups are needed, create groups "admins01", "admins02", etc.) When VMs are created, the creator-administrator gets "rwx" rights to the VM, and the group gets "rx" rights. This means only the creator-administrator can make changes to the VM.

The difference between schemes is made by:

As you can see, the difference between scemes is quite small. It is still very significant. Using scheme B also the "vmadmins" group can make required changes to the VM. Now, depending on your environment, this could, or could not be what you want. The earlier you make your choise between schemes, the easier.

VMware ESX Server tools and documentation

Most, if not all VMware Service Console utilities have a good man page. Enter "man xxx" to get the manual page for command "xxx". Example: man vmkfstools.

VMware ESX Server vmfs partition mount utilities are "mount-vmfs" and "umount-vmfs". The only parameter needed for these commands is the vmhda-name. By default public vmfs partitions are mounted under /vmfs/ directory. You have to mount public and shared partitions manually or you could mount them automatically at every boot by placing correct mount-vmfs lines to the end of /etc/rc.d/rc.local file on the Service Console.

General and very comprehensive vmfs filesystem utility is "vmkfstools". It can format, convert, and rename volumes. It can shrink and expand .dsk files. It can do many many more important things.

"vdf" tool displays the amount of free space on different volumes, including VMFS volumes.

ESX netcard utility for locating correct netcard among many netcards is "findnic".

Both of these, and other tools are properly documented in the VMware ESX Server manual.

VMware ESX Server-related Linux files and directories

There are a couple of files and directories you should know about. The most important ones are listed below.

/etc/modules.conf
This file contains a list of devices in the system available to the Service Console. Usually the devices allocated solely to VMs, but physically existing on the system are also shown here in the commented-out ("#") lines. This is an important file for root and administrators.
/etc/fstab
This file defines the local and remote filesystems which are mounted at ESX Server boot.
/etc/rc.d/rc.local
This file is for server local customisations required at the server bootup. Potential additions to this file are public/shared vmfs mounts.
/etc/syslog.conf
This file configures what things are logged and where. Some examples are given below:
*.crit     /dev/tty12
This example logs all log items at level "crit" (critical) or higher to the virtual terminal at tty12. You can see this log by pressing [Alt]-[F12] on the console.
*.=err     /dev/tty11
This example logs all log items at exactly level "err" (error) to the virtual terminal at tty11. You can see this log by pressing [Alt]-[F11] on the console.
*.=warning     /dev/tty10
This example logs all log items at exactly level "warning" to the virtual terminal at tty10. You can see this log by pressing [Alt]-[F10] on the console.
*.*     192.168.31.3
This example forwards everything (all syslog entries) to another (central) syslog server. Pay attention to that server's security.
/etc/logrotate.conf
This is the main configuration file for log file rotation program. It defines the defaults for log file rotation, log file compression, and time to keep the old log files. Processing the contents of /etc/logrotate.d/ directory is also defined here.
/etc/logrotate.d/
This directory contains instructions service by service for log file rotation, log file compression, and time to keep the old log files. For the three vmk* files, raise "250k" to "4096k", and enable compression.
/etc/inittab
Here you can change the amount of virtual terminals available on the Service Console. Default is 6, but you can go up to 9. I always go :-)
/etc/bashrc
The system default $PS1 is defined here. It is a good idea to change "\W" to "\w" here to always see the full path while logged on the Service Console. This is one of my favourites.
/etc/profile.d/colorls.sh
Command "ls" is aliased to "ls --colortty" here. Many admins don't like this colouring. You can comment-out ("#") this line. I always do this one, too.
/etc/init.d/
This directory contains the actual start-up scripts.
/etc/rc3.d/
This directory contains the K(ill) and S(tart) scripts for the default runlevel 3. The services starting with "S" are started on this runlevel, and the services Starting with "K" are killed, i.e. not started..
/var/log/
This directory contains all the log files. VMware's log files start with letters "vm". The general main log file is "messages".
/etc/ssh/
This directory contains all the SSH daemon configuration files, public and public keys. The defaults are both secure and flexible and rarely need any changing.
/etc/vmware/
This directory contains the most important vmkernel configuration files.
/etc/xinetd.conf
This is the main and defaults setting configuration file for xinet daemon. Processing the contents of /etc/xinetd.d/ directory is also defined here.
/etc/xinetd.d/
This directory contains instructions service by service for if and how to start the service. Of the services here, vmware-authd, wu-ftpd, and telnet are most interesting to us. Two of the most interesting parameter lines are "bind =" and "only_from =", which allows limiting service usage.
/etc/ntp.conf
This file configures the NTP daemon. Usable public NTP servers in Finland are ntp1.funet.fi, and ntp2.funet.fi. Remember to change the service to autostart at runlevel 3.

VMware ESX Server-related Linux commands

There are several commands you should familiarise yourself with. Most of them and some more are listed here. All of them have an online manual page, which you can read with the command "man command-name".

man
Prints the manual page for a command or a configuration file entered as a parameter to this command.
reboot
Does a nice reboot on the system. Does "Force Power Off" for the VMs.
halt
Does a nice halt on the system. Does "Force Power Off" for the VMs.
shutdown
Generic command for shutting down or rebooting the system.
fdisk
Command line disk partitioning program in Linux. It is powerful and has a very simple user interface.
fdisk /dev/sdb
On command line, starts fdisk against second available SCSI disk. "sda" is the first SCSI disk, "sdc" is the third SCSI disk etc. VMware ESX Server is installed on /dev/sda, and the external storage is /dev/sdb, and maybe some others too.
p
Fdisk subcommand, prints the current partition table on current disk.
d
Fdisk subcommand, deletes an existing partition. Enter the partition number to delete. It is recommended to printout the current partition table before deleting anything.
n
Fdisk subcommand, creates a new partition. Select partition type (primary, extended, or logical). Almost always you should use the default starting cylinder. For size, enter "+NNNNNm", where NNNNN is the size in megabytes.
t
Fdisk subcommand, change partition type (id). By default fdisk creates ext2 type partitions. We might also want to use id "fb", the vmfs type, or some other type.
w
Fdisk subcommand, writes the current partition table to disk. If you don't get any errors, you don't have to reboot. If you get errors at this point, the new partition table is used only after next system boot.
mke2fs
This command formats a partition for ext2, or ext3 filesystem.
mke2fs -j /dev/sdb1
Formats /dev/sdb1 using ext3 filesystem.
mke2fs /dev/sdb1
Formats /dev/sdb1 using ext2 filesystem.
mkdir
Makes a directory.
mkdir /data
Creates directory /data for the VM configs.
mount|umount
These commands manually mount/umount CDs, floppies, local partitions, and remote directories to a selected local directory. The local (empty) directory must exist before the mount can succeed. Example mound command would be "mount /dev/sdb5 /data". Permanent mounting is done by editing the /etc/fstab file.
mount
Shows all the active mounts.
mount -a
Remounts everything specified in /etc/fstab file.
mount /dev/cdrom
This command does the default mounting of a CD to the default mountpoint. In Service Console the CD is mounted to /mnt/cdrom directory.
mount /mnt/floppy
Mounts a normal 1440KB floppy (/dev/fd0) to the specified directory.
mount -t iso9660 -o loop /local/w2005srv.iso /mnt/isocd
Mount a CD/DVD ISO image file to the specified directory. This is very useful for testing and other purposes. The mountpoint directory must exist (mkdir /mnt/isocd) before mounting.
umount /mnt/cdrom
Unmount anything mounted to the specified mountpoint. If nothing is mounted, the command does nothing.
rm
Removes files and/or directories.
mv
Moves and/or renames files and/or directories.
kudzu
This is the RedHat's tool to detect and configure hardware: adding new and removing old. When you run kudzu, or system runs it at bootup, be careful. Kudzu might offer to remove hardware you have dedicated solely to the VMs. Know your hardware and configuration. It might be a good idea to refer to /etc/modules.conf file before running kudzu. A safe action to select in kudzu is "Do nothing". Select it when in doubt.
service
RedHat-made tool for daemon (service) starting/stopping/restarting/status querying. Syntax is "service servname [start|stop|restart|status]". Alternate to this command, which works with all Linuces is to call the script directly, like "/etc/init.d/sshd restart".
groupadd
Adds a new group to the Service Console. It is recommended to use one non-root group for VM admins and add operator/admin users there. To create that group, enter the following command:
groupadd -g 7777 vmadmins
Create a group with groupid number 7777. This number is an arbitary number. For practical (not explained here) reasons this number should have four digits.
useradd
Adds a new user to the Service Console with status disabled. To create an account for the new admins, enter the following commands:
useradd -c "VMware ESX server operator" helpdesk
Create a single userid, which will be able to operate all of the VMs.
useradd -g 7777 johndoe
Create a userid, and make groupid 7777 (vmadmins) as its primary group.
useradd -g 7777 -c "Kari Mattsson" mattkar2
Create a userid, and make groupid 7777 (vmadmins) as its primary group.
usermod
Changes settings for a user. Usually used for user group manipulation.
usermod -G wheel mattkar2
passwd
Changes the password for the userid entered as a parameter for the command. Only root can change the password for other users. They can only change their own password with command "passwd". Userids are disabled by default. They are activated by setting a password for them. An example command for root to set a password is the following command:
passwd johndoe
chown
Changes the owner user and optionally owner group of a directory, or a file. Optionally this command works recursively with parameter "-R". The assignment parameter is of type "user.group", or just "user". Some examples are given below:
chown -R helpdesk.vmadmins /vmfs /data
Recursively change the user-owner, and the group-owner of specified files/directories to userid.groupid.
chown helpdesk.vmadmins /vmfs/local/*
chown -R root /data/vmware
chown root.vmadmins /etc/modules.conf
chgrp
Changes the owner group of a directory, or a file. Optionally this command works recursively with parameter "-R". Examples for "chown" apply here, but without the "root." part, as only the group is changed here.
chattr
Change special attribute of a directory, or a file. Immutable attribute is set with parameter "-i".
chmod
This command is the main command for changing file modes. Like chown, it can do things recursively with parameter "-R". Below are some example commands:
chmod -R 0775 /vmfs/* /data/*
chmod u=rwx,g=rwx,o=r /vmfs/freebsd462/*
chmod g+rwx /vmfs/vm007/*
chmod -R u+rwx,g=r,o-rwx /var/log/*
chmod u=rw,g=rw,o=r /etc/modules.conf
chmod 664 /etc/modules.conf
chmod u=rw,g=rw /vmfs/*/*.dsk
It appears, that this last example works rather nicely. Note, that those VMs which are powered-on, have their .dsk files locked.
dd
With this 'disk dump' command you can create ISO images and floppy images. You can also use it to create imagefiles of partitions and whole disks. Below are some example commands:
dd if=/dev/cdrom of=/local/suse90pro-dvd1.iso bs=2048
dd if=/dev/cdrom of=/local/w2003srv.iso bs=2048
The above two examples create an ISO image of a CD/DVD. You can safely ignore the error message usually shown at the end of the media.
dd if=/dev/fd0 of=/local/bootfloppy1.img bs=1440k
This command creates a floppy image quickly.
dd if=/dev/fd0 of=/local/bootfloppy2.img bs=512
This is a bit slower version of the above example.
cat
ConCATenate file from start to standard output (terminal screen by default). Usually takes filename as a parameter.
ls
LiSt files in a directory. -R makes it recursive, and -l shows more information on each item.
stat
Show statistics of a file. This is the most comprehensive directory entry examiner.
tac
Like "cat", but starts from the end of the file (or standard input).
head
Show selected amount of lines from the start of a file.
tail
Like "head", but start from the end of the file. Practical command to follow what is happening with a log file is command like "tail -f /var/log/messages".
grep
Search for a string from standard input or from a file. This is a powerful command.
find
Find files by name or many of the other attributes. Another very powerful command. Below are some example commands:
find /vmfs -type f -iname *.dsk
find /data -type f -iname *.vmx
find / -type f -iname *.bak
find . -type d -name sbin
find / -type f -name *
tar
Tape ARchive, a command which combines many files into one for backup purposes. Below are some example commands:
tar -cvzf /local/servcons.tar.gz --exclude /proc --exclude /local --exclude /vmfs --exclude /data /
Create a gzipped tar backup file the whole Service Console.
tar -cf /local/vm-configs.tar /data
Create a tar backup file of all files in and under /data directory.
tar -xvzf /local/vm007-config.tar.gz
Extract gzipped tar backup file to current directory.
find / -type f -iname vm007* | tar -czvf /local/vm007-backup.tar.gz -
Find all files starting as 'vm007', and create a compressed backup tar file of them.
gzip|gunzip
These command compress and decompress files. The recommended and default extension is .gz.
more|less
These commands are almost the same, and usually act in a pipe. They are used for file pagination to terminal. Below are some example commands:
zcat /var/log/vmksummary.1.gz | less
more /etc/passwd
ntpdate
This command takes an NTP server as a parameter and synchronises the clock once. This command doesn't work when local NTP daemon is running. Example: ntpdate europe.pool.ntp.org

Install VMware tools to Linux

Ádmins with Windows background intuitively know how to install the tools in Windows VMs. This Linux information is for those admins.

When installing X to the Linux VM, select "Unsupported VGA compatible" as the display adapter.

First you have to open "xterm" or similar CLI. You also could press [Ctrl]+[Alt]+[F2] to switch to TTY2. Login as root there, if not already logged-in. Then follow the instructions.

After selecting in the Remote Console "Settings", and then "VMware Tools Install". Enter the following commands: 

cd /tmp
mount /dev/cdrom
cp /mnt/cdrom/* /tmp           * See the note below!
umount /dev/cdrom
tar -xzf vmware-linux-tools.tar.gz
cd vmware-tools-distrib
./vmware-install.pl

A note on the 'cp' command above:
On RedHat-family of Linuces CD/DVD is mounted to '/mnt/cdrom'. On SUSE it is mounted to '/media/cdrom', and/or '/media/dvd'. On Debian it is mounted to '/cdrom'. So change the 'cp' command accordingly.

...now just accept the defaults by pressing [Enter], select proper resolution when asked, and you are done with the VMware Tools install.

Network: A note on Service Console (Linux) security in production use

This discussion presumes the default high security level is set during the initial VMware ESX Server configuration. In this security level only the following ports are open on the Service Console:

22/tcp
SSH daemon listens to this port for remote connections. By default password authentication is used for logons. RSA/DSA public/private key authentication can be used and it is actually tried first. Userid/password authentication is actually tried second. For higher security and for automated/scripted logons RSA/DSA authentication must be used.
902/tcp
VMware authd, the web management UI (MUI) and remote console authentication daemon (service) for VMware ESX Server uses this port. The daemon is not listening this port directly, but xinetd does. When someone open connection to port 902, xinetd then launches authd, and the actual authentication starts. Xinetd-related authd security is defined in the file /etc/xinetd.d/vmware-authd.
80/tcp and 443/tcp
The httpd.vmware application web server listens to these ports. With high security on, all connections to port 80 are automatically redirected to port 443.
8222/tcp and 8333/tcp
These ports are used by ESX Server's web UI. They are just forwards to ports 80 and 443 respectively. These ports need not to be open on the firewalls.

Remember, that sshd is by default always running on the Service Console, so you can always connect to it and do low level management directly to the Service Console files. An example of this kind of management is when the MUI stops responding. Just login using your account via ssh, and enter the following command to restart the webserver responsible for the MUI: su -c "/etc/init.d/httpd.vmware restart". You normally need the root's password to complete the task. You could (should!) also use sudo to make thing even easier.

Additional tasks to increase security

This is mainly a list of things to do. Not all of this applies to all installations.

  1. Disable SSH login for root account
    Edit file /etc/ssh/sshd_config and change line # PermitRootLogin yes to PermitRootLogin no. The restart sshd with command /etc/init.d/sshd restart. From now on you have to use you normal account to login via SSH, and then use su - command to switch to root, or use sudo command to execute commands as a root.
  2. Limit access to the "su" command
    This is done by adding each user to a special group "wheel" with command usermod -G wheel userid. Then edit file /etc/pam.d/su and remove comment mark "#" from the following line: auth required /lib/security/pam_wheel.so use_uid. Now users in "wheel" group can use su command.
  3. Enable sudo command for the user group "vmadmins"
    With this setting enabled, all the users in "vmadmins" group can execute commands that require root privileges. When they do, they are asked their own password before the command is executed. That way reasonable security is enforced. This task is easily done by executing the "visudo" command, and adding the following line to the end of the file: %vmadmins ALL=(ALL) ALL, or rather more specific %vmadmins ALL=(ALL) "/usr/local/vmadmin-scripts". On the latter example, you also have to link all the scripts you want to enable to the specified directory. You could fine-tune this by editing the "/etc/sudoers" file with visudo.
  4. Use PAM to selectively disable access to selected services with pam_listfile.so module
    You can limit what is allowed user by user, service by service. Discussion on this is beyond the scope of this document.

Backup VMs to other host with sshd on the Service Console

Please note, that from the "other host", where you have backed up the VMs, the VMs can be backed up to tape, etc.

This discussion presumes the following: Now the following shell script can be used on the "other host" to fetch a VM passed as a parameter to the script: 
#!/bin/bash
# Fetch files related to VM $1 from VMware ESX Server using userid 'backup'.
export vmname=$1
mkdir /backup/esx01.trivore.com/${vmname} 2>/dev/null
cd /backup/esx01.trivore.com/${vmname}
scp -p backup@esx01.trivore.com:/vmfs/${vmname}/* .

Name the above script "/usr/local/bin/esx-vm-backup", and execute it with parameter "vm001", or any other VM name. Example command: /usr/local/bin/esx-vm-backup vm001

Task: Do a full backup of the Service Console

This backup, to ensure all files make it to the backup, shoulp be done in runlevel 1, i.e. in single user state. In that runlevel, network, and vmfs volumes are all unavailable. Also the ESX Server is not up in runlevel 1.

There are two easy ways to enter runlevel 1 when at the VMware ESX Server console:

It is presumed here, that a sufficiently large partition exists, and is mounted to /backup/. If not, then some adaptations has to be made. A working example command to make a compressed tar backup file is: tar -czf /local/service-console-2003-11-22.tar.gz --exclude /proc --exclude /local /

The above command also backups all the VM configs in and under /data directory. If you do not want that, just add "--exclude /data" to the tar backup command.

Normal size for the backup file is under 200 MB. If you install a lot of software to the Service Console, it can grow to, or a bit over 200 MB.

For help in restoration you have to contact someone with enough Linux knowledge, as the skenarios, options, and choices on what/how to do it vary so much.

Summary of initial tasks to be done just after installation (...or much later :-)

These command could be combined to a script to be executed. I just haven't bothered to do it. Please note, that currently there are not much comments here on what the commands do. 

useradd -c "VMware ESX server operator" helpdesk
groupadd -g 7777 vmadmins
useradd -g 7777 admin01 ; useradd -g 7777 admin02
passwd admin01 ; passwd admin02
echo "*.=crit     /dev/tty12" >> /etc/syslog.conf
echo "*.=err      /dev/tty11" >> /etc/syslog.conf
echo "*.=warning  /dev/tty10" >> /etc/syslog.conf

Task: fdisk /dev/sdb:
   create 200 MB sdb1, ext2, or ext3 filesystem
   create rest-of-drive sdb2, extended.
   then you can enter the following commands:
      mke2fs -j /dev/sdb1
      mkdir /data
      echo '/dev/sdb1  /data  ext3  defaults  0 1' >>/etc/fstab
      mount -a
Task: nano /etc/bashrc:
   change 'umask 022' to 'umask 002' to everyone.
   change '\W' in PS1 environment variable to '\w'.
Task: nano /etc/inittab: 
   add mgettys for tty7, tty8, and tty9.

# these commands below should propably be executed more often
# to fix the owners and rights to the VMs:
chown -R helpdesk.vmadmins /data /vmfs
chmod 0775 /data /vmfs
find /data -type f -name *.vmx -exec chmod 0770 {} \;
find /data -type f -name nvram -exec chmod 0660 {} \;
find /data -type f -name vmware*.log -exec chmod 0660 {} \;
find /vmfs -type f -name *.dsk -exec chmod 0660 {} \;

A hint for printing this document: Set the left and right margins to as narrow as possible on your browser. An alternative hint is to print this document in landscape form, not as portrait.

© 2003-2004 Trivore® Corp.